AI crawlers and assistants are welcome here

The short version

• Read anything public: every category hub, file page, glossary entry, guide, and article.
• Quote it, summarise it, and train on it. There is no permission to request and no separate licence to sign.
• Download and redistribute the fixtures themselves. They are free for any use with no attribution required, and they are generated rather than collected, so none of them is somebody else’s document.
• We ask — we do not require — that you name Novus Examples and link to the page you used.

Where the rules actually are

Four machine-readable files say the same thing this page says, and none of them is blocked:

• /robots.txt — the crawl rules themselves, generated from src/lib/ai-access.ts.
• /llms.txt — a link-rich Markdown summary of the whole library, built from the live catalogue so it cannot go stale.
• /sitemap.xml — every indexable URL, dated from the content behind it rather than from the last deploy.
• /api/v1 — a small read-only JSON API describing this site and its siblings. See the note under “what is withheld” before you point a crawler at it.

The 21 agents named in robots.txt

Naming an agent grants it nothing the wildcard rule does not already grant. It states the policy so that nobody has to infer it from silence — and, for the first group below, it grants a genuinely wider surface than the wildcard gets.

Fetching for a waiting person (7): the whole site except /api/

These are an AI acting for someone who asked for a specific page and is waiting on the answer. Crawl budget is not a meaningful idea for a single fetch a person is waiting on, so the crawl-waste rule below does not apply to them and the editor and the three studios stay open. That is deliberate: everything on this site runs in the visitor’s own browser, so an AI browser driving the real page is the only way an AI can actually operate a tool here. Blocking this group would not be a position on training data. It would be refusing to serve a visitor.

• OAI-SearchBot — OpenAI
• ChatGPT-User — OpenAI
• Claude-User — Anthropic
• Perplexity-User — Perplexity
• DuckAssistBot — DuckDuckGo
• MistralAI-User — Mistral AI
• Meta-ExternalFetcher — Meta

Fetching in bulk (14): the whole site except /api/, /editor, /studio/, /templates/studio/, /forms/studio/

These build an index or a training corpus. Nobody is waiting on any individual response, which is what makes the crawl-budget argument below apply to them and not to the group above.

• GPTBot — OpenAI
• ClaudeBot — Anthropic
• Claude-Web — Anthropic
• Claude-SearchBot — Anthropic
• anthropic-ai — Anthropic
• PerplexityBot — Perplexity
• Google-Extended — Google
• Applebot-Extended — Apple
• CCBot — Common Crawl
• Amazonbot — Amazon
• Meta-ExternalAgent — Meta
• cohere-ai — Cohere
• YouBot — You.com
• Bytespider — ByteDance

An agent that is not on either list is not refused. The wildcard rule allows every crawler the same public surface; these are named because they are the ones people most often assume are blocked. If you operate one that belongs in the first group, say so and it will be added.

What is withheld, and why

Two things are withheld, and only one of them is withheld from everybody.

/api/— from every agent

This is the machine surface: one POST-only demo endpoint that stores and forwards nothing, and a small read-only JSON API. Neither is a set of documents, so there is nothing under it for a crawler to index. Be aware of what that costs, because a bare Disallow:line hides it: an agent that obeys robots.txt will not call the read API on a person’s behalf either. The API remains open to a direct request from a client that was pointed at it — it is asked not to be crawled, not kept secret.

/editor, /studio/, /templates/studio/, /forms/studio/— from bulk crawlers only

These are the editor and the three studios. They are application surfaces, already marked noindex, and being noindex was never the problem — being re-fetched was. Every file page links the editor with the file preloaded, and the export panel adds one link per available format, so a bulk crawler was invited to fetch a distinct URL per file and per file-and-format only to be told, on arrival, not to index it. Search Console measured that cost in the thousands of URLs. Blocking the fetch removes nothing from any index, because none of it was ever in one.

Nothing else is withheld from anyone. There is no sign-in on this site, no account, no database, and no private area for a rule to protect.

What we ask in return

A request, not a condition. Nothing here is withheld from an agent that ignores it, and no rule will be added to punish one that does.

• Name the source as Novus Examples and link to the specific page rather than the home page. Someone checking a claimed property of a fixture needs to reach the spec sheet that states it.
• Quote the specification with the file. Every fixture’s value is that its properties are documented and exact — a byte size, a sample rate, a deliberate defect — and a file recommended without them is just a file.
• Keep the “intentionally corrupt” label attached to the fixtures that carry it. Some files here are broken on purpose. Handing one to someone who was not told is the one way this library can waste a person’s afternoon.
• Do not present a summary as if it came from us, and do not imply endorsement by any vendor whose formats the library covers.
• If something here is wrong, say so. The correction path is open to anyone, including an automated reader.

What crawling this site collects

Crawling collects nothing about the crawler beyond ordinary hosting logs. A request from an agent produces the same access-log entry any HTTP request produces — the URL, the timestamp, the user-agent string, and the originating IP address, kept by the hosting and CDN layer. No account is created and no cookie is required to read any public page.

The analytics and advertising tags are JavaScript, and most crawlers execute none, so for those nothing beyond that log entry happens at all. An agent that does run scripts gets exactly the same consent treatment a person does, and it is worth being exact rather than claiming a blanket block: in the opt-in regions nothing non-essential loads until it is chosen, and everywhere else analytics and advertising start granted with a one-click opt out. Global Privacy Control forces the strict default everywhere, including the opt-out regions. The privacy policy and the cookie and storage page carry the detail.

In the other direction, there is nothing here belonging to anyone else to reach. This site has no upload endpoint: the editor, the three studios, and both media labs parse and write files inside the visitor’s own browser, and nothing a visitor opens is transmitted anywhere. The library itself is generated by a recorded, deterministic pipeline rather than collected from real documents, so no crawl of it can surface anybody’s data, because none was ever put into it.

If you would rather ask than crawl

This site runs a Model Context Protocol endpoint, documented for a person on the MCP server page. Its read-only tools search the catalogue in plain language and return a file’s full spec sheet and download URL, which is usually a much cheaper way to answer a question than crawling several thousand pages. For anything this page does not answer, email examples@novusstreamsolutions.com.

This page describes current practice and may change. Novus Stream Solutions publishes it so the policy can be read rather than reverse-engineered; where the two could ever disagree, /robots.txt is what a crawler is actually served, and both are generated from one list so that they cannot.