Crawler File — Robots Disallow Admin
SAMPLE crawler/policy text file (robots-disallow-admin) for bot and ads.txt parsers.
User-agent: *
Disallow: /admin/
Disallow: /api/internal/
Allow: /
Sitemap: https://sample.example/sitemap.xml
Specifications
- Wave
- I
- Role
- crawler-file
What is a .txt file?
TXT is a plain-text file containing unformatted character data with no styling or structure beyond line breaks. Its interpretation depends on character encoding, most commonly UTF-8, and on line-ending convention. It is the most universal and portable text container.
How to use this file
Use an example TXT to test encoding detection, line-ending (LF versus CRLF) handling, and any tool that reads or streams raw text input.
How to use this file for testing
“Crawler File — Robots Disallow Admin” is a deterministic Novus Examples fixture for Web assets. Favicons, web app manifests, service workers, robots and sitemap files, Open Graph images, and .well-known resources — for testing web tooling, crawlers, PWA installers, and asset pipelines.
Documented properties for this file: crawler-file. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such — expect parsers to fail loudly rather than silently accept them.
Web-platform fixtures are standards-compliant samples against fictional example.com data. Test crawlers, PWA installers and manifest validators, favicon/icon pipelines, service-worker registration, or .well-known parsers against the documented structure.
Related files
- txtads.txtAn IAB ads.txt listing authorised digital sellers with account IDs and relationships (sample data) — for testing ads.txt parsers and ad-fraud tooling.
- txthumans.txtA humans.txt crediting the people and stack behind a site, in the conventional TEAM/SITE block format — for testing plain-text metadata parsers.
- txtrobots.txtA robots.txt with wildcard and per-agent rules, a crawl-delay, and a sitemap reference — for testing robots parsers and crawler policy handling.
- xmlsitemap.xmlAn XML sitemap (sitemaps.org 0.9) with loc, lastmod, changefreq, and priority for several URLs — for testing sitemap parsers and crawl-scheduling tools.
- txtads.txt — Wave G SAMPLEads.txt — Wave G SAMPLE for crawler and .well-known tooling.
- pngApple touch icon (180×180 PNG)A 180×180 opaque PNG apple-touch-icon (iOS rounds the corners itself) — for testing home-screen icon extraction and Apple touch-icon handling.
Generated by generation/pad_wave_i.py. Free for any use, no attribution required — license.