txt219 B
robots.txt
A robots.txt with wildcard and per-agent rules, a crawl-delay, and a sitemap reference — for testing robots parsers and crawler policy handling.
Preview — first 14 linestxt
# robots.txt for example.com (sample fixture)
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sessionid=
Allow: /
User-agent: GPTBot
Disallow: /
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml
Specifications
- Format
- robots.txt
- Encoding
- UTF-8
- Data
- fictional example.com
What is a .txt file?
TXT is a plain-text file containing unformatted character data with no styling or structure beyond line breaks. Its interpretation depends on character encoding, most commonly UTF-8, and on line-ending convention. It is the most universal and portable text container.
How to use this file
Use an example TXT to test encoding detection, line-ending (LF versus CRLF) handling, and any tool that reads or streams raw text input.
Related files
- txtads.txtAn IAB ads.txt listing authorised digital sellers with account IDs and relationships (sample data) — for testing ads.txt parsers and ad-fraud tooling.
- txthumans.txtA humans.txt crediting the people and stack behind a site, in the conventional TEAM/SITE block format — for testing plain-text metadata parsers.
- xmlsitemap.xmlAn XML sitemap (sitemaps.org 0.9) with loc, lastmod, changefreq, and priority for several URLs — for testing sitemap parsers and crawl-scheduling tools.
- txtsecurity.txt (RFC 9116)An RFC 9116 security.txt with Contact, Expires, Encryption, and Policy fields — normally served at /.well-known/security.txt, for testing security.txt parsers.
- jsService Worker (sw.js)A sample cache-first service worker handling install, activate, and fetch events with an offline fallback — for testing service-worker registration, JS parsers, and PWA tooling.
- pngApple touch icon (180×180 PNG)A 180×180 opaque PNG apple-touch-icon (iOS rounds the corners itself) — for testing home-screen icon extraction and Apple touch-icon handling.
Generated by generation/web_assets.py. Free for any use, no attribution required — license.