# robots.txt - non-standard and unknown fields mixed with real rules User-agent: * Disallow: /admin/ Noindex: /legacy/ # never standardised; unsupported since 2019 Host: www.example.com # single-crawler canonical-host hint Clean-param: ref /catalog/ # single-crawler query-stripping hint Nofollow: /out/ Disallow-All: yes X-Made-Up-Field: 1 User-agent: * Disallow: /tmp/