Your robots.txt is the first file a well-behaved crawler asks for, and most of the files in circulation were last edited for a search engine, years ago. They still work, in the sense that nothing breaks. They also assume a reader that stopped being the only one once assistants began fetching pages on people's behalf. This post takes one such file, works through it line by line against the rules a parser actually applies, and ends with a complete file you can adapt. Three checks in our rubric read robots.txt directly: A1 (exists and parses, two points), A2 (six named AI crawlers allowed, six points) and A3 (sitemap referenced, one of its three points). The field guide covers what each crawler is for. This post is about the file.
Find out what is actually being served
Before editing anything, fetch the live file rather than the one in your repository. A CDN can append to it, and a framework can serve something else at that path without anyone noticing.
curl -sI https://www.example.com/robots.txt
curl -s https://www.example.com/robots.txt | head -n 20
There are four outcomes, and A1 scores each differently.
A 200 with Content-Type: text/plain and recognisable directives is the goal. A1 awards both points.
A 404 scores one point of two. No file means nothing is disallowed, so the site is open by default and A2 treats all six tokens as allowed. It is still sloppier than an empty file. Crawlers get no sitemap reference, so A3 loses its "referenced" point, and nobody can tell whether you decided to be open or never thought about it. A file containing only User-agent: * and an empty Disallow: says the same thing on purpose.
A 200 that is actually HTML scores zero. This happens when a single-page application's catch-all route serves the app shell for any unknown path, or when a CMS returns its "not found" page with a 200. The parser finds no directives, marks the file invalid, and the report advises checking for HTML being served in its place. This is worse than a 404, because the crawler asked and was handed something it could not read.
Any other status also scores zero. Our crawler follows redirects, so what matters is the final status. A 403 on robots.txt is almost always the edge, not the file, and the Cloudflare post covers it.
A typical legacy file
Every line in this composite is common.
User-agent: *
Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /search
Crawl-delay: 10
User-agent: Googlebot
Allow: /
User-agent: GPTBot
User-agent: CCBot
Disallow: /
User-agent: Claude
Disallow: /
Sitemap: http://www.example.com/sitemap_index.xml
Here is what each part does under the Robots Exclusion Protocol (RFC 9309), as our parser applies it.
The wildcard group does not merge
User-agent: * is a fallback for any crawler that has no group of its own. It is not a base that named groups extend. A crawler that finds a group naming it obeys that group and only that group, so every named group below starts from a blank slate.
Crawl-delay is not a control
Crawl-delay is not part of RFC 9309. Our parser ignores it, and not every crawler honours it. Leaving it in is harmless, but it is not rate limiting. If you need to slow a crawler down, do it at the server and confirm the crawlers you want still receive a 200, because check A5 counts a 429 as a refusal.
The Googlebot group opened the back door
User-agent: Googlebot followed by Allow: / was written to make sure Google could crawl everything. It did more than that. Googlebot now obeys only its own group, which has no disallows, so /wp-admin/ and /search are open to it. Whoever wrote the wildcard list assumed it carried over. It did not. A named group carries the same disallow list as the wildcard group, or it carries none.
The training opt-out
GPTBot and CCBot share a group because consecutive User-agent lines accumulate into one. Refusing OpenAI's training crawler is a defensible decision, and the report treats it as one. It costs a single A2 point, because GPTBot is one of the six tokens scored and CCBot is not; if it is deliberate, the report says there is nothing to do. The field guide goes through the OpenAI tokens, and the rest, in detail.
The Claude group refuses more than it meant to
User agent matching is case-insensitive and matches on the prefix, in common implementations including ours. User-agent: Claude therefore matches ClaudeBot, which collects content for training, and also Claude-User, which fetches a page because a person asked Claude about it. The owner meant to opt out of training. The file also refuses every prospect who pastes a URL into Claude. That is two A2 points gone, and the second is the expensive one, because it refuses a person rather than a crawler.
The Sitemap line is load-bearing
Three problems. It uses http while the site serves https. It points at an index file. And it is last.
The Sitemap directive is independent of groups and may appear anywhere, so position does not change its meaning. It does change what gets read. Our crawler fetches the first sitemap declared. A3 awards its "referenced" point for any declaration, but check D4 reads lastmod from the URLs in whichever file that first declaration returns. An index file lists child sitemaps rather than pages, with no per-page lastmod, so D4 sees none. List the file that contains real page entries with real lastmod values first. The index and any chunks can follow.
The rewrite
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml
User-agent: *
Allow: /
Disallow: /wp-admin/
Disallow: /search
Disallow: /checkout/thanks
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Googlebot
User-agent: Bingbot
Allow: /
Disallow: /wp-admin/
Disallow: /search
Disallow: /checkout/thanks
What changed, and why.
The sitemap comes first, over https, and it is the file with page entries. A3 scores this as referenced, and D4 has something to read.
The wildcard group keeps the hygiene list and drops a path that no longer exists and a directive that never did anything. Allow: / is not required, since a group with no rules allows everything, but it makes the intent unambiguous.
Every crawler we have a view on is named in full, never by prefix, and the named group carries exactly the same disallow list as the wildcard group, because it will not inherit it. Our own robots.txt emits one group per token; stacking the User-agent lines as above is equivalent under the REP and shorter. Split a token into its own group when its decision differs. Moving GPTBot, ClaudeBot and Google-Extended into a group with Disallow: / opts out of training while keeping search indexing and user fetches open. That file scores three of six on A2, and the report says which three. Which combination fits which kind of site is a decision by site type, and the field guide gives the cost of each block.
Within a group, the longest matching path wins. Allow: / and Disallow: /search in the same group means /search and everything under it is refused and everything else is permitted. On an exact tie, Allow wins, so a group containing both Disallow: / and Allow: / allows everything. An empty Disallow: also allows everything.
Test the result
curl -sIon the live URL returns 200 with atext/plaincontent type.curl -son the same URL prints your directives, not markup. Check again after each deploy.- The first
SitemapURL returns 200 and contains aurlsetwithlastmodvalues. - Re-run the fetch with
-A "GPTBot"and-A "ClaudeBot"and confirm a 200. This file cannot fix a refusal at the edge; that is check A5, and the Cloudflare post covers the most common source of those refusals. - Run a scan and read the evidence under A1, A2 and A3, which records how many groups and sitemaps parsed and which of the six tokens are blocked.
Our bot page documents what our crawler does with your file, including that it obeys it.
Check your own site
The free scan reads your live robots.txt, parses it under the rules above and reports A1, A2 and A3 with the evidence attached. The three checks are defined under discovery and access on the methodology page.