"AI crawlers" is a group name for at least three different jobs. One kind of bot collects pages to train a model. Another builds a search index so a product can cite sources. A third fetches a single page because a person, right now, asked an assistant to look at it. And one of the six tokens our rubric scores is not a crawler at all but a control switch that never sends a request.
Blocking them as a group treats those jobs as one, and the cost of that is not evenly distributed. This post goes through the six user agent tokens that check A2 tests, GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended, says what each vendor publicly documents it for, and states what refusing each one actually refuses. Where a detail is not clearly documented, we say less rather than guess.
Three jobs and a switch
The vendors have converged on a similar split, and it is worth naming before the tokens.
A training crawl gathers content that may be used to train future models. Blocking it is a decision about whether your content contributes to a model's training data.
A search-index crawl gathers content so that a product can surface and link to your pages when it answers a question. Blocking it is a decision about whether you appear as a source in that product.
A user-triggered fetch happens when a person asks an assistant about a specific page or site, and the assistant retrieves it in response. Blocking it refuses the person, not the company.
A control token is a name you can put in robots.txt that a vendor reads as a preference, without any crawler of that name ever fetching a page.
The six tokens A2 scores
Check A2 is worth six points, the heaviest single check in the rubric, scored proportionally at one point per token. It reads your robots.txt and asks, for each of the six, whether the root path / is allowed. A blanket Disallow: / for all six scores zero.
GPTBot
OpenAI's crawler. OpenAI documents it as the agent that collects content which may be used to train its models, and documents it separately from its search crawler, so that the two decisions can be made independently. Blocking GPTBot opts your site out of that training collection. It does not, per OpenAI's documentation, govern the search crawler below.
OAI-SearchBot
OpenAI's search-index crawler. OpenAI documents it as the agent used to surface and link to websites in its search features, and states that it is not used to gather training data. Blocking it refuses the index crawl those features depend on. A site that wants to opt out of training while remaining a source can block GPTBot and allow OAI-SearchBot; the two tokens exist so that this is possible.
ClaudeBot
Anthropic's crawler, documented as the agent that collects content for training its models. Blocking it is the same category of decision as blocking GPTBot.
Claude-User
Anthropic's user-triggered fetcher. Anthropic documents it as the agent that retrieves a page when a person asks Claude about that page or site. Blocking it does not affect training or indexing. It refuses the one fetch a person asked for, at the moment they asked for it.
This is the token most often blocked by accident. It shares a prefix with ClaudeBot, and a robots.txt group written as User-agent: Claude matches both in common parser implementations, ours included, which match the token as a prefix. Sites that meant to opt out of training end up refusing their own prospects' requests. If you want to block the crawler and keep the fetcher, name each token in full.
PerplexityBot
Perplexity's crawler, documented as the agent that indexes pages for use in its answer product. Blocking it refuses that index crawl. Perplexity also documents a separate user-triggered fetcher, Perplexity-User, which A2 does not score but which the same accidental-prefix problem applies to.
Google-Extended
Not a crawler. Google documents Google-Extended as a standalone product token that publishers can use in robots.txt to manage whether content Google has crawled may be used for training its Gemini models. It does not have its own user agent string; the fetching is done by Googlebot, which is governed by its own rules. Google's documentation also states that the token does not affect a site's inclusion or ranking in Google Search.
Blocking Google-Extended, then, costs nothing in search. It is a training-consent decision and only that. It is in the rubric because the rubric scores whether a site is open to the ecosystem of models and agents as a whole, and a site that has opted its content out of one vendor's training is, for that purpose, one sixth less open. Whether that trade is right for you is a policy question we do not try to answer; the check simply records the choice.
The tokens we name but do not score
Our own robots.txt names thirteen tokens, not six: the six above plus ChatGPT-User, Claude-SearchBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider and meta-externalagent. A2 scores the six the rubric names. We name the rest because naming a token explicitly is the difference between passing by accident and passing on purpose: a permissive * default lets everything in today, and stops doing so the moment someone adds a Disallow to it.
How A2 actually reads your file
A2 does not send a request under these user agents. It parses robots.txt and applies the Robots Exclusion Protocol rules, which are easy to get subtly wrong. The ones that matter:
- User agent matching is case-insensitive and matches on prefix. Consecutive
User-agentlines share one rule group. - The most specific matching group wins, and
*is only a fallback. A named group's rules do not merge with the*group's rules. If you write aUser-agent: GPTBotgroup with a singleDisallow: /private, GPTBot obeys only that group, and inherits nothing from*. - Between
AllowandDisallow, the longest matching path wins. On an exact tie,Allowwins. Disallow:with an empty value permits everything.- No
robots.txtat all means the site is open by default. A1 scores a 404 as partial credit for sloppiness, but A2 scores it as open.
A file that opts out of training, stays in search, and keeps user-triggered fetches working looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
That file scores three of six on A2, and the scan report lists exactly which three tokens are blocked. If that is deliberate, there is nothing to do, and the recommendation text says so. The check exists to make the choice visible, not to make it for you.
Necessary, not sufficient
A robots.txt that allows all six is where the work starts, not where it ends. The file is a request that well-behaved crawlers honour; it does nothing to stop a CDN or a web application firewall from refusing those same user agents at the edge, before robots.txt is ever read.
That is what check A5 is for. The crawler sends real requests to your homepage under the GPTBot, ClaudeBot and PerplexityBot user agents and records the status code and time to first byte for each. Two points if all three receive a 200 rather than a 403 or 429, and one point if the median time to first byte is under 1.5 seconds. When A2 passes and A5 fails, the report says so explicitly: your robots.txt welcomes these agents, and something in front of your origin is refusing them anyway. It is almost always a default the site owner never chose, and the Cloudflare post covers the most common instance.
We have nothing in front of our own origin for exactly this reason. Our robots.txt and our live report show both checks together.
Check your own site
The free scan reports A2 and A5 side by side, so you can see whether your robots.txt says what you think it says and whether your edge agrees with it. Both checks are defined under discovery and access on the methodology page.