OpenAI documents three user agent tokens, and there are three because they do three different things. GPTBot collects content that may be used to train models. OAI-SearchBot builds the index behind OpenAI's search features and, per its documentation, is not used for training. ChatGPT-User fetches a page because a person asked for it. A site can therefore opt out of training while remaining a citable source and remaining readable on request, and the file that does so is a few lines long. On check A2 it scores five of six. This post shows the file, the score, the two ways the group gets written wrongly, and what each side of the decision costs.
Three tokens, three decisions
GPTBot is the training crawler. OpenAI's documentation says content it fetches may be used to train its generative models, and that a Disallow for GPTBot in robots.txt is honoured. That is the whole scope of the token: whether your pages contribute to training.
OAI-SearchBot is the search-index crawler. OpenAI documents it as the agent used to surface and link to websites in its search features, and states that it is not used to gather training data. Its job is to know your pages exist and what they say, so that an answer can cite them.
ChatGPT-User is the fetcher. OpenAI documents it as acting on behalf of a user, retrieving a page when a person asks about it, and states that it is not used for automatic crawling or for training. It fetches one page because one person asked.
OpenAI publishes separate documentation and separate IP address ranges for each token. That is what makes the three decisions independent, and it is also how a request carrying one of these names can be verified against the published ranges rather than trusted on its user agent string alone.
A2 scores GPTBot and OAI-SearchBot as two of its six tokens. It does not score ChatGPT-User. We name it in our own robots.txt regardless, on the principle that applies to all three: name what you have a view on, in full.
The file
Opt out of training, stay citable, stay fetchable:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: *
Allow: /
Disallow: /admin
Each named group stands alone. Under the Robots Exclusion Protocol a crawler that finds a group naming it obeys only that group, so OAI-SearchBot with a lone Allow: / means everything, including /admin. If you keep a hygiene disallow list, repeat it in every named group; the wildcard group's rules do not carry over. robots.txt for the agent era works through that and the other REP rules line by line.
What this file scores. A2 tests each of its six tokens against the root path. GPTBot is refused. OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended are allowed, because the first has its own permissive group and the other four fall through to the wildcard group, which permits /. Five of six, and the evidence on the report reads allowed: 5, of: 6, blocked: [GPTBot]. The recommendation text says that if the block is deliberate there is nothing to do. It is deliberate, and the report has made it visible rather than penalised it further.
If the opt-out is narrower than the whole site, scope it by path rather than by token:
User-agent: GPTBot
Allow: /
Disallow: /archive/
The root stays open, so A2 scores six of six, and the archive is withheld from training. The longest matching path wins, so anything under /archive/ is refused and everything else is permitted.
The group that catches more, or nothing
There are two ways to write the token wrongly, in opposite directions.
Too wide. In common implementations, ours included, user agent matching is case-insensitive and matches the token as a prefix, so User-agent: GPT matches GPTBot, which may be what you meant. Implementations that also match a token appearing anywhere in the name, and ours does, match ChatGPT-User as well, and now the training opt-out has refused the person. A truncated token is not a shortcut; it is a different rule.
Dead. User-agent: OpenAI matches none of the three, because no token begins with it. The group is inert, the wildcard group applies, and the owner who believes they have opted out of training has not. Neither error is visible in a browser. In the A2 evidence, the first shows GPTBot blocked as intended while the harm lands on a token A2 does not score; the second shows GPTBot absent from the blocked list when you expected it there.
The fix for both is the same: write each token exactly as the vendor documents it, and write one group per decision.
What a publisher gives up, and what it protects
Blocking GPTBot protects a contribution. Content the crawler would have fetched is not used to train OpenAI's models, to the extent OpenAI honours the file, which its documentation says it does. For a site whose text is the product, that is a decision about an asset, and the rubric charges one point for it without arguing.
Blocking OAI-SearchBot gives up being a source. OpenAI documents that crawler as the way sites are surfaced and linked in its search features. A site absent from that index cannot be cited by it, and nothing reports the omission to the person asking; the answer is assembled from what could be indexed. For a publisher, a citation is the mechanism by which an answer sends a reader to you, and blocking the index crawler removes the link.
The two decisions are independent by design. "Not training data, but still a source" is available, and so is "neither", but the second is two decisions written as two groups, not one line that happens to cover both. The mistake we see is a "block all AI" list, copied whole, that puts OAI-SearchBot and ChatGPT-User beside GPTBot when the owner only ever meant the first. The six AI crawlers sets out which combination fits which kind of site, and the publisher row is the one this post belongs to.
One boundary. A Disallow in robots.txt is honoured by crawlers that read the file; it is not a refusal at the network. Check A5 sends a real request under the GPTBot user agent and records the status. A site that disallows GPTBot in robots.txt still returns a 200 to that probe, because the file is a request, not a firewall, and A5 is unaffected. A site whose CDN refuses GPTBot at the edge returns a 403, and the same edge rule usually refuses OAI-SearchBot too, which is the accidental version of this post's subject. The Cloudflare post covers it.
Check your own site
The free scan reads your live robots.txt and reports exactly which of the six tokens it blocks, so you can confirm the file names GPTBot and only GPTBot if that is what you meant. A2 is defined under discovery and access on the methodology page.