The field guide lists the six user agent tokens check A2 scores and what each vendor documents them for. This post is the decision that follows from it. For four common kinds of site, it says which of the six a reasonable owner would allow, what each combination scores, and what each block actually gives up. The short version is that only one of the four has a strong reason to block anything, and even that one should not block all six.
One point per token
A2 is worth six points, scored proportionally. The engine parses robots.txt, asks for each token whether the root path / is allowed, and awards one point per token permitted: six of six scores six, three of six scores three, none scores zero. It sends no requests under these names; that is check A5, which tests whether the edge agrees with the file. The six tokens do three jobs.
| Job | Tokens A2 scores | What a block refuses |
|---|---|---|
| Training crawl | GPTBot, ClaudeBot, Google-Extended | Your content's contribution to a vendor's future models |
| Search index | OAI-SearchBot, PerplexityBot | Being fetched, indexed and cited as a source in that product |
| Fetch on a person's behalf | Claude-User | The one page a person just asked about |
Google-Extended is a control token rather than a crawler. Google documents it as governing whether crawled content may be used for training its models, and states that it does not affect a site's inclusion in Search. For decision purposes it belongs in the training row.
The matrix
| Site type | Allow | Block | A2 |
|---|---|---|---|
| Commerce | All six | None | 6/6 |
| SaaS | All six | None | 6/6 |
| Publisher | OAI-SearchBot, PerplexityBot, Claude-User | GPTBot, ClaudeBot, Google-Extended | 3/6 |
| Services firm | All six | None | 6/6 |
Three rows are identical, and that is the finding. For most businesses the text on the site is not the asset; it exists to be found, repeated and acted on. The exception is the site whose text is the product.
Commerce
A shop's product descriptions were written to be repeated. A training crawl contributes the copy you wanted circulated. A search-index crawl is how an answer to "which of these should I buy" can point at your product page rather than someone else's. And a block on Claude-User refuses the shopper who pasted your product URL and asked about sizing or delivery. There is no row of the table where a shop gains from a block.
If a shop has editorial content it does value, buying guides or a magazine section, the right granularity is the path, not the token. User-agent: GPTBot with Allow: / and Disallow: /guides/ withholds the guides from training and leaves the catalogue open. A2 tests the root path, so a path-scoped opt-out keeps the point, and more importantly it says what you meant.
SaaS
Documentation, pricing and a changelog are the pages an assistant fetches when a person asks whether a product does something, or how. Allowing the training crawlers means that copy contributes to what future models are trained on; allowing the index crawlers means the docs can be cited; allowing Claude-User means a prospect can paste the pricing page and ask about it. Some SaaS owners prefer to keep docs out of training on principle. That variant blocks GPTBot, ClaudeBot and Google-Extended, scores three of six, and is a deliberate choice the report records as one.
Publisher
The only row with a real trade-off. When the text is the product, opting out of training is a decision about an asset, and the rubric charges three points for it without arguing. The question is what to do with the other three tokens.
OAI-SearchBot exists so that a site can be a cited source without being training data; OpenAI documents the two crawlers separately for exactly that reason. Blocking it removes you from the index that citations come from. PerplexityBot is the same decision for a different product, and some publishers refuse it; that variant scores two of six. Claude-User is a reader. A person looking at one of your articles has asked for a summary or a clarification, and the fetch is that request. Blocking it refuses the reader, not the vendor. The field guide covers the OpenAI tokens in detail, and our own robots.txt shows every token named in full.
Services firm
A law firm, an agency, a clinic or a consultancy has little in its copy worth withholding from training and a great deal to lose from not being read. The pages that matter answer "who are these people, what do they do, how do I contact them", and the risk runs the other way: an assistant asked about you answers from a directory listing or a review site, because your own site refused the fetch. Allow all six.
Variants and what they score
| Intent | Blocked tokens | A2 |
|---|---|---|
| Open to all | None | 6/6 |
| Opt out of one vendor's training | GPTBot | 5/6 |
| Opt out of training everywhere | GPTBot, ClaudeBot, Google-Extended | 3/6 |
| Opt out of training and of answer engines | GPTBot, ClaudeBot, Google-Extended, PerplexityBot | 2/6 |
| A copied "block all AI" list | All six | 0/6 |
Every row is a coherent intent except the last. The report lists exactly which tokens are blocked and, where the block is deliberate, says there is nothing to do. The check exists to make the decision visible, not to make it for you.
What each block actually forgoes
Blocking a training crawl forgoes a contribution. Your content is left out of what that vendor's future models are trained on, to the extent the vendor honours the file. It does not change whether an assistant can read your site today, and it is the block with the clearest case for it.
Blocking a search-index crawl forgoes being a source. A citation is a link, and an answer that could have pointed at your page points elsewhere. Nothing reports the omission to the person; the answer is assembled from what could be fetched.
Blocking a fetch on a person's behalf forgoes the person. They asked about your page, at that moment, with intent, and the fetch was refused. This is where the owner's intent and the file's effect diverge most, because nobody writes "refuse my prospects" in a robots.txt on purpose. The field guide sets out the two Anthropic tokens side by side.
Why "block them all" is almost never what was meant
Block lists circulate as a unit. A list of forty tokens is copied from a repository or switched on in a plugin, and the owner reads it as "do not train on my content". The file it produces says "do not train on my content, do not cite my content, and do not read my pricing page for the customer who asked". A2's zero for a blanket block is the rubric noticing that three jobs were treated as one.
The fix is not to allow everything. It is to write the file per token, each named in full, carrying the decision you actually made. robots.txt for the agent era is the line-by-line rewrite. And the file is necessary, not sufficient: a CDN that refuses these user agents at the edge overrides all of it, which is what check A5 and the Cloudflare post are about.
Check your own site
The free scan parses your live robots.txt and lists which of the six tokens it blocks, so you can compare the file against the row of the matrix you meant to be in. A2 is defined under discovery and access on the methodology page.