ClaudeBot collects content that may be used to train Anthropic's models. Claude-User fetches a page because a person asked Claude about it. Same vendor, same first six letters, different jobs, and a robots.txt group that names only the shared prefix refuses both. So do most copied block lists. The concrete consequence is this: a prospect on your pricing page pastes the URL into Claude, asks a question, and the fetch is refused. Both tokens are among the six that check A2 scores. "Block ClaudeBot, allow Claude-User" scores five of six. "Block Claude" scores four of six and refuses the prospect.
What Anthropic documents
ClaudeBot is Anthropic's crawler. Anthropic documents it as gathering publicly available web content that may be used for training its models, and documents that it respects robots.txt.
Claude-User is the fetcher. Anthropic documents it as the agent that retrieves a page when a person using Claude asks about that page or site, and publishes the token so that site owners can address it in robots.txt separately from the crawler. It is not a crawl. It is the request the person made, carried out.
Claude-SearchBot is a third token, which Anthropic documents as a crawler that navigates the web to improve the quality of search results for Claude users. We name it in our own robots.txt so that our position on it is explicit. A2 does not score it, and this post says no more about it than the documentation does.
Anthropic publishes the three as separate tokens so that the three decisions can be made separately, in the same way OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User. Blocking GPTBot but not OAI-SearchBot is the OpenAI version of this post.
How the accidental block happens
There are two common routes to it.
The first is the prefix. In common implementations, ours included, user agent matching is case-insensitive and matches the token as a prefix. A group written as User-agent: Claude, or User-agent: claude, or User-agent: Claude*, matches ClaudeBot, Claude-User and Claude-SearchBot alike. The owner meant the crawler. The file says all three.
The second is the copied list. A "block all AI" list from a repository or a plugin runs to dozens of tokens, and Claude-User sits in it next to ClaudeBot. The owner reads the vendor name, thinks "training", and pastes the lot. The file now names the fetcher explicitly, so there is no prefix to blame; the token is simply in the wrong list.
Both produce the same A2 result: two of the six tokens blocked, four of six scored, and a report whose evidence lists ClaudeBot, Claude-User as blocked.
What the block does to a prospect
The scenario is ordinary. Someone evaluating your product has your pricing page open. They open Claude, paste the URL, and ask which tier includes the feature they need and whether there is an annual discount. Claude sends Claude-User to fetch the page. The fetch is governed by your robots.txt, which is why Anthropic documents the token at all, and the rubric treats a matching Disallow: / as a refusal: the page is not retrieved.
Whatever follows is assembled without your page. The person asked a specific question about a page they were looking at, and your site declined to answer it. That is the whole cost of the accidental block, and it has nothing to do with training.
Blocking ClaudeBot has no effect on that scenario. It affects only whether the content is collected for training. The two tokens exist so that you can refuse one thing without refusing the other.
The file for each intent
Open to Anthropic entirely:
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
Allow: /
Six of six on A2, provided the other four tokens are also allowed. Consecutive User-agent lines share one group under the REP.
Opt out of training, stay fetchable and indexable:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
Five of six. The evidence reads blocked: [ClaudeBot], and the recommendation text says that if the block is deliberate there is nothing to do.
The accidental one:
User-agent: Claude
Disallow: /
Four of six, and the prospect refused.
Two notes on the fix. In our parser, when several groups match a user agent, the group with the longest matching token wins, so adding an explicit User-agent: Claude-User group with Allow: / alongside the prefix group would restore the fetcher. Other implementations may not resolve the conflict the same way, so the safer fix is to delete the prefix group and name each token in full. And a named group does not inherit the wildcard group's disallows; if you keep a hygiene list under User-agent: *, repeat it in the Claude-User group. robots.txt for the agent era covers both rules.
If the training opt-out is narrower than the whole site, scope it by path. User-agent: ClaudeBot with Allow: / and Disallow: /blog/ keeps the root open, scores six of six, and withholds the archive from training. The longest matching path wins, so the archive is refused and nothing else is.
Why the distinction is the point
The fetcher is a person's request. Of all the traffic that arrives under an AI vendor's name, it is the closest thing to a customer walking in and asking a question. The training crawl is a policy decision about future models. Putting both behind one line means the policy decision refuses the customer, and it does so silently, because nothing reports the refusal to the person and the owner never sees a request declined at the file rather than at the server.
The rubric's proportional scoring is built around this. A site that blocks ClaudeBot keeps five points and a note that the choice is recorded. A site that blocks the prefix loses a second point for a decision it did not make. The six AI crawlers sets out the same logic across all six tokens and four kinds of site.
One boundary. robots.txt governs crawlers that read it. A CDN that refuses the ClaudeBot user agent at the edge returns a 403 before the file is read, and the same edge rule commonly refuses Claude-User too. Check A5 probes under ClaudeBot and records what comes back, and the Cloudflare post covers that layer.
Check your own site
The free scan reads your live robots.txt and lists which of the six tokens it blocks. If Claude-User appears in that list and you only meant the crawler, the fix is the file above. A2 is defined under discovery and access on the methodology page.