Google-Extended is a name you can put in robots.txt that no crawler ever answers to. Google documents it as a standalone product token: a switch that tells Google whether content its crawlers have fetched from your site may be used to train and ground its Gemini models. It has no user agent string of its own; the fetching is done by Googlebot, under Googlebot's rules. And, per the same documentation, it has no effect on whether your pages are included in Google Search or how they rank. Most of the trouble people have with it comes from expecting it to be something else.
This post takes that one token in depth. The field guide covers all six tokens that check A2 scores in a paragraph each; this is the long version for the one most often set for the wrong reason.
A control token is not a crawler
In an ordinary robots.txt group, the name after User-agent: is the string a crawler sends when it fetches. GPTBot sends GPTBot; ClaudeBot sends ClaudeBot. When either requests a page it looks for the group matching its own name and obeys it. The name in the file and the name on the wire are the same thing.
Google-Extended breaks that link on purpose. No request ever arrives carrying it. Googlebot fetches the page, and separately Google reads your robots.txt for a group named Google-Extended and treats a Disallow there as an instruction about what may be done with the content afterwards. Google's documentation says as much: the token is used in a control capacity, and crawling is done under existing Google user agent strings.
The practical consequence is that you cannot see Google-Extended in your logs, cannot rate-limit it, cannot block it at the edge by accident, and cannot test it with a curl loop the way you can test a real crawler. It exists only in the file, and the file is the only place to get it right.
What it switches off, and what it leaves alone
| Question | Governed by Google-Extended? |
|---|---|
| Whether Googlebot crawls your pages | No. Googlebot's own group governs that. |
| Whether pages are included in Google Search | No. Google documents no effect on inclusion. |
| How pages rank in Google Search | No. Google documents no effect on ranking. |
| Search features built on the index | No. Google-Extended does not control Search features. |
| Whether crawled content trains future Gemini models | Yes. |
| Whether crawled content is used to ground those models | Yes, per Google's documentation. |
The first four rows are controlled by mechanisms you already know, Googlebot's own robots.txt group and the indexing and snippet controls Google documents for Search. Google-Extended is a separate lever attached to a separate decision.
Why it is misread
The two mistakes are mirror images.
The first is believing that blocking Google-Extended protects you from something it does not. Someone who wants to stay out of a Search feature, or out of any Google surface that quotes their pages, adds the disallow and believes the job is done. It is not, because the token never controlled those features. If the goal is a change on the Search side, the controls for that are on the Search side, and Google documents them separately.
The second is believing that allowing Google-Extended costs search ranking, or that blocking it is safe because Googlebot will still visit. The first half is wrong by Google's own statement. The second half is true but beside the point: the decision being made is whether your content contributes to Gemini training and grounding, not whether Googlebot calls. That may well be the decision you want to make. Make it knowing which one it is.
Three misconfigurations
Blocking Googlebot when you meant Google-Extended
User agent matching in robots.txt is case-insensitive and matches on prefix, in our parser and in most others. A group written as User-agent: Google matches Googlebot, Google-Extended and every other Google token. Intending to opt out of training, the author has opted out of Search.
# Intended: opt out of Gemini training. Actual: also blocks Googlebot.
User-agent: Google
Disallow: /
Write the token in full, every time.
Blocking Google-Extended and expecting a Search feature to change
Here the directive is correct and the expectation is not. Blocking Google-Extended does one thing, and Google's documentation is explicit that Search features are not it. If you want a Search-side change you need a Search-side control, and you may or may not also want Google-Extended blocked. The two are independent, and the file should reflect whichever combination you actually chose.
Leaving it out and assuming the default covers it
If your file has a permissive * group and nothing else, Google-Extended is allowed, and the * group is doing that work. The problem is what happens when the file grows. Under the Robots Exclusion Protocol a crawler that finds a group matching its own name obeys only that group; * is a fallback, not a base layer that named groups inherit from. So a file that blocks everyone by default and names Googlebot as the exception sends Google-Extended to the * group, where it is blocked.
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
# Google-Extended is not Googlebot. It falls to * and is blocked.
That may be exactly what you want. If so, it deserves a group of its own, so the next person to edit the file can see that it was chosen rather than inherited. The robots.txt rewrite goes through the grouping rules in full.
What A2 records
Check A2 is worth six points, scored proportionally across six tokens: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended. Each is one point. The check parses robots.txt and asks, for each name, whether the root path / is allowed; it sends no requests, which for Google-Extended is the only possible approach. Blocking it costs one point, and the report lists the token by name as blocked. A file blocking all six scores zero. No robots.txt at all is scored as open.
The rubric records the choice and does not judge it. The recommendation on a partial score says so: if the block is deliberate, there is nothing to do. The token is in the rubric at all because A2 measures how open a site is to the ecosystem of models and agents as a whole, and a site that has opted its content out of one vendor's training and grounding is, for that purpose, one sixth less open. Whether that is a price worth paying is a policy question, and the answer differs between a publisher whose text is the product and a company whose site exists to be found. The check makes the answer visible; it does not supply one. Six crawlers, six jobs sets out the same argument for the other five tokens.
Writing it down on purpose
Our own robots.txt names Google-Extended explicitly, in its own group, alongside Googlebot and the other crawlers we name, each with the same allow and the same short disallow list. Naming a token is the difference between passing by accident and passing on purpose: a permissive * group admits it today and stops the day someone adds a Disallow to *.
A file that keeps Search entirely as it was and opts out of Gemini training and grounding looks like this:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
The reverse, allowing both, is two Allow groups. Either is a decision. The file to avoid is the one that makes the decision by accident, in either direction.
Check your own site
The free scan parses your robots.txt and shows which of the six A2 tokens are allowed, with Google-Extended listed by name and the evidence attached. The check is defined under discovery and access on the methodology page.