The most common serious finding in a scan is one the site owner never chose. Their robots.txt welcomes every AI crawler by name. Their edge returns a 403 to all of them. The two facts sit side by side in the report, and the second one wins, because the block happens at the CDN before the crawler ever asks for robots.txt.
In almost every case the CDN is Cloudflare, and the block is a default. This article explains what the default does, how to see it from your own terminal in about thirty seconds, and precisely which settings to change if you decide you want agents to reach you. It also covers the case where you do not, because that is a legitimate choice, as long as it is a choice.
What actually happens to the request
When a site sits behind Cloudflare, every request lands at a Cloudflare edge node first. The node decides whether to pass the request to your origin, serve it from cache, challenge it, or refuse it. That decision is made from the request itself: the IP, the TLS fingerprint, the headers, and above all the User-Agent.
Cloudflare maintains a list of known AI crawlers and, since mid-2025, blocks them by default on new zones. Existing zones were offered the same block as a one-click toggle, and a great many owners turned it on during the period when "block the AI bots" sounded like free hygiene. Either way the rule runs at the edge, in front of everything you control.
That includes robots.txt. A polite crawler's first request to a new host is GET /robots.txt. If the edge refuses that request, the crawler never sees your Allow: /. It sees a 403, or a challenge page that returns 403 with a cf-mitigated: challenge header, and it leaves. Your carefully written directives are correct and unread.
This is why the two checks in the rubric that look at access are separate. A2 parses robots.txt and awards points for each of six named crawlers it permits. A5 ignores robots.txt entirely and simply fetches your homepage three times, as GPTBot, as ClaudeBot and as PerplexityBot, and records what comes back. A site can score full marks on A2 and zero on A5, and when it does, the evidence block on the report says so in plain words: the block is happening at your CDN or WAF, before robots.txt is ever read.
Why it is the default
Cloudflare's position is that AI companies have been taking content without permission or payment, and that the honest default for a publisher is closed until a deal is struck. For a newspaper, a stock-photo library or a forum whose entire value is the text on the page, that is a reasonable place to start. Blocking the training crawlers protects an asset.
Most businesses are not publishers. A software company, a shop, a law firm or a clinic wants to be found, described accurately and, increasingly, acted on. For them the block does not protect an asset; it removes them from the set of sites an assistant can use to answer a question or complete a task. The assistant does not report the 403 to the person who asked. It answers from whatever it could read, which is a competitor.
The default is therefore right for some sites and wrong for others, and the problem is not the default itself. It is that the setting was made for you, at the infrastructure layer, and nothing in a browser will ever show it.
How to check in thirty seconds
You need a terminal and curl. Fetch your homepage as a normal browser, then as each of the three crawlers the scanner uses, and compare the status codes.
for ua in \
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \
"Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
do
printf '%-14s ' "$(echo "$ua" | grep -oE 'GPTBot|ClaudeBot|PerplexityBot' || echo Browser)"
curl -s -o /dev/null -w '%{http_code} %{time_starttransfer}s\n' -A "$ua" https://example.com/
done
A healthy result is four 200s. The signature of an edge block is a 200 for the browser and 403 for the rest. A 429 means rate limiting is catching the crawlers; a 503 with a challenge page in the body means the request is being asked to run JavaScript, which a crawler cannot do. All three are the same finding for practical purposes: the crawler does not get the page.
Run the same loop against /robots.txt. If that returns 403 to the crawlers too, you have confirmed that they never read your directives at all.
If you would rather not use a terminal, the free scan does exactly this and keeps the evidence: the three probes, their status codes and their time to first byte are recorded under A5 on the report.
Deciding what you actually want
Before changing anything, decide which crawlers you want and why. There are three kinds, and the distinction matters more than the vendor.
Training crawlers collect text to train models. GPTBot, Google-Extended (a directive rather than a crawler, but it controls the same thing) and CCBot are the main ones. Blocking these keeps your content out of future training runs. It has no effect on whether an assistant can read your site today.
Search and retrieval crawlers index pages so an assistant can cite them in answers. OAI-SearchBot and Claude-SearchBot are examples. Blocking these removes you from the answers, which is the opposite of what most businesses want.
User-triggered fetches happen when a person asks an assistant to look at a specific page. ChatGPT-User, Claude-User and Perplexity-User carry the request. Blocking these means that when a prospect pastes your pricing page into an assistant and asks a question, the assistant cannot read it.
You can allow the second and third kinds while refusing the first. Cloudflare's per-crawler controls make that possible, and the field guide to AI user agents sets out what each token is for. The rubric's A2 is scored proportionally across six agents for the same reason: a site that opts out of training but stays reachable for search and user fetches has made a defensible choice and keeps most of the points. A blanket block of all six scores zero because it is almost never what the owner meant.
The settings to change
Names in the Cloudflare dashboard move around, so this is described by function rather than by exact label. There are four places a crawler can be refused, and you need to look at all four.
The AI crawler block. Under Security, then Bots, there is a toggle that blocks known AI crawlers on every page. On zones created after mid-2025 it is on. Turn it off if you want agents in, or leave it on and use the per-crawler controls below to open specific ones.
Per-crawler controls. The AI crawl control panel lists each crawler Cloudflare recognises with an allow or block per row, and shows how often each has been refused, which is a useful measure of how much traffic the block has been costing. Set the retrieval and user-fetch crawlers to allow. Decide on the training crawlers separately.
Bot Fight Mode and Super Bot Fight Mode. These challenge or block traffic that looks automated. Crawlers on Cloudflare's verified list are meant to pass, but a "definitely automated: block" setting still catches the ones it does not recognise, and the verified list is not the same as your allow list. If you use these, confirm with the curl loop that the crawlers you allowed actually get through.
WAF custom rules and rate limits. Search your custom rules for anything matching on
http.user_agentor on the bot score, and check whether a rate-limiting rule is set low enough to catch a crawler that fetches six pages in a few seconds. A rule written two years ago to stop a scraper is often the thing refusing ClaudeBot today.
There is also a managed robots.txt feature that appends directives to your file. It is worth knowing about because it can make your robots.txt say something you did not write; check the live file, not the one in your repository.
After each change, run the curl loop again. Cloudflare applies rule changes within seconds, so there is no need to wait. When the four requests return 200, run a fresh scan so the improvement is measured rather than assumed.
What it costs on the rubric, and what it costs in practice
The rubric charges two points for the block, under A5, because the rubric measures what a crawl can prove and the block is a single fact: three probes, three refusals. If the block also hides your robots.txt, A2 falls too, and any check that depends on fetching pages under a crawler's identity can only report what it was allowed to see.
Two points understates it. Every other check in the rubric describes how well an agent can use your site once it is inside. This one decides whether it gets in. A site with excellent structured data, server-rendered pages and a clean form layer that returns 403 to every crawler has done all of that work for nobody.
The reason we lead cold outreach with this finding is that it reads as news rather than criticism. Nobody chose it, nobody can see it from a browser, and it is fixed in a few minutes by someone who already has dashboard access.
Not only Cloudflare
Other edges have equivalent rules. Most large CDNs and hosting platforms now offer a managed bot ruleset with an AI crawler category, and some have begun defaulting it on for new projects. The mechanism is identical, and so is the test: fetch as the crawler, read the status code, and believe that over anything the dashboard says. Cloudflare's own documentation for its bot products is at developers.cloudflare.com, and a general tour of every layer that can refuse a crawler, vendor by vendor, follows later this week. Our own site runs with nothing in front of the origin at all, which is a deliberate choice explained on our live report, not a recommendation for everyone.
Check your own site
Run the curl loop above, or run the free scan and read the A5 evidence. Both take under a minute. If the probes return 403 and you want them to return 200, the settings above are the whole fix. The detail of how A2 and A5 are scored is on the methodology page.