A robots.txt is a request, and the crawler reads it only after it has been let through the door. Everything in front of your origin, a CDN, a web application firewall, a bot-management product, a rate limiter, a reverse proxy with opinions about user agents, decides whether the door opens at all, and none of it consults robots.txt first. A site can have a flawless file and still return a 403 to every AI crawler that asks for it.
Check A5 exists to catch that gap. This post is a tour of the layers that cause it: what each looks like from the crawler's side, how it appears in the A5 evidence, and what to change in principle. Cloudflare is the most common instance and has its own post with the dashboard walkthrough; nothing below depends on which vendor you use.
What A5 measures
The scanner fetches your homepage three times, once each under the GPTBot, ClaudeBot and PerplexityBot user agent strings, and records the status code and the time each request took. Two points if all three receive a 200 rather than a 403 or 429. One further point if the median of the three timings is under 1.5 seconds. Three points, rated Low effort, because the fix is nearly always a setting rather than a build.
The evidence block shows the three probes with their status and ttfbMs, the medianTtfbMs, and a flag named edgeBlockingDespiteRobots. The flag is set when any probe failed while your robots.txt permits all six crawlers that A2 tests, or does not exist, which counts as open. It names the contradiction: the file says yes, the edge says no.
The layers, one at a time
Managed bot rulesets
Most firewalls ship a managed ruleset with a category for known bots and, increasingly, a sub-category for AI crawlers, each settable to block, challenge, log or allow. Set to block, it returns 403 on every path, including /robots.txt, so the crawler never reads your directives at all. All three probes show 403 and the flag is true. The change is to find the AI-crawler category and allow the crawlers you want by name, rather than switching the ruleset off wholesale.
Bot-management scores
Bot-management products give each request a likelihood of being automated and let you act on buckets, typically "likely automated" and "definitely automated". A crawler is automated by definition, so it lands in the bottom bucket every time, and a rule that blocks or challenges that bucket refuses every crawler regardless of intent. Many products keep a verified-bots list that bypasses the score, but the list is the vendor's, not yours, and need not include the crawlers your robots.txt allows. This shows as a 403 or a challenge. The change is to exempt the crawlers you want by user agent, and by address range where the crawler's operator publishes one, in a rule that runs before the score is consulted.
Rate limiting tuned for humans
A limit written for people browsing sees a crawler fetching robots.txt, a sitemap, four text files and six pages within a few seconds as an attack, and answers 429. The evidence pattern is often one probe at 200 and the next two at 429, because the limit tripped part way through. The change is to set thresholds to what a polite crawler actually does (ours is documented at /bot: one concurrent request per host, six pages) and to key the limit on the client address rather than the user agent, so that exemptions for named crawlers hold.
JavaScript challenges
A challenge page asks the client to run a script and prove it is a browser. A crawler cannot, so it stops there. The status is usually 503 or 403 with a challenge document in the body, recorded as a plain non-200. The more insidious variant returns 200 with the challenge as the body: A5 passes, because the status is fine, and A6 catches it instead from the challenge markup. The change is to scope challenges to where abuse actually happens, the login form, the checkout, the signup, and never to the whole site.
Geographic rules
A rule that refuses traffic from regions where you do not sell refuses crawlers too, because crawlers run from cloud regions unrelated to where their users live. It shows as a 403 for some crawlers and not others, depending on where each fetched from. The change is to allow verified crawlers before the geographic rule runs, or to use geography to route rather than refuse.
Origin-level filters
None of this is exclusive to a CDN. A reverse proxy or application firewall on your own servers can key on the User-Agent header, and a rule written years ago to stop a scraper by matching on "bot" matches GPTBot, ClaudeBot and PerplexityBot today. Some configurations drop the connection rather than answering; the evidence then shows a status of 0, because no response arrived. The change is to search proxy configuration and framework middleware for anything that reads the user agent, and make the allow list explicit.
The slow origin
The third point is the one people miss. If the median of the three timings is 1,500 ms or over, the point is lost, however healthy the status codes. That is a soft block. An agent working through a task has a budget per fetch shorter than a person's patience, and a page that arrives late is, for its purposes, a page that did not arrive. Our crawler gives up at ten seconds; many agents give up sooner.
The time usually goes to one of three places: a cache keyed on the user agent or bypassed for anything that is not a browser, so bots always hit a cold origin; a serverless origin that the first request has to wake; or a homepage that does real work, a database query or a third-party call, on every request. The median forgives one slow probe; two lose the point.
Test it yourself
Fetch / and /robots.txt under a browser user agent and then under the three strings our crawler sends, and compare status code and time to first byte. Any difference between the first row and the rest is the edge.
for ua in \
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \
"Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
do
for path in / /robots.txt; do
curl -s -o /dev/null -A "$ua" \
-w "%{http_code} %{time_starttransfer}s $path\n" "https://example.com$path"
done
done
A 200 is not the end of the test. Fetch the body under a crawler user agent and read the first few hundred bytes; a challenge page and your homepage both return 200, and only one of them contains your content. Then run the loop twice: a rate limit will often pass the first run and fail the second, which is what a real crawl looks like from its side.
What to change, in principle
Three rules cover every layer above. The rule that admits the crawlers you want must run before any rule that scores, limits, challenges or refuses; ordering is the whole game at the edge. Name the crawlers you want in rather than maintaining a list of the ones you want out; a deny list of tokens is never finished. And test under the crawler's identity after every change, because the dashboard describes what a rule intends and the status code describes what it does.
Check your own site
The free scan runs the three probes and records their status, timing and the edgeBlockingDespiteRobots flag under A5, next to what your robots.txt says under A2. Both checks are defined under discovery and access on the methodology page.