The framework decides what is in your HTML. The host decides whether a crawler receives it. Every platform guide on this blog so far has been about the first layer; this one is about the second, the network a request crosses before it reaches the code you wrote, and what four common hosts run there by default. It closes with what we chose for our own site and why, stated as our decision rather than as advice about any vendor.
The two checks, and the gap between them
Two checks in the rubric look at access, and they are separate on purpose. A2, six points, parses robots.txt and awards credit in proportion to how many of six named AI crawlers it permits. A5, three points, ignores robots.txt and fetches the homepage three times, as GPTBot, ClaudeBot and PerplexityBot, awarding two points if all three receive a 200 and one more if the median time to first byte is under 1.5 seconds.
When A2 is full and A5 is not, the report sets a flag in the A5 evidence named edgeBlockingDespiteRobots: the file says yes, something in front of it says no. That something is the subject of this post. Your robots.txt is perfect and you are still blocking agents is the vendor-neutral tour of the layers that can do it; here the question is which of them each host runs without being asked.
What in front of the origin means
On every host below, a request is answered first by the vendor's network: TLS is terminated there, a cache may answer there, and any firewall, bot-management or rate-limiting product the vendor offers runs there, before your code sees the request. There is no such thing as a request that bypasses that layer. The difference between hosts is what runs in it by default, and what you can find out from the dashboard versus what you can only find out with curl.
Everything below is limited to what each vendor documents as of this writing. Product names and defaults move; the documentation root is linked for each so the current state can be checked.
Vercel
A site on Vercel is served entirely from Vercel's edge network; there is no origin of yours to point at directly. Vercel's documentation describes a firewall with platform-level DDoS mitigation that is always on, custom rules and managed rulesets that you configure, and a challenge mode that, when you switch it on, presents a browser challenge to every visitor. A challenge is a closed door to a crawler, so that mode is one to enable for an incident and switch off afterwards, not to leave on. The documentation also describes bot-management rules and a list of verified bots; whether any of that is active on a given project depends on when the project was created and what has been toggled, so read the current Vercel documentation and then fetch your homepage as a crawler rather than reasoning from the dashboard.
One default worth knowing is deployment protection. Vercel's documentation describes authentication on preview deployments as enabled by default, which returns a 401 to anyone without a session, crawlers included. It does not apply to a production domain unless you enable it there, but a domain pointed at the wrong deployment is a complete block that looks, from outside, like a decision.
Netlify
Netlify likewise serves from its own edge network. Its documentation describes firewall traffic rules, rate limiting and a web application firewall as features you configure, some of them tied to particular plans, and describes site-level password and access control that, if enabled, answers with a 401. We are not aware of a documented default that refuses AI crawlers by user agent as of this writing; that is a statement about the Netlify documentation, not about your site, and the curl test settles it in seconds.
Fly.io
Fly.io is a different shape. Its proxy terminates TLS and routes the request to a machine running your container; there is no managed firewall, bot-management or WAF product in the request path by default as of this writing. If a crawler receives a 403 on Fly, the rule that produced it is in your own application or in a reverse proxy you installed, which makes it easy to find.
The default that matters on Fly is a different one. Its documentation describes machines that stop when idle and start on demand. The first request after an idle period waits for the start, and A5 measures time to first byte with a 1.5-second budget on the median of three probes. A site that returns three 200s and loses the third point is the pattern to look for; the Fly.io documentation describes the settings that keep a minimum number of machines running.
Cloudflare Pages
A site on Cloudflare Pages is a zone on Cloudflare's network, and every zone-level security setting applies to it, including the one that costs the most points on this rubric. As covered in Cloudflare blocks AI crawlers by default, Cloudflare's documentation describes a setting that blocks known AI crawlers, on by default for zones created after mid-2025, running at the edge before robots.txt is read. A Pages project inherits it like any other zone. The Cloudflare Pages documentation and the broader Cloudflare developer documentation cover the per-crawler controls that open specific ones.
The default that costs the most
Of everything above, the AI-crawler block is the expensive one, for three reasons. It is on without anyone choosing it. It runs before robots.txt, so the file that would have said allowed is never read; A5 loses two points, and if the block covers /robots.txt as well, A2 falls too, because the engine cannot parse a file it was refused. And nothing in a browser shows it, so it survives every visual check a team does before launch.
The other defaults in this post cost a point or a preview URL. That one costs the front door.
What we chose, and why
We sell agent readiness, and a prospect's first move after reading a cold email is to scan us, so our own site is built to score at or near 100 on the rubric it publishes. That constraint decided the hosting.
The domain's nameservers point at Vercel, which serves the web application, and they were pointed there specifically so that no proxy of ours sits in front of it. We have enabled no firewall rule that keys on the user agent and no challenge mode. The scan worker, the part that crawls other sites, runs as a separate application on Fly.io in a single region, and it is not in the request path for the website at all, so its cold starts affect how quickly a scan begins rather than how quickly a crawler gets a page. The result is public in our live report, including anything we have not yet fixed.
That is our decision for our constraint. A publisher whose value is the text on the page may want the block on; a shop or a software company almost certainly does not. The point is to choose it, then test it.
Test it in one loop
for ua in \
"Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
"Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
"Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
do
curl -s -o /dev/null -A "$ua" -w '%{http_code} %{time_starttransfer}s\n' https://example.com/
done
Three 200s under 1.5 seconds and the hosting layer is not your problem. Anything else, and the section above for your host says where to look. Run it twice; a rate limit or a cold start often shows only on the second pass.
Check your own site
The free scan runs the same three probes and records their status, timing and the edgeBlockingDespiteRobots flag under A5, beside what your robots.txt says under A2. Both are defined under discovery and access on the methodology page, and our crawler page describes exactly what the scanner fetches.