A challenge page asks the client to prove something before it gets the content: that it can run JavaScript, that it can solve a puzzle, that it is a person. An agent can do none of those things, and it does not try. It records whatever the challenge served and moves on. What a challenge costs you is therefore decided entirely by where you put it. On a login form it costs nothing an agent needed. On every request it removes the site.
Check A6 in the rubric, "No hard interstitial", is worth two points and rated Medium effort. This post covers how the scanner detects a challenge, the difference between a challenge on every request and one scoped to the places abuse actually happens, and what to change in each case.
How the scanner sees a challenge
A6 looks at the homepage only. If a headless render was captured it examines the rendered HTML, so that a challenge injected by a script counts; otherwise it examines the raw response body. It then tests the markup against a short list of patterns and records each one that matches under a markers field in the evidence:
| Marker | Fires on |
|---|---|
| Cloudflare challenge | cf-challenge, cf_chl_ or challenge-platform in the markup |
| CAPTCHA | g-recaptcha, recaptcha/api.js or hcaptcha |
| Cookie wall | the ids and class names of common consent-banner libraries |
| Interstitial page | the text "Just a moment..." |
| JS-required interstitial | "Please enable JavaScript to continue" and its variants |
No markers scores the full two points. If the only marker is the cookie wall, the check scores one, because a consent dialog hides part of a page rather than all of it. Any other marker scores zero. The recommendation attached to a zero states the reason plainly: an agent cannot dismiss a consent dialog or solve a challenge, so whatever sits behind it is unreachable.
The detection is deliberately conservative. It fires on the presence of the markup, not on whether the markup happens to be blocking anything at that moment. A reCAPTCHA script loaded in a shared layout for a newsletter form in the footer sets the CAPTCHA marker on every page, homepage included. Whether that is "really" a challenge is a fair question, and the answer does not change the fix: keep the script off pages that do not carry the form.
Two ways a challenge reaches an agent
A challenge can be served in two shapes, and the rubric catches them in two different places.
The first returns a non-200 status, usually 403 or 503, with the challenge document as the body. That is caught by A5, which fetches the homepage under three crawler user agents and records the status codes. Your robots.txt is perfect and you are still blocking agents walks through that layer, and the Cloudflare post covers the most common vendor.
The second returns 200 with the challenge as the body. The status is healthy, A5 passes, and the page the agent received is a form asking it to prove it is a browser. A6 exists for this shape. It reads what came back rather than the code it came back with.
Both shapes have the same downstream effect, and it is larger than two points. Every check in pillars B, C and D is measured on the page the crawler received. If that page is a challenge, the schema it finds is the challenge's schema, the forms it audits are the challenge's forms, and the headings it counts are the challenge's headings. The report describes a page you did not write.
Challenge on every request
A sitewide challenge comes from one of three places. A "under attack" or equivalent mode that challenges every visitor, usually switched on during an incident and never switched off. A firewall rule that challenges every path for any request the bot-management product scores as automated, which a crawler is by definition. Or a managed challenge applied to the whole zone as a precaution, which was the cheapest rule to write at the time.
Each of these is a decision to treat the whole site as the thing worth protecting. For a login page, a checkout or a signup form that is true. For the homepage, the pricing page, the documentation and the product catalogue it is not: those pages are public precisely so that they can be read, and the readers now include software acting for a buyer. Cloudflare's own product documentation at developers.cloudflare.com describes challenges as a per-rule action, which is the right mental model: a challenge is something a rule does to a request, and the rule decides which requests.
There is a second, quieter form of sitewide challenge: a CAPTCHA widget loaded on every page. Score-based variants that run invisibly are the usual cause, because they are documented as working best when present everywhere. To a plain fetch the widget is just a script tag, and the page content is still there, but the marker fires all the same, and the widget's presence tends to accompany server-side rules that refuse submissions without a token. If you use one, load it on the pages that have a form to protect, not in the layout.
Challenges scoped to where they belong
Abuse clusters in four places: credential stuffing on login, card testing on checkout, fake accounts on signup, and spam on contact forms. A challenge on each of those is a reasonable defence and costs an agent little, because an agent reading a site to answer a question never needs to pass the login page.
Two refinements make the scoped version cost even less.
Challenge the submission, not the page. A rule that fires on the POST to the contact endpoint, rather than on the GET of the contact page, lets an agent read the form, which is what check C1 in the actionability pillar audits, while still refusing to accept a submission without proof. The form is visible, the fields are labelled, the submit button is real, and the abuse is still stopped at the moment it would have happened.
Decide checkout deliberately. Check C5 for commerce sites looks for a cart or checkout path that returns 200 without a login, because an agent acting for a buyer will attempt to reach it. A challenge on checkout is a trade: it costs the agent the transaction in exchange for protection against card testing. That can be the right trade. It should be one you made, not one inherited from a rule written for a different problem.
See it yourself
Fetch your homepage under a crawler user agent and search for the same patterns the scanner uses:
curl -sL -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://example.com/ \
| grep -oiE 'cf-challenge|cf_chl_|challenge-platform|g-recaptcha|recaptcha/api\.js|hcaptcha|Just a moment\.\.\.' \
| sort | uniq -c
Empty output means no markers in the raw response. Because the scanner prefers the rendered HTML when it has one, also open the page in a browser, wait for it to settle, and search the live DOM in the inspector for the same strings. A widget injected by a tag manager appears only there.
Then run the same fetch against /pricing, /docs and a product page. A6 is scored on the homepage, but an agent does not stop at the homepage, and a challenge on the pages it goes to next costs the same in practice whatever the score says.
What to change, in order
- Turn off any sitewide challenge mode, and if you need one during an incident, time-box it. It is the single most expensive setting in this post and the quickest to revert.
- Put an allow rule for the crawlers you want above every rule that challenges on bot score. Rule order is the whole game at the edge, as the robots.txt post explains.
- Move CAPTCHA and challenge scripts out of the shared layout and onto the pages that carry the protected form.
- Where the platform supports it, attach the challenge to the form submission rather than to the page.
- Leave login, signup, checkout and contact submission challenged, and record that as a decision so the next person to read the rules knows it was one.
Vendor defaults and dashboard labels move; the mechanism does not. After each change, fetch under a crawler user agent and read what came back, because a 200 that contains a challenge is a 200 you do not want.
Check your own site
The free scan records the markers it found under A6 and the three status codes under A5 side by side, so you can tell which shape of challenge you have. A6 is defined in full under discovery and access, and our own crawler's behaviour, including the user agent it sends, is published at /bot.