# AgentFriendlyRank — full reference > AI agents are already visiting your website. We audit whether they can actually read, navigate and buy from it, then fix what they can't. Operated by Socio360, a sole proprietorship registered in Karnataka, India. Contact: hello@agentfriendlyrank.com --- ## The rubric (v1.0.1) Four pillars worth 25 points each, 100 in total. Every check is automatable from an unauthenticated crawl and returns captured evidence rather than an opinion. ### Pillar A — Discovery & access (25 points) _Can agents find and reach the content at all?_ - **A1 robots.txt exists and parses** (2 pts, fix effort: Low). Returns 200 with valid directives. A 404 scores partial credit — the site is open by default, but the omission is sloppy. - **A2 Major AI crawlers allowed** (6 pts, fix effort: Low). Allow/deny tested for GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended. Scored proportionally to how many are permitted. A blanket Disallow for all six scores zero. - **A3 XML sitemap valid and referenced** (3 pts, fix effort: Low). One point for existing, one for parsing with at least one URL, one for being referenced from robots.txt. - **A4 llms.txt present** (4 pts, fix effort: Low). An llms.txt at the root with three or more described links scores three points; an llms-full.txt adds one. An empty or boilerplate file scores one. - **A5 Bot user-agent HTTP health** (3 pts, fix effort: Low). All three bot user agents receive a 200 rather than a 403 or 429, plus one point for a median TTFB under 1.5 seconds. This is where edge and CDN blocking surfaces, regardless of what robots.txt says. - **A6 No hard interstitial** (2 pts, fix effort: Medium). The rendered page is not dominated by a CAPTCHA, challenge screen or cookie wall — detected from known challenge markup and from any overlay covering more than half the viewport. - **A7 Server-rendered content parity** (5 pts, fix effort: High). Visible text in the raw HTML compared against the fully rendered DOM. A ratio at or above 0.8 scores full marks; below 0.5 scores zero, because an agent using a plain fetch sees essentially nothing. ### Pillar B — Understanding (25 points) _Once fetched, can a machine parse what this site is and sells?_ - **B1 JSON-LD present and valid** (5 pts, fix effort: Medium). At least one parseable application/ld+json block on the homepage, and on half or more of the crawled pages. Invalid JSON scores zero for that page. - **B2 Correct schema types for the site type** (5 pts, fix effort: Medium). Organization or WebSite sitewide, plus type-specific markup: Product and Offer with price and availability for commerce; Article, FAQPage or SoftwareApplication for SaaS and content sites. - **B3 Semantic HTML structure** (4 pts, fix effort: Medium). Exactly one h1 per page, no skipped heading levels, and real landmarks — main, nav and footer, or their ARIA equivalents. - **B4 Meta and Open Graph** (3 pts, fix effort: Low). A unique title of 15–70 characters, a meta description of 50–160, and og:title, og:description and og:image all present. - **B5 Clean text ratio** (3 pts, fix effort: Medium). Visible text as a proportion of HTML bytes, excluding the contents of script and style elements. At or above 0.10 scores full marks, 0.05–0.10 scores two. This penalises div soup and megabyte pages. Script contents are excluded deliberately: server-rendering frameworks inline a serialised payload that duplicates the page text, and counting it would penalise the server-rendered architecture check A7 rewards. - **B6 Alt text coverage** (2 pts, fix effort: Low). Content images — icons under 50px excluded — carrying non-empty alt text. 80% or better scores full marks. - **B7 Canonical and duplication hygiene** (3 pts, fix effort: Low). A self-consistent rel=canonical, plus www/non-www and http/https all resolving to one canonical host. ### Pillar C — Actionability (25 points) _Can an agent actually do things — navigate, search, fill forms, transact?_ - **C1 Form usability** (5 pts, fix effort: Medium). Every input carries a label or aria-label, input types are correct (email, tel, number), and submit is a real button. A site with no forms at all scores partial credit. - **C2 Navigable link graph** (4 pts, fix effort: Medium). Navigation uses real anchor elements rather than click-handled divs, fewer than 5% of links use contextless text like 'click here', and there are breadcrumbs or a clear URL hierarchy. - **C3 Interactive element semantics** (4 pts, fix effort: Medium). Buttons are button elements rather than clickable divs, interactive elements are focusable with sane tab order, and no keyboard traps appear under render. - **C4 Machine endpoints and agent manifests** (5 pts, fix effort: High). Any of an agents.json, an MCP or WebMCP manifest, a published API documentation link, or a discoverable OpenAPI spec. More than one scores full marks. Most sites score zero here today — which is precisely why it separates the leaders. - **C5 Commerce actionability** (4 pts, fix effort: High). Commerce sites only. Product pages within two clicks of the homepage, price and availability in schema matching the page, a cart or checkout path returning 200 without login, and a guest or express checkout signal. Redistributed across C1–C4 for non-commerce sites. - **C6 On-site search** (3 pts, fix effort: Medium). A search form exists, and it uses GET with a query parameter so an agent can construct search URLs directly rather than having to drive a form. ### Pillar D — Trust & freshness (25 points) _Should an agent — or the model behind it — trust and cite this site?_ - **D1 Transport security** (4 pts, fix effort: Low). Valid TLS, an HSTS header, and no mixed content under render. - **D2 Verifiable identity** (4 pts, fix effort: Low). A contact page or discoverable contact details, plus Organization schema with sameAs links to at least one social or registry profile. - **D3 Policies discoverable** (4 pts, fix effort: Low). A linked privacy policy, plus shipping and returns pages for commerce sites or a terms page otherwise. - **D4 Freshness signals** (4 pts, fix effort: Medium). Sitemap lastmod present, at least 20% of URLs modified in the last 90 days, and visible dates on articles or a changelog. - **D5 Identity consistency** (3 pts, fix effort: Low). The site name agrees across the title element, og:site_name and the schema name, and both favicon and og:image resolve. - **D6 security.txt** (2 pts, fix effort: Low). A valid /.well-known/security.txt per RFC 9116, including a future-dated Expires field. - **D7 Multi-page stability** (4 pts, fix effort: Medium). Every crawled page returns 200, no page exceeds 4 seconds TTFB, and template and navigation stay consistent across pages. ### Grading - A+: 95–100 - A: 85–94 - B: 70–84 - C: 55–69 - D: 40–54 - F: 0–39 --- ## Services ### Agent readiness audit A full audit of what AI agents can and cannot do on your site. A manual audit against the published 100-point rubric: what agents can fetch, parse, and act on, with captured evidence for every finding and a ranked fix list. - Typical timeline: 5 business days - Investment: From $2,000 - URL: https://agentfriendlyrank.com/services/agent-readiness-audit Deliverables: - Score against the published rubric, with per-pillar breakdown - Every finding backed by captured evidence: request, response, and what we saw - Ranked remediation plan, impact against effort - Rendering-parity analysis: what an agent sees versus what a browser sees - Bot access report across GPTBot, ClaudeBot and PerplexityBot - A 60-minute walkthrough call with whoever will do the work **What we actually look at** We fetch your homepage and the pages that matter commercially: product, pricing, docs, contact, checkout. Each is fetched twice. Once as raw HTML, the way most agents see it, and once fully rendered in a real browser. The gap between those two fetches is where the majority of serious problems live, and it is invisible to anyone testing in a browser. We then re-fetch under the GPTBot, ClaudeBot and PerplexityBot user agents and record what comes back. This is the step that catches edge and CDN blocking, which is both the most common failure we find and the one clients are most surprised by, because their robots.txt is usually perfect. Beyond access, we check whether a machine can understand the page (schema coverage and validity, semantic landmarks, heading structure, text-to-markup ratio) and whether it can act: real form labels, correct input types, buttons that are actually buttons, search that an agent can construct a URL for, and whether any machine-readable endpoint exists at all. **Why manual, when there is a free scanner** The automated score is deliberately shallow: six pages, a fixed set of probes, no judgement. It is excellent at telling you that something is wrong and roughly how much it costs you. The audit is where we go deeper than a crawler safely can. We follow the actual purchase or signup path, read your templates rather than inferring them, and separate the findings that matter commercially from the findings that are merely true. A scanner will tell you that 40% of your images lack alt text. The audit tells you which of them are on the pages that convert. **What you get at the end** A score against the published rubric, a full findings list with captured evidence for each one, and a ranked remediation plan ordered by impact against effort, so the work that is cheap and valuable is unambiguous. We will also tell you plainly when something is not worth fixing. An audit that recommends everything is an audit that has not done its job. ### Implementation We make the fixes, in your codebase or your CMS. Done-for-you remediation of agent readiness findings, from schema to rendering to machine endpoints, delivered as reviewable pull requests or CMS changes. - Typical timeline: 2–8 weeks, depending on architecture - Investment: Typically $4,000–$25,000 - URL: https://agentfriendlyrank.com/services/implementation Deliverables: - Reviewable pull requests, or documented CMS changes - Re-scan after each batch so movement is visible, not asserted - Structured data implemented across templates, not page by page - Rendering fixes for the routes that matter commercially - Optional: agents.json, OpenAPI spec, or an MCP server - Handover notes so your team can maintain it without us **How the work lands** Wherever possible we work directly in your repository and deliver pull requests, scoped small enough to review properly. Your team keeps ownership and veto; nothing lands without your approval. For CMS-driven sites we work in the CMS and document every change we make. We do not take over your site, we do not install an overlay, and we do not add a script tag that promises to fix things at runtime. Runtime patches are exactly the kind of client-side dependency that causes agent readiness problems in the first place. **What we typically change** At the cheap end: llms.txt, robots directives, security.txt, canonical hygiene, meta and Open Graph coverage, alt text. These are hours of work and they move the score immediately. In the middle: structured data across templates, heading and landmark structure, form labelling and input types, navigation that uses real anchors, and search that an agent can drive by URL. At the deep end: rendering architecture. If your content only exists after hydration, no amount of schema will help; the fix is server rendering or prerendering for the routes that matter. This is the most expensive work and the one with the largest score movement, which is why we scope it against your architecture rather than your page count. **Machine endpoints** Most sites score zero on machine endpoints today, because almost nobody publishes one. That makes it the cheapest available differentiation: an agents.json, a documented read-only API with an OpenAPI spec, or an MCP server exposing your catalogue or documentation. This is genuinely new territory, and we would rather be honest that the standards are still moving than sell it as settled. We build to what is actually being consumed today, and we tell you which parts are a bet. ### Monitoring Standards move. We tell you what broke, before a customer does. Ongoing re-scanning against the evolving agent readiness rubric, with alerts when a deploy, a CDN setting, or a new standard breaks something. - Typical timeline: Monthly retainer - Investment: Scoped to site size and change rate - URL: https://agentfriendlyrank.com/services/monitoring Deliverables: - Scheduled re-scans with change diffs against your baseline - Alerting on regressions, routed to a human who knows your site - Re-scoring and explanation when the rubric version changes - Quarterly review of what moved and what is now worth doing **What actually breaks, and how** The failure mode is almost never dramatic. A CDN provider changes a default and your bot access disappears overnight. A framework upgrade moves a route from server-rendered to client-rendered. A template refactor drops the structured data from product pages. A certificate renewal quietly loses the HSTS header. None of these produce an error your team would notice. Your site looks fine in a browser, your analytics look fine, and the traffic you lose is traffic you never knew you were going to get. **What we do about it** We re-scan on a schedule, diff against your last result, and tell you what changed and why it matters. The rubric itself will change as WebMCP and agentic commerce protocols settle. When it does, we re-score you against the new version and explain the delta rather than silently moving your number. Alerts go to a human who has read your audit, not to an automated dashboard nobody opens. If something breaks that we can fix in under an hour, we will usually just fix it. --- ## Pricing Audits start at $2,000. Implementation engagements typically run $4,000–$25,000. Pricing is scoped by architecture, not page count: content and configuration fixes are cheapest, template and platform work is mid-range, and rendering architecture or building machine endpoints is the expensive end. The audit fee is credited in full against implementation if the engagement is agreed within 60 days. Everything is quoted and invoiced in USD. --- ## Leaderboard and reports Every completed scan publishes a permanent evidence report at https://agentfriendlyrank.com/site/{domain} — all 27 check results with the raw evidence captured during the crawl, plus the fixes ranked by impact and effort. The leaderboard at https://agentfriendlyrank.com/leaderboard ranks listed sites by their latest score, with category views for e-commerce, SaaS, content and general sites. Two sites can be compared check-by-check at https://agentfriendlyrank.com/compare. Sites scoring grade B or better get an embeddable badge at https://agentfriendlyrank.com/badge/{domain}.svg linking back to their report. Listing rules: every scanned site is listed automatically — scores are measured from a public crawl and listing is opt-out. Owners can have their domain hidden on request (honoured within 48 hours) or block scanning via a robots.txt disallow, which is honoured before anything is published. Our own score is measured by the same pipeline and published on our own report page. --- ## Articles ### Your robots.txt is perfect and you are still blocking agents Discovery & access · Published 9 September 2026 · https://agentfriendlyrank.com/blog/robots-txt-perfect-still-blocking-agents A robots.txt is a request, and the crawler reads it only after it has been let through the door. Everything in front of your origin, a CDN, a web application firewall, a bot-management product, a rate limiter, a reverse proxy with opinions about user agents, decides whether the door opens at all, and none of it consults robots.txt first. A site can have a flawless file and still return a 403 to every AI crawler that asks for it. Check A5 exists to catch that gap. This post is a tour of the layers that cause it: what each looks like from the crawler's side, how it appears in the A5 evidence, and what to change in principle. Cloudflare is the most common instance and has [its own post](/blog/cloudflare-blocks-ai-crawlers-by-default) with the dashboard walkthrough; nothing below depends on which vendor you use. ## What A5 measures The scanner fetches your homepage three times, once each under the GPTBot, ClaudeBot and PerplexityBot user agent strings, and records the status code and the time each request took. Two points if all three receive a 200 rather than a 403 or 429. One further point if the median of the three timings is under 1.5 seconds. Three points, rated Low effort, because the fix is nearly always a setting rather than a build. The evidence block shows the three probes with their `status` and `ttfbMs`, the `medianTtfbMs`, and a flag named `edgeBlockingDespiteRobots`. The flag is set when any probe failed while your robots.txt permits all six crawlers that A2 tests, or does not exist, which counts as open. It names the contradiction: the file says yes, the edge says no. ## The layers, one at a time ### Managed bot rulesets Most firewalls ship a managed ruleset with a category for known bots and, increasingly, a sub-category for AI crawlers, each settable to block, challenge, log or allow. Set to block, it returns 403 on every path, including `/robots.txt`, so the crawler never reads your directives at all. All three probes show 403 and the flag is true. The change is to find the AI-crawler category and allow the crawlers you want by name, rather than switching the ruleset off wholesale. ### Bot-management scores Bot-management products give each request a likelihood of being automated and let you act on buckets, typically "likely automated" and "definitely automated". A crawler is automated by definition, so it lands in the bottom bucket every time, and a rule that blocks or challenges that bucket refuses every crawler regardless of intent. Many products keep a verified-bots list that bypasses the score, but the list is the vendor's, not yours, and need not include the crawlers your robots.txt allows. This shows as a 403 or a challenge. The change is to exempt the crawlers you want by user agent, and by address range where the crawler's operator publishes one, in a rule that runs before the score is consulted. ### Rate limiting tuned for humans A limit written for people browsing sees a crawler fetching robots.txt, a sitemap, four text files and six pages within a few seconds as an attack, and answers 429. The evidence pattern is often one probe at 200 and the next two at 429, because the limit tripped part way through. The change is to set thresholds to what a polite crawler actually does (ours is documented at [/bot](/bot): one concurrent request per host, six pages) and to key the limit on the client address rather than the user agent, so that exemptions for named crawlers hold. ### JavaScript challenges A challenge page asks the client to run a script and prove it is a browser. A crawler cannot, so it stops there. The status is usually 503 or 403 with a challenge document in the body, recorded as a plain non-200. The more insidious variant returns 200 with the challenge as the body: A5 passes, because the status is fine, and A6 catches it instead from the challenge markup. The change is to scope challenges to where abuse actually happens, the login form, the checkout, the signup, and never to the whole site. ### Geographic rules A rule that refuses traffic from regions where you do not sell refuses crawlers too, because crawlers run from cloud regions unrelated to where their users live. It shows as a 403 for some crawlers and not others, depending on where each fetched from. The change is to allow verified crawlers before the geographic rule runs, or to use geography to route rather than refuse. ### Origin-level filters None of this is exclusive to a CDN. A reverse proxy or application firewall on your own servers can key on the `User-Agent` header, and a rule written years ago to stop a scraper by matching on "bot" matches GPTBot, ClaudeBot and PerplexityBot today. Some configurations drop the connection rather than answering; the evidence then shows a status of 0, because no response arrived. The change is to search proxy configuration and framework middleware for anything that reads the user agent, and make the allow list explicit. ### The slow origin The third point is the one people miss. If the median of the three timings is 1,500 ms or over, the point is lost, however healthy the status codes. That is a soft block. An agent working through a task has a budget per fetch shorter than a person's patience, and a page that arrives late is, for its purposes, a page that did not arrive. Our crawler gives up at ten seconds; many agents give up sooner. The time usually goes to one of three places: a cache keyed on the user agent or bypassed for anything that is not a browser, so bots always hit a cold origin; a serverless origin that the first request has to wake; or a homepage that does real work, a database query or a third-party call, on every request. The median forgives one slow probe; two lose the point. ## Test it yourself Fetch `/` and `/robots.txt` under a browser user agent and then under the three strings our crawler sends, and compare status code and time to first byte. Any difference between the first row and the rest is the edge. ```bash for ua in \ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \ "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \ "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \ "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" do for path in / /robots.txt; do curl -s -o /dev/null -A "$ua" \ -w "%{http_code} %{time_starttransfer}s $path\n" "https://example.com$path" done done ``` A 200 is not the end of the test. Fetch the body under a crawler user agent and read the first few hundred bytes; a challenge page and your homepage both return 200, and only one of them contains your content. Then run the loop twice: a rate limit will often pass the first run and fail the second, which is what a real crawl looks like from its side. ## What to change, in principle Three rules cover every layer above. The rule that admits the crawlers you want must run before any rule that scores, limits, challenges or refuses; ordering is the whole game at the edge. Name the crawlers you want in rather than maintaining a list of the ones you want out; a deny list of tokens is never finished. And test under the crawler's identity after every change, because the dashboard describes what a rule intends and the status code describes what it does. ## Check your own site The [free scan](/scan) runs the three probes and records their status, timing and the `edgeBlockingDespiteRobots` flag under A5, next to what your robots.txt says under A2. Both checks are defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### ClaudeBot and Claude-User are different things Discovery & access · Published 8 September 2026 · https://agentfriendlyrank.com/blog/claudebot-claude-user-different-things ClaudeBot collects content that may be used to train Anthropic's models. Claude-User fetches a page because a person asked Claude about it. Same vendor, same first six letters, different jobs, and a robots.txt group that names only the shared prefix refuses both. So do most copied block lists. The concrete consequence is this: a prospect on your pricing page pastes the URL into Claude, asks a question, and the fetch is refused. Both tokens are among the six that check A2 scores. "Block ClaudeBot, allow Claude-User" scores five of six. "Block Claude" scores four of six and refuses the prospect. ## What Anthropic documents **ClaudeBot** is Anthropic's crawler. Anthropic documents it as gathering publicly available web content that may be used for training its models, and documents that it respects robots.txt. **Claude-User** is the fetcher. Anthropic documents it as the agent that retrieves a page when a person using Claude asks about that page or site, and publishes the token so that site owners can address it in robots.txt separately from the crawler. It is not a crawl. It is the request the person made, carried out. **Claude-SearchBot** is a third token, which Anthropic documents as a crawler that navigates the web to improve the quality of search results for Claude users. We name it in our own [robots.txt](/robots.txt) so that our position on it is explicit. A2 does not score it, and this post says no more about it than the documentation does. Anthropic publishes the three as separate tokens so that the three decisions can be made separately, in the same way OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User. [Blocking GPTBot but not OAI-SearchBot](/blog/blocking-gptbot-but-oai-searchbot) is the OpenAI version of this post. ## How the accidental block happens There are two common routes to it. The first is the prefix. In common implementations, ours included, user agent matching is case-insensitive and matches the token as a prefix. A group written as `User-agent: Claude`, or `User-agent: claude`, or `User-agent: Claude*`, matches ClaudeBot, Claude-User and Claude-SearchBot alike. The owner meant the crawler. The file says all three. The second is the copied list. A "block all AI" list from a repository or a plugin runs to dozens of tokens, and Claude-User sits in it next to ClaudeBot. The owner reads the vendor name, thinks "training", and pastes the lot. The file now names the fetcher explicitly, so there is no prefix to blame; the token is simply in the wrong list. Both produce the same A2 result: two of the six tokens blocked, four of six scored, and a report whose evidence lists `ClaudeBot, Claude-User` as blocked. ## What the block does to a prospect The scenario is ordinary. Someone evaluating your product has your pricing page open. They open Claude, paste the URL, and ask which tier includes the feature they need and whether there is an annual discount. Claude sends Claude-User to fetch the page. The fetch is governed by your robots.txt, which is why Anthropic documents the token at all, and the rubric treats a matching `Disallow: /` as a refusal: the page is not retrieved. Whatever follows is assembled without your page. The person asked a specific question about a page they were looking at, and your site declined to answer it. That is the whole cost of the accidental block, and it has nothing to do with training. Blocking ClaudeBot has no effect on that scenario. It affects only whether the content is collected for training. The two tokens exist so that you can refuse one thing without refusing the other. ## The file for each intent Open to Anthropic entirely: ```text User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot Allow: / ``` Six of six on A2, provided the other four tokens are also allowed. Consecutive `User-agent` lines share one group under the REP. Opt out of training, stay fetchable and indexable: ```text User-agent: ClaudeBot Disallow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / ``` Five of six. The evidence reads `blocked: [ClaudeBot]`, and the recommendation text says that if the block is deliberate there is nothing to do. The accidental one: ```text User-agent: Claude Disallow: / ``` Four of six, and the prospect refused. Two notes on the fix. In our parser, when several groups match a user agent, the group with the longest matching token wins, so adding an explicit `User-agent: Claude-User` group with `Allow: /` alongside the prefix group would restore the fetcher. Other implementations may not resolve the conflict the same way, so the safer fix is to delete the prefix group and name each token in full. And a named group does not inherit the wildcard group's disallows; if you keep a hygiene list under `User-agent: *`, repeat it in the Claude-User group. [robots.txt for the agent era](/blog/robots-txt-agent-era) covers both rules. If the training opt-out is narrower than the whole site, scope it by path. `User-agent: ClaudeBot` with `Allow: /` and `Disallow: /blog/` keeps the root open, scores six of six, and withholds the archive from training. The longest matching path wins, so the archive is refused and nothing else is. ## Why the distinction is the point The fetcher is a person's request. Of all the traffic that arrives under an AI vendor's name, it is the closest thing to a customer walking in and asking a question. The training crawl is a policy decision about future models. Putting both behind one line means the policy decision refuses the customer, and it does so silently, because nothing reports the refusal to the person and the owner never sees a request declined at the file rather than at the server. The rubric's proportional scoring is built around this. A site that blocks ClaudeBot keeps five points and a note that the choice is recorded. A site that blocks the prefix loses a second point for a decision it did not make. [The six AI crawlers](/blog/six-ai-crawlers-each) sets out the same logic across all six tokens and four kinds of site. One boundary. robots.txt governs crawlers that read it. A CDN that refuses the ClaudeBot user agent at the edge returns a 403 before the file is read, and the same edge rule commonly refuses Claude-User too. Check A5 probes under ClaudeBot and records what comes back, and [the Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers that layer. ## Check your own site The [free scan](/scan) reads your live robots.txt and lists which of the six tokens it blocks. If Claude-User appears in that list and you only meant the crawler, the fix is the file above. A2 is defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### Blocking GPTBot but not OAI-SearchBot, the distinction that matters Discovery & access · Published 7 September 2026 · https://agentfriendlyrank.com/blog/blocking-gptbot-but-oai-searchbot OpenAI documents three user agent tokens, and there are three because they do three different things. GPTBot collects content that may be used to train models. OAI-SearchBot builds the index behind OpenAI's search features and, per its documentation, is not used for training. ChatGPT-User fetches a page because a person asked for it. A site can therefore opt out of training while remaining a citable source and remaining readable on request, and the file that does so is a few lines long. On check A2 it scores five of six. This post shows the file, the score, the two ways the group gets written wrongly, and what each side of the decision costs. ## Three tokens, three decisions **GPTBot** is the training crawler. OpenAI's documentation says content it fetches may be used to train its generative models, and that a `Disallow` for GPTBot in robots.txt is honoured. That is the whole scope of the token: whether your pages contribute to training. **OAI-SearchBot** is the search-index crawler. OpenAI documents it as the agent used to surface and link to websites in its search features, and states that it is not used to gather training data. Its job is to know your pages exist and what they say, so that an answer can cite them. **ChatGPT-User** is the fetcher. OpenAI documents it as acting on behalf of a user, retrieving a page when a person asks about it, and states that it is not used for automatic crawling or for training. It fetches one page because one person asked. OpenAI publishes separate documentation and separate IP address ranges for each token. That is what makes the three decisions independent, and it is also how a request carrying one of these names can be verified against the published ranges rather than trusted on its user agent string alone. A2 scores GPTBot and OAI-SearchBot as two of its six tokens. It does not score ChatGPT-User. We name it in our own [robots.txt](/robots.txt) regardless, on the principle that applies to all three: name what you have a view on, in full. ## The file Opt out of training, stay citable, stay fetchable: ```text User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: * Allow: / Disallow: /admin ``` Each named group stands alone. Under the Robots Exclusion Protocol a crawler that finds a group naming it obeys only that group, so `OAI-SearchBot` with a lone `Allow: /` means everything, including `/admin`. If you keep a hygiene disallow list, repeat it in every named group; the wildcard group's rules do not carry over. [robots.txt for the agent era](/blog/robots-txt-agent-era) works through that and the other REP rules line by line. What this file scores. A2 tests each of its six tokens against the root path. GPTBot is refused. OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended are allowed, because the first has its own permissive group and the other four fall through to the wildcard group, which permits `/`. Five of six, and the evidence on the report reads `allowed: 5, of: 6, blocked: [GPTBot]`. The recommendation text says that if the block is deliberate there is nothing to do. It is deliberate, and the report has made it visible rather than penalised it further. If the opt-out is narrower than the whole site, scope it by path rather than by token: ```text User-agent: GPTBot Allow: / Disallow: /archive/ ``` The root stays open, so A2 scores six of six, and the archive is withheld from training. The longest matching path wins, so anything under `/archive/` is refused and everything else is permitted. ## The group that catches more, or nothing There are two ways to write the token wrongly, in opposite directions. Too wide. In common implementations, ours included, user agent matching is case-insensitive and matches the token as a prefix, so `User-agent: GPT` matches GPTBot, which may be what you meant. Implementations that also match a token appearing anywhere in the name, and ours does, match `ChatGPT-User` as well, and now the training opt-out has refused the person. A truncated token is not a shortcut; it is a different rule. Dead. `User-agent: OpenAI` matches none of the three, because no token begins with it. The group is inert, the wildcard group applies, and the owner who believes they have opted out of training has not. Neither error is visible in a browser. In the A2 evidence, the first shows GPTBot blocked as intended while the harm lands on a token A2 does not score; the second shows GPTBot absent from the blocked list when you expected it there. The fix for both is the same: write each token exactly as the vendor documents it, and write one group per decision. ## What a publisher gives up, and what it protects Blocking GPTBot protects a contribution. Content the crawler would have fetched is not used to train OpenAI's models, to the extent OpenAI honours the file, which its documentation says it does. For a site whose text is the product, that is a decision about an asset, and the rubric charges one point for it without arguing. Blocking OAI-SearchBot gives up being a source. OpenAI documents that crawler as the way sites are surfaced and linked in its search features. A site absent from that index cannot be cited by it, and nothing reports the omission to the person asking; the answer is assembled from what could be indexed. For a publisher, a citation is the mechanism by which an answer sends a reader to you, and blocking the index crawler removes the link. The two decisions are independent by design. "Not training data, but still a source" is available, and so is "neither", but the second is two decisions written as two groups, not one line that happens to cover both. The mistake we see is a "block all AI" list, copied whole, that puts OAI-SearchBot and ChatGPT-User beside GPTBot when the owner only ever meant the first. [The six AI crawlers](/blog/six-ai-crawlers-each) sets out which combination fits which kind of site, and the publisher row is the one this post belongs to. One boundary. A `Disallow` in robots.txt is honoured by crawlers that read the file; it is not a refusal at the network. Check A5 sends a real request under the GPTBot user agent and records the status. A site that disallows GPTBot in robots.txt still returns a 200 to that probe, because the file is a request, not a firewall, and A5 is unaffected. A site whose CDN refuses GPTBot at the edge returns a 403, and the same edge rule usually refuses OAI-SearchBot too, which is the accidental version of this post's subject. [The Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers it. ## Check your own site The [free scan](/scan) reads your live robots.txt and reports exactly which of the six tokens it blocks, so you can confirm the file names GPTBot and only GPTBot if that is what you meant. A2 is defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### The six AI crawlers and what each one is for, a decision matrix Discovery & access · Published 6 September 2026 · https://agentfriendlyrank.com/blog/six-ai-crawlers-each [The field guide](/blog/field-guide-to-ai-user-agents) lists the six user agent tokens check A2 scores and what each vendor documents them for. This post is the decision that follows from it. For four common kinds of site, it says which of the six a reasonable owner would allow, what each combination scores, and what each block actually gives up. The short version is that only one of the four has a strong reason to block anything, and even that one should not block all six. ## One point per token A2 is worth six points, scored proportionally. The engine parses robots.txt, asks for each token whether the root path `/` is allowed, and awards one point per token permitted: six of six scores six, three of six scores three, none scores zero. It sends no requests under these names; that is check A5, which tests whether the edge agrees with the file. The six tokens do three jobs. | Job | Tokens A2 scores | What a block refuses | |---|---|---| | Training crawl | GPTBot, ClaudeBot, Google-Extended | Your content's contribution to a vendor's future models | | Search index | OAI-SearchBot, PerplexityBot | Being fetched, indexed and cited as a source in that product | | Fetch on a person's behalf | Claude-User | The one page a person just asked about | Google-Extended is a control token rather than a crawler. Google documents it as governing whether crawled content may be used for training its models, and states that it does not affect a site's inclusion in Search. For decision purposes it belongs in the training row. ## The matrix | Site type | Allow | Block | A2 | |---|---|---|---| | Commerce | All six | None | 6/6 | | SaaS | All six | None | 6/6 | | Publisher | OAI-SearchBot, PerplexityBot, Claude-User | GPTBot, ClaudeBot, Google-Extended | 3/6 | | Services firm | All six | None | 6/6 | Three rows are identical, and that is the finding. For most businesses the text on the site is not the asset; it exists to be found, repeated and acted on. The exception is the site whose text is the product. ### Commerce A shop's product descriptions were written to be repeated. A training crawl contributes the copy you wanted circulated. A search-index crawl is how an answer to "which of these should I buy" can point at your product page rather than someone else's. And a block on Claude-User refuses the shopper who pasted your product URL and asked about sizing or delivery. There is no row of the table where a shop gains from a block. If a shop has editorial content it does value, buying guides or a magazine section, the right granularity is the path, not the token. `User-agent: GPTBot` with `Allow: /` and `Disallow: /guides/` withholds the guides from training and leaves the catalogue open. A2 tests the root path, so a path-scoped opt-out keeps the point, and more importantly it says what you meant. ### SaaS Documentation, pricing and a changelog are the pages an assistant fetches when a person asks whether a product does something, or how. Allowing the training crawlers means that copy contributes to what future models are trained on; allowing the index crawlers means the docs can be cited; allowing Claude-User means a prospect can paste the pricing page and ask about it. Some SaaS owners prefer to keep docs out of training on principle. That variant blocks GPTBot, ClaudeBot and Google-Extended, scores three of six, and is a deliberate choice the report records as one. ### Publisher The only row with a real trade-off. When the text is the product, opting out of training is a decision about an asset, and the rubric charges three points for it without arguing. The question is what to do with the other three tokens. OAI-SearchBot exists so that a site can be a cited source without being training data; OpenAI documents the two crawlers separately for exactly that reason. Blocking it removes you from the index that citations come from. PerplexityBot is the same decision for a different product, and some publishers refuse it; that variant scores two of six. Claude-User is a reader. A person looking at one of your articles has asked for a summary or a clarification, and the fetch is that request. Blocking it refuses the reader, not the vendor. The [field guide](/blog/field-guide-to-ai-user-agents) covers the OpenAI tokens in detail, and our own [robots.txt](/robots.txt) shows every token named in full. ### Services firm A law firm, an agency, a clinic or a consultancy has little in its copy worth withholding from training and a great deal to lose from not being read. The pages that matter answer "who are these people, what do they do, how do I contact them", and the risk runs the other way: an assistant asked about you answers from a directory listing or a review site, because your own site refused the fetch. Allow all six. ## Variants and what they score | Intent | Blocked tokens | A2 | |---|---|---| | Open to all | None | 6/6 | | Opt out of one vendor's training | GPTBot | 5/6 | | Opt out of training everywhere | GPTBot, ClaudeBot, Google-Extended | 3/6 | | Opt out of training and of answer engines | GPTBot, ClaudeBot, Google-Extended, PerplexityBot | 2/6 | | A copied "block all AI" list | All six | 0/6 | Every row is a coherent intent except the last. The report lists exactly which tokens are blocked and, where the block is deliberate, says there is nothing to do. The check exists to make the decision visible, not to make it for you. ## What each block actually forgoes Blocking a training crawl forgoes a contribution. Your content is left out of what that vendor's future models are trained on, to the extent the vendor honours the file. It does not change whether an assistant can read your site today, and it is the block with the clearest case for it. Blocking a search-index crawl forgoes being a source. A citation is a link, and an answer that could have pointed at your page points elsewhere. Nothing reports the omission to the person; the answer is assembled from what could be fetched. Blocking a fetch on a person's behalf forgoes the person. They asked about your page, at that moment, with intent, and the fetch was refused. This is where the owner's intent and the file's effect diverge most, because nobody writes "refuse my prospects" in a robots.txt on purpose. The [field guide](/blog/field-guide-to-ai-user-agents) sets out the two Anthropic tokens side by side. ## Why "block them all" is almost never what was meant Block lists circulate as a unit. A list of forty tokens is copied from a repository or switched on in a plugin, and the owner reads it as "do not train on my content". The file it produces says "do not train on my content, do not cite my content, and do not read my pricing page for the customer who asked". A2's zero for a blanket block is the rubric noticing that three jobs were treated as one. The fix is not to allow everything. It is to write the file per token, each named in full, carrying the decision you actually made. [robots.txt for the agent era](/blog/robots-txt-agent-era) is the line-by-line rewrite. And the file is necessary, not sufficient: a CDN that refuses these user agents at the edge overrides all of it, which is what check A5 and [the Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) are about. ## Check your own site The [free scan](/scan) parses your live robots.txt and lists which of the six tokens it blocks, so you can compare the file against the row of the matrix you meant to be in. A2 is defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### robots.txt for the agent era, a practical rewrite Discovery & access · Published 5 September 2026 · https://agentfriendlyrank.com/blog/robots-txt-agent-era Your robots.txt is the first file a well-behaved crawler asks for, and most of the files in circulation were last edited for a search engine, years ago. They still work, in the sense that nothing breaks. They also assume a reader that stopped being the only one once assistants began fetching pages on people's behalf. This post takes one such file, works through it line by line against the rules a parser actually applies, and ends with a complete file you can adapt. Three checks in our rubric read robots.txt directly: A1 (exists and parses, two points), A2 (six named AI crawlers allowed, six points) and A3 (sitemap referenced, one of its three points). [The field guide](/blog/field-guide-to-ai-user-agents) covers what each crawler is for. This post is about the file. ## Find out what is actually being served Before editing anything, fetch the live file rather than the one in your repository. A CDN can append to it, and a framework can serve something else at that path without anyone noticing. ```bash curl -sI https://www.example.com/robots.txt curl -s https://www.example.com/robots.txt | head -n 20 ``` There are four outcomes, and A1 scores each differently. A 200 with `Content-Type: text/plain` and recognisable directives is the goal. A1 awards both points. A 404 scores one point of two. No file means nothing is disallowed, so the site is open by default and A2 treats all six tokens as allowed. It is still sloppier than an empty file. Crawlers get no sitemap reference, so A3 loses its "referenced" point, and nobody can tell whether you decided to be open or never thought about it. A file containing only `User-agent: *` and an empty `Disallow:` says the same thing on purpose. A 200 that is actually HTML scores zero. This happens when a single-page application's catch-all route serves the app shell for any unknown path, or when a CMS returns its "not found" page with a 200. The parser finds no directives, marks the file invalid, and the report advises checking for HTML being served in its place. This is worse than a 404, because the crawler asked and was handed something it could not read. Any other status also scores zero. Our crawler follows redirects, so what matters is the final status. A 403 on robots.txt is almost always the edge, not the file, and [the Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers it. ## A typical legacy file Every line in this composite is common. ```text User-agent: * Disallow: /wp-admin/ Disallow: /cgi-bin/ Disallow: /search Crawl-delay: 10 User-agent: Googlebot Allow: / User-agent: GPTBot User-agent: CCBot Disallow: / User-agent: Claude Disallow: / Sitemap: http://www.example.com/sitemap_index.xml ``` Here is what each part does under the Robots Exclusion Protocol (RFC 9309), as our parser applies it. ### The wildcard group does not merge `User-agent: *` is a fallback for any crawler that has no group of its own. It is not a base that named groups extend. A crawler that finds a group naming it obeys that group and only that group, so every named group below starts from a blank slate. ### Crawl-delay is not a control `Crawl-delay` is not part of RFC 9309. Our parser ignores it, and not every crawler honours it. Leaving it in is harmless, but it is not rate limiting. If you need to slow a crawler down, do it at the server and confirm the crawlers you want still receive a 200, because check A5 counts a 429 as a refusal. ### The Googlebot group opened the back door `User-agent: Googlebot` followed by `Allow: /` was written to make sure Google could crawl everything. It did more than that. Googlebot now obeys only its own group, which has no disallows, so `/wp-admin/` and `/search` are open to it. Whoever wrote the wildcard list assumed it carried over. It did not. A named group carries the same disallow list as the wildcard group, or it carries none. ### The training opt-out `GPTBot` and `CCBot` share a group because consecutive `User-agent` lines accumulate into one. Refusing OpenAI's training crawler is a defensible decision, and the report treats it as one. It costs a single A2 point, because GPTBot is one of the six tokens scored and CCBot is not; if it is deliberate, the report says there is nothing to do. [The field guide](/blog/field-guide-to-ai-user-agents) goes through the OpenAI tokens, and the rest, in detail. ### The Claude group refuses more than it meant to User agent matching is case-insensitive and matches on the prefix, in common implementations including ours. `User-agent: Claude` therefore matches ClaudeBot, which collects content for training, and also Claude-User, which fetches a page because a person asked Claude about it. The owner meant to opt out of training. The file also refuses every prospect who pastes a URL into Claude. That is two A2 points gone, and the second is the expensive one, because it refuses a person rather than a crawler. ### The Sitemap line is load-bearing Three problems. It uses `http` while the site serves `https`. It points at an index file. And it is last. The `Sitemap` directive is independent of groups and may appear anywhere, so position does not change its meaning. It does change what gets read. Our crawler fetches the first sitemap declared. A3 awards its "referenced" point for any declaration, but check D4 reads `lastmod` from the URLs in whichever file that first declaration returns. An index file lists child sitemaps rather than pages, with no per-page `lastmod`, so D4 sees none. List the file that contains real page entries with real `lastmod` values first. The index and any chunks can follow. ## The rewrite ```text Sitemap: https://www.example.com/sitemap.xml Sitemap: https://www.example.com/sitemap-products.xml User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /search Disallow: /checkout/thanks User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Googlebot User-agent: Bingbot Allow: / Disallow: /wp-admin/ Disallow: /search Disallow: /checkout/thanks ``` What changed, and why. The sitemap comes first, over `https`, and it is the file with page entries. A3 scores this as referenced, and D4 has something to read. The wildcard group keeps the hygiene list and drops a path that no longer exists and a directive that never did anything. `Allow: /` is not required, since a group with no rules allows everything, but it makes the intent unambiguous. Every crawler we have a view on is named in full, never by prefix, and the named group carries exactly the same disallow list as the wildcard group, because it will not inherit it. Our own [robots.txt](/robots.txt) emits one group per token; stacking the `User-agent` lines as above is equivalent under the REP and shorter. Split a token into its own group when its decision differs. Moving GPTBot, ClaudeBot and Google-Extended into a group with `Disallow: /` opts out of training while keeping search indexing and user fetches open. That file scores three of six on A2, and the report says which three. Which combination fits which kind of site is a decision by site type, and the [field guide](/blog/field-guide-to-ai-user-agents) gives the cost of each block. Within a group, the longest matching path wins. `Allow: /` and `Disallow: /search` in the same group means `/search` and everything under it is refused and everything else is permitted. On an exact tie, `Allow` wins, so a group containing both `Disallow: /` and `Allow: /` allows everything. An empty `Disallow:` also allows everything. ## Test the result 1. `curl -sI` on the live URL returns 200 with a `text/plain` content type. 2. `curl -s` on the same URL prints your directives, not markup. Check again after each deploy. 3. The first `Sitemap` URL returns 200 and contains a `urlset` with `lastmod` values. 4. Re-run the fetch with `-A "GPTBot"` and `-A "ClaudeBot"` and confirm a 200. This file cannot fix a refusal at the edge; that is check A5, and [the Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers the most common source of those refusals. 5. Run a scan and read the evidence under A1, A2 and A3, which records how many groups and sitemaps parsed and which of the six tokens are blocked. Our [bot page](/bot) documents what our crawler does with your file, including that it obeys it. ## Check your own site The [free scan](/scan) reads your live robots.txt, parses it under the rules above and reports A1, A2 and A3 with the evidence attached. The three checks are defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### Agent readiness is not SEO, and not GEO either Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/agent-readiness-is-not-seo Three disciplines now claim the same website, and each asks it a different question. SEO asks whether a search engine will rank a page. GEO, generative engine optimisation, asks whether a generated answer will cite it. Agent readiness asks whether a program acting for a person can complete a task on it. The first two are about visibility. The third is about use, and a site can be highly visible and unusable at the same time. The overlap between the three is real, and a team with good technical SEO hygiene starts with a meaningful head start on our rubric. But the overlap is confined to roughly half of the rubric, and the half it misses is the half where the commercial damage happens. ## Three questions, one site It helps to be precise about what each discipline is optimising for. SEO optimises for position in a ranked list. The consumer is a search crawler that fetches pages, renders them in a headless browser, follows links, and feeds an index that orders results by relevance and authority. Success is relative: you are above or below a competitor. GEO optimises for citation. The consumer is a model, or the retrieval system in front of it, assembling an answer from sources. Success is being one of the sources the answer names. It is still visibility, and still relative, but the unit is a claim rather than a page, and the signals that matter shift towards clarity, extractability and provenance. Agent readiness optimises for task completion. The consumer is a program that has been asked to do something specific: find the price, check availability, fill in the enquiry form, compare two plans, put an item in a basket. Success is absolute. Either the task completed or it did not, and no amount of ranking compensates for a form that cannot be submitted. ## Where they overlap About half of our rubric would be recognised by any technical SEO. In [pillar A](/methodology#pillar-A), a `robots.txt` that parses (A1) and a valid, referenced sitemap (A3) are decades-old hygiene. In [pillar B](/methodology#pillar-B), parseable JSON-LD (B1), the right schema types for the site (B2), one `h1` and sane heading order (B3), title and description lengths (B4), alt text (B6) and canonical hygiene (B7) are the standard technical audit. In [pillar D](/methodology#pillar-D), TLS (D1), `lastmod` freshness (D4) and consistent identity (D5) are likewise familiar. If you have done that work properly you have already earned a good share of those pillars, and we would rather say so than pretend the disciplines are unrelated. The points are the same points. What differs is why they matter and what sits beside them. ## Where SEO stops The first divergence is rendering. The dominant search crawler renders JavaScript before indexing, and its operator documents that it does. Most agents fetch the raw HTML and do not. A site built as a client-rendered application can index perfectly and still hand an agent an empty shell. That is check A7, server-rendered content parity, five points and rated High effort, and it sits upstream of everything: schema, forms and links that only exist after hydration are, to a plain fetch, not there. [The two fetches](/blog/the-two-fetches) explains how it is measured. The second is access. A `robots.txt` written with a search crawler in mind says nothing about GPTBot, ClaudeBot or Claude-User, and a CDN's default bot rules can refuse those user agents before `robots.txt` is read. Check A2 scores what the file says about six named crawlers; check A5 sends real requests under three of them and records what comes back. A site can pass every crawlability test in a search console and fail both. [The field guide to AI user agents](/blog/field-guide-to-ai-user-agents) covers what each crawler is for. The third is the nature of the score. Ranking is relative, so SEO is a contest and the result depends on who else is competing. Usability is not a contest. A form either has a labelled email input with a real submit button or it does not, and the answer is the same whether or not a competitor's form is worse. ## Where GEO stops GEO is closer to agent readiness than SEO is, and the two are often confused because they share vocabulary. Both care about `llms.txt` (A4). Both care about the search-index crawlers being let in. Both reward freshness signals (D4), a verifiable identity (D2) and a site name that agrees with itself across the title, `og:site_name` and schema (D5), because a model weighing whether to cite a source can check those things and a human reader cannot easily. But GEO stops at citation. Its question is whether your page becomes a source in an answer. It does not ask whether a program could then do anything on the page: submit the form the answer pointed at, run the search the answer recommended, or complete the purchase the answer suggested. A site can be cited constantly and be unusable for every one of those tasks. The citation is a link, and the link leads to the same client-rendered shell, the same click-handled `div` navigation, the same login-gated cart. Being cited and being usable are different properties. The first is worth having. The second is where the task actually completes, or fails silently. ## What only agent readiness asks [Pillar C](/methodology#pillar-C), actionability, has no equivalent in either discipline. Its six checks are worth 25 points, a quarter of the score, and none of them has ever been a ranking factor. C1 asks whether every form input carries a label or `aria-label`, whether input types are correct (`email`, `tel`, `number`), and whether submit is a real button. C2 asks whether navigation uses anchor elements rather than click-handled `div`s, and whether fewer than 5% of links use contextless text such as "click here". C3 asks whether buttons are `button` elements, focusable, in a sane tab order, with no keyboard traps under render. C4 asks whether any machine endpoint exists at all, an `agents.json`, an MCP or WebMCP manifest, published API documentation or a discoverable OpenAPI spec. C5, for commerce sites, asks whether a product is within two clicks of the homepage, whether price and availability in schema match the page, and whether a cart or checkout path returns 200 without a login. C6 asks whether on-site search uses GET with a query parameter, so an agent can construct a search URL rather than having to drive a form. Most of these overlap heavily with accessibility rather than with SEO. A form a screen reader can fill is a form an agent can fill, for the same reasons. If your organisation has done WCAG work, that is the team whose knowledge transfers. The failure mode is also different. A page that drops in rankings shows up in a dashboard. An agent that cannot submit your form does not file a report. It uses a competitor it could read, and the enquiry you did not receive is not counted anywhere. ## What this means for who does the work The practical consequence is that agent readiness is not a job you can hand entirely to whoever does your SEO, and not one you can hand entirely to engineering either. The parts that overlap with SEO, most of pillars B and D and the hygiene half of pillar A, are Low and Medium effort in the rubric and can be owned by the same people who own your search work. The rendering question in A7 and the machine endpoints in C4 are rated High effort because they are architecture, not content. The rest of pillar C is engineering work of the ordinary kind: templates, form markup, the search route. [The three tiers of fixes](/blog/three-tiers-of-fixes) maps the effort ratings to what the work involves, and the [implementation service](/services/implementation) exists for the parts a marketing team cannot do in a CMS. ## Check your own site The [free scan](/scan) scores all four pillars, so you can see directly which of your points came from SEO hygiene and which are missing because nobody has ever asked the actionability question. The full definitions are on the methodology page, starting with [discovery and access](/methodology#pillar-A) and continuing to [actionability](/methodology#pillar-C), the pillar the other two disciplines do not cover. ### Cloudflare blocks AI crawlers by default, before it reads your robots.txt Discovery & access · Published 4 September 2026 · https://agentfriendlyrank.com/blog/cloudflare-blocks-ai-crawlers-by-default The most common serious finding in a scan is one the site owner never chose. Their robots.txt welcomes every AI crawler by name. Their edge returns a 403 to all of them. The two facts sit side by side in the report, and the second one wins, because the block happens at the CDN before the crawler ever asks for robots.txt. In almost every case the CDN is Cloudflare, and the block is a default. This article explains what the default does, how to see it from your own terminal in about thirty seconds, and precisely which settings to change if you decide you want agents to reach you. It also covers the case where you do not, because that is a legitimate choice, as long as it is a choice. ## What actually happens to the request When a site sits behind Cloudflare, every request lands at a Cloudflare edge node first. The node decides whether to pass the request to your origin, serve it from cache, challenge it, or refuse it. That decision is made from the request itself: the IP, the TLS fingerprint, the headers, and above all the `User-Agent`. Cloudflare maintains a list of known AI crawlers and, since mid-2025, blocks them by default on new zones. Existing zones were offered the same block as a one-click toggle, and a great many owners turned it on during the period when "block the AI bots" sounded like free hygiene. Either way the rule runs at the edge, in front of everything you control. That includes robots.txt. A polite crawler's first request to a new host is `GET /robots.txt`. If the edge refuses that request, the crawler never sees your `Allow: /`. It sees a 403, or a challenge page that returns 403 with a `cf-mitigated: challenge` header, and it leaves. Your carefully written directives are correct and unread. This is why the two checks in the rubric that look at access are separate. [A2](/methodology#pillar-A) parses robots.txt and awards points for each of six named crawlers it permits. [A5](/methodology#pillar-A) ignores robots.txt entirely and simply fetches your homepage three times, as GPTBot, as ClaudeBot and as PerplexityBot, and records what comes back. A site can score full marks on A2 and zero on A5, and when it does, the evidence block on the report says so in plain words: the block is happening at your CDN or WAF, before robots.txt is ever read. ## Why it is the default Cloudflare's position is that AI companies have been taking content without permission or payment, and that the honest default for a publisher is closed until a deal is struck. For a newspaper, a stock-photo library or a forum whose entire value is the text on the page, that is a reasonable place to start. Blocking the training crawlers protects an asset. Most businesses are not publishers. A software company, a shop, a law firm or a clinic wants to be found, described accurately and, increasingly, acted on. For them the block does not protect an asset; it removes them from the set of sites an assistant can use to answer a question or complete a task. The assistant does not report the 403 to the person who asked. It answers from whatever it could read, which is a competitor. The default is therefore right for some sites and wrong for others, and the problem is not the default itself. It is that the setting was made for you, at the infrastructure layer, and nothing in a browser will ever show it. ## How to check in thirty seconds You need a terminal and curl. Fetch your homepage as a normal browser, then as each of the three crawlers the scanner uses, and compare the status codes. ```bash for ua in \ "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \ "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \ "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \ "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" do printf '%-14s ' "$(echo "$ua" | grep -oE 'GPTBot|ClaudeBot|PerplexityBot' || echo Browser)" curl -s -o /dev/null -w '%{http_code} %{time_starttransfer}s\n' -A "$ua" https://example.com/ done ``` A healthy result is four 200s. The signature of an edge block is a 200 for the browser and 403 for the rest. A 429 means rate limiting is catching the crawlers; a 503 with a challenge page in the body means the request is being asked to run JavaScript, which a crawler cannot do. All three are the same finding for practical purposes: the crawler does not get the page. Run the same loop against `/robots.txt`. If that returns 403 to the crawlers too, you have confirmed that they never read your directives at all. If you would rather not use a terminal, the [free scan](/scan) does exactly this and keeps the evidence: the three probes, their status codes and their time to first byte are recorded under A5 on the report. ## Deciding what you actually want Before changing anything, decide which crawlers you want and why. There are three kinds, and the distinction matters more than the vendor. **Training crawlers** collect text to train models. GPTBot, Google-Extended (a directive rather than a crawler, but it controls the same thing) and CCBot are the main ones. Blocking these keeps your content out of future training runs. It has no effect on whether an assistant can read your site today. **Search and retrieval crawlers** index pages so an assistant can cite them in answers. OAI-SearchBot and Claude-SearchBot are examples. Blocking these removes you from the answers, which is the opposite of what most businesses want. **User-triggered fetches** happen when a person asks an assistant to look at a specific page. ChatGPT-User, Claude-User and Perplexity-User carry the request. Blocking these means that when a prospect pastes your pricing page into an assistant and asks a question, the assistant cannot read it. You can allow the second and third kinds while refusing the first. Cloudflare's per-crawler controls make that possible, and [the field guide to AI user agents](/blog/field-guide-to-ai-user-agents) sets out what each token is for. The rubric's A2 is scored proportionally across six agents for the same reason: a site that opts out of training but stays reachable for search and user fetches has made a defensible choice and keeps most of the points. A blanket block of all six scores zero because it is almost never what the owner meant. ## The settings to change Names in the Cloudflare dashboard move around, so this is described by function rather than by exact label. There are four places a crawler can be refused, and you need to look at all four. 1. **The AI crawler block.** Under Security, then Bots, there is a toggle that blocks known AI crawlers on every page. On zones created after mid-2025 it is on. Turn it off if you want agents in, or leave it on and use the per-crawler controls below to open specific ones. 2. **Per-crawler controls.** The AI crawl control panel lists each crawler Cloudflare recognises with an allow or block per row, and shows how often each has been refused, which is a useful measure of how much traffic the block has been costing. Set the retrieval and user-fetch crawlers to allow. Decide on the training crawlers separately. 3. **Bot Fight Mode and Super Bot Fight Mode.** These challenge or block traffic that looks automated. Crawlers on Cloudflare's verified list are meant to pass, but a "definitely automated: block" setting still catches the ones it does not recognise, and the verified list is not the same as your allow list. If you use these, confirm with the curl loop that the crawlers you allowed actually get through. 4. **WAF custom rules and rate limits.** Search your custom rules for anything matching on `http.user_agent` or on the bot score, and check whether a rate-limiting rule is set low enough to catch a crawler that fetches six pages in a few seconds. A rule written two years ago to stop a scraper is often the thing refusing ClaudeBot today. There is also a managed robots.txt feature that appends directives to your file. It is worth knowing about because it can make your robots.txt say something you did not write; check the live file, not the one in your repository. After each change, run the curl loop again. Cloudflare applies rule changes within seconds, so there is no need to wait. When the four requests return 200, run a fresh scan so the improvement is measured rather than assumed. ## What it costs on the rubric, and what it costs in practice The rubric charges two points for the block, under A5, because the rubric measures what a crawl can prove and the block is a single fact: three probes, three refusals. If the block also hides your robots.txt, A2 falls too, and any check that depends on fetching pages under a crawler's identity can only report what it was allowed to see. Two points understates it. Every other check in the rubric describes how well an agent can use your site once it is inside. This one decides whether it gets in. A site with excellent structured data, server-rendered pages and a clean form layer that returns 403 to every crawler has done all of that work for nobody. The reason we lead cold outreach with this finding is that it reads as news rather than criticism. Nobody chose it, nobody can see it from a browser, and it is fixed in a few minutes by someone who already has dashboard access. ## Not only Cloudflare Other edges have equivalent rules. Most large CDNs and hosting platforms now offer a managed bot ruleset with an AI crawler category, and some have begun defaulting it on for new projects. The mechanism is identical, and so is the test: fetch as the crawler, read the status code, and believe that over anything the dashboard says. Cloudflare's own documentation for its bot products is at [developers.cloudflare.com](https://developers.cloudflare.com/), and a general tour of every layer that can refuse a crawler, vendor by vendor, follows later this week. Our own site runs with nothing in front of the origin at all, which is a deliberate choice explained on [our live report](/site/agentfriendlyrank.com), not a recommendation for everyone. ## Check your own site Run the curl loop above, or run the [free scan](/scan) and read the A5 evidence. Both take under a minute. If the probes return 403 and you want them to return 200, the settings above are the whole fix. The detail of how A2 and A5 are scored is on the [methodology page](/methodology#pillar-A). ### Who is actually fetching your site, a field guide to AI user agents Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/field-guide-to-ai-user-agents "AI crawlers" is a group name for at least three different jobs. One kind of bot collects pages to train a model. Another builds a search index so a product can cite sources. A third fetches a single page because a person, right now, asked an assistant to look at it. And one of the six tokens our rubric scores is not a crawler at all but a control switch that never sends a request. Blocking them as a group treats those jobs as one, and the cost of that is not evenly distributed. This post goes through the six user agent tokens that check A2 tests, GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot and Google-Extended, says what each vendor publicly documents it for, and states what refusing each one actually refuses. Where a detail is not clearly documented, we say less rather than guess. ## Three jobs and a switch The vendors have converged on a similar split, and it is worth naming before the tokens. A **training crawl** gathers content that may be used to train future models. Blocking it is a decision about whether your content contributes to a model's training data. A **search-index crawl** gathers content so that a product can surface and link to your pages when it answers a question. Blocking it is a decision about whether you appear as a source in that product. A **user-triggered fetch** happens when a person asks an assistant about a specific page or site, and the assistant retrieves it in response. Blocking it refuses the person, not the company. A **control token** is a name you can put in `robots.txt` that a vendor reads as a preference, without any crawler of that name ever fetching a page. ## The six tokens A2 scores Check A2 is worth six points, the heaviest single check in the rubric, scored proportionally at one point per token. It reads your `robots.txt` and asks, for each of the six, whether the root path `/` is allowed. A blanket `Disallow: /` for all six scores zero. ### GPTBot OpenAI's crawler. OpenAI documents it as the agent that collects content which may be used to train its models, and documents it separately from its search crawler, so that the two decisions can be made independently. Blocking GPTBot opts your site out of that training collection. It does not, per OpenAI's documentation, govern the search crawler below. ### OAI-SearchBot OpenAI's search-index crawler. OpenAI documents it as the agent used to surface and link to websites in its search features, and states that it is not used to gather training data. Blocking it refuses the index crawl those features depend on. A site that wants to opt out of training while remaining a source can block GPTBot and allow OAI-SearchBot; the two tokens exist so that this is possible. ### ClaudeBot Anthropic's crawler, documented as the agent that collects content for training its models. Blocking it is the same category of decision as blocking GPTBot. ### Claude-User Anthropic's user-triggered fetcher. Anthropic documents it as the agent that retrieves a page when a person asks Claude about that page or site. Blocking it does not affect training or indexing. It refuses the one fetch a person asked for, at the moment they asked for it. This is the token most often blocked by accident. It shares a prefix with ClaudeBot, and a `robots.txt` group written as `User-agent: Claude` matches both in common parser implementations, ours included, which match the token as a prefix. Sites that meant to opt out of training end up refusing their own prospects' requests. If you want to block the crawler and keep the fetcher, name each token in full. ### PerplexityBot Perplexity's crawler, documented as the agent that indexes pages for use in its answer product. Blocking it refuses that index crawl. Perplexity also documents a separate user-triggered fetcher, Perplexity-User, which A2 does not score but which the same accidental-prefix problem applies to. ### Google-Extended Not a crawler. Google documents Google-Extended as a standalone product token that publishers can use in `robots.txt` to manage whether content Google has crawled may be used for training its Gemini models. It does not have its own user agent string; the fetching is done by Googlebot, which is governed by its own rules. Google's documentation also states that the token does not affect a site's inclusion or ranking in Google Search. Blocking Google-Extended, then, costs nothing in search. It is a training-consent decision and only that. It is in the rubric because the rubric scores whether a site is open to the ecosystem of models and agents as a whole, and a site that has opted its content out of one vendor's training is, for that purpose, one sixth less open. Whether that trade is right for you is a policy question we do not try to answer; the check simply records the choice. ## The tokens we name but do not score Our own `robots.txt` names thirteen tokens, not six: the six above plus ChatGPT-User, Claude-SearchBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider and meta-externalagent. A2 scores the six the rubric names. We name the rest because naming a token explicitly is the difference between passing by accident and passing on purpose: a permissive `*` default lets everything in today, and stops doing so the moment someone adds a `Disallow` to it. ## How A2 actually reads your file A2 does not send a request under these user agents. It parses `robots.txt` and applies the Robots Exclusion Protocol rules, which are easy to get subtly wrong. The ones that matter: - User agent matching is case-insensitive and matches on prefix. Consecutive `User-agent` lines share one rule group. - The most specific matching group wins, and `*` is only a fallback. A named group's rules do not merge with the `*` group's rules. If you write a `User-agent: GPTBot` group with a single `Disallow: /private`, GPTBot obeys only that group, and inherits nothing from `*`. - Between `Allow` and `Disallow`, the longest matching path wins. On an exact tie, `Allow` wins. - `Disallow:` with an empty value permits everything. - No `robots.txt` at all means the site is open by default. A1 scores a 404 as partial credit for sloppiness, but A2 scores it as open. A file that opts out of training, stays in search, and keeps user-triggered fetches working looks like this: ```text User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / ``` That file scores three of six on A2, and the scan report lists exactly which three tokens are blocked. If that is deliberate, there is nothing to do, and the recommendation text says so. The check exists to make the choice visible, not to make it for you. ## Necessary, not sufficient A `robots.txt` that allows all six is where the work starts, not where it ends. The file is a request that well-behaved crawlers honour; it does nothing to stop a CDN or a web application firewall from refusing those same user agents at the edge, before `robots.txt` is ever read. That is what check A5 is for. The crawler sends real requests to your homepage under the GPTBot, ClaudeBot and PerplexityBot user agents and records the status code and time to first byte for each. Two points if all three receive a 200 rather than a 403 or 429, and one point if the median time to first byte is under 1.5 seconds. When A2 passes and A5 fails, the report says so explicitly: your `robots.txt` welcomes these agents, and something in front of your origin is refusing them anyway. It is almost always a default the site owner never chose, and [the Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers the most common instance. We have nothing in front of our own origin for exactly this reason. Our [robots.txt](/robots.txt) and our [live report](/site/agentfriendlyrank.com) show both checks together. ## Check your own site The [free scan](/scan) reports A2 and A5 side by side, so you can see whether your `robots.txt` says what you think it says and whether your edge agrees with it. Both checks are defined under [discovery and access](/methodology#pillar-A) on the methodology page. ### Reading your scan report, a check-by-check walkthrough Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/reading-your-scan-report A scan report is a statement about one domain on one date under one version of the rubric. Everything on the page follows from that. The score is not an opinion about your site; it is the sum of 27 checks, each of which shows you what it captured and how many points it awarded. This is a walk through the page from top to bottom, section by section, so that when you open [your own report](/scan) you know what you are looking at and what to do with it. ## The header Under the domain name is a single line that fixes the context of everything below it: the date of the scan, the rubric version it was scored under, and the site type it was scored as. The site type is one of e-commerce, SaaS, content or general, and it matters because check C5 (commerce actionability) only applies to commerce sites. For everyone else its four points are redistributed across C1 to C4, so a non-commerce site can still reach 100. There is also a re-scan link. Results are cached for seven days, so if you have shipped a fix and want to see it reflected, re-scan rather than waiting. ## The score and the four bars On the right of the page is the score dial and, under it, four bars, one per pillar: discovery and access, understanding, actionability, and trust and freshness. Each pillar is worth 25 points, so a bar at full width means that pillar's checks all passed. The bars are the quickest way to see the shape of a problem. A site with three full bars and an empty first bar is usually being blocked at the edge before anything else can be measured. A site with a full first bar and a weak second bar can be fetched but not understood. The total maps to a grade as follows. | Grade | Score | |-------|-------| | A+ | 95–100 | | A | 85–94 | | B | 70–84 | | C | 55–69 | | D | 40–54 | | F | 0–39 | ## What to fix first The first section in the main column is a numbered list of at most five items. It is the same list the scoring engine produces, and the ordering is part of the product rather than a presentation choice. Only checks that failed or scored partial credit are eligible. They are sorted by impact first, from high to low, then by effort, from low to high, and any remaining ties go to whichever check has the most points still on the table. Impact is derived from the check's point weight rather than set by hand, so a five-point check always ranks above a two-point one. Effort is the rating published in the rubric: Low for something you can upload, Medium for template and schema work, High for rendering architecture, machine endpoints or checkout. Each item shows the check id (linked to the methodology page), its name, its impact and effort, and a recommendation written for that specific result. If every applicable check passed, this section is replaced by a note saying there is nothing left to fix. We would rather say that than manufacture a list. The practical rule is simple: work down the list. Anything marked high impact and low effort is a file you can publish this afternoon that moves the score immediately. The [three tiers post](/blog/three-tiers-of-fixes) goes into what each effort band involves. ## Every check, with the evidence Below the fix list is the full run, grouped under the four pillars. Each check is a card with five things on it. The check id, which you can look up in the [methodology](/methodology) to read exactly how it is evaluated. A status pill: Pass, Partial, Fail or Not applicable. The check's name. The points earned against the points possible, for example `2/4`. And, if the status is anything other than Pass, the recommendation. Partial is worth paying attention to. Several checks award points in pieces. A3 gives one point for the sitemap existing, one for it parsing with at least one URL, and one for it being referenced from robots.txt, so `2/3` tells you exactly which piece is missing. A2 is scored proportionally to how many of the six named AI crawlers are permitted, so `4/6` means two of them are being refused somewhere. ### The evidence block Every card carries a collapsed section labelled "What we captured". Open it and you get the raw evidence for that check as JSON: the URLs fetched, the status codes returned, the values parsed, the counts compared against the threshold. This is the part of the page that makes the rest of it trustworthy. If A5 says a bot user agent received a 403, the evidence block shows the request and the response. If B4 says the meta description is too long, the block shows the description and its length. If you disagree with a result, this is where the argument starts, and it is where you should look before assuming the scanner is wrong. The most common surprise in these blocks is an edge or CDN layer answering AI crawlers with a challenge page while robots.txt says everything is allowed; the [two fetches post](/blog/the-two-fetches) covers the other common one. The evidence block is also what you hand to whoever will make the fix. A recommendation says what to change; the evidence says what the crawler actually saw, which is what a developer needs. ## The sidebar The remaining cards on the right are about the report as an object rather than its contents. "Keep this report" emails you the permanent link and the ranked fix list. The report is already public; the form adds to it rather than gating it, and it is one message, not a mailing list. "Want this fixed?" links to the [full audit](/services/agent-readiness-audit) and to a booking page. The automated scan is deliberately shallow, six pages and a fixed set of probes, and the audit is where the judgement comes in. If the grade is B or better there is a badge card. It shows the SVG and the exact markup to copy, a plain image inside a link back to the report. The badge updates when the score does, and it links to the evidence rather than standing alone. "Own this site?" explains that the report comes from a public crawl and lets you claim the domain by proving you control an email address on it. Once claimed, you can hide the report or publish it again yourself. If you would rather we did not scan at all, disallowing `AgentFriendlyRankBot` in robots.txt is honoured automatically, and the page for an opted-out domain says so plainly: no crawl, no score, no report. The [bot page](/bot) documents the crawler. The last card offers a scan of your own site and a comparison view between this domain and another. ## How to decide what to do first Read the fix list, then open the evidence for each item. Three questions settle most of it. Is anything in pillar A failing? If bots are blocked or the raw HTML is empty, nothing further down the page can be seen by an agent, and fixing understanding or trust first is wasted effort. Which items are low effort? Those go first regardless of position, because they are hours of work and they move the score at once. Which high-effort items are actually worth it for your site? Server-rendered parity (A7) is expensive and sometimes decisive. Machine endpoints (C4) is expensive and, for a lot of sites, the cheapest available differentiation because almost nobody has done it. Commerce actionability (C5) matters only if you sell. The report cannot make that call for you; it can only show you the evidence and the ranking. ## Check your own site Run a [free scan](/scan), open the evidence blocks, and read them against the [rubric](/methodology). If the biggest item is in pillar A, start at [/methodology#pillar-A](/methodology#pillar-A); the fix is usually a file or a CDN setting, and it unlocks everything else. ### The two fetches, what an agent sees versus what your browser sees Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/the-two-fetches Every page on your site exists in two versions. The first is the bytes your server sends in response to a GET request. The second is the document object model that exists in a browser after your JavaScript has run. You have only ever looked at the second. Most agents only ever receive the first. The gap between those two versions is where the majority of serious agent readiness findings live, and it is invisible to anyone who tests in a browser, because a browser cannot show you the first version without being asked to. Check A7 in our rubric exists to measure it. This post explains what the two fetches are, why the gap swallows so much of the rest of the rubric, how A7 turns it into a number, and how to compute a rough version of that number yourself in about a minute. ## Fetch one, the raw HTML This is what a plain HTTP client gets. No JavaScript is executed, no further requests are made, nothing is hydrated. It is exactly what `curl` returns: ```bash curl -sL https://example.com/ > raw.html ``` It is also what the browser shows under "view source", which is the one place a browser will show you the first fetch. Whatever is in that file is what a program without a JavaScript engine has to work with: the text, the links, the forms, the JSON-LD, the meta tags, all of it or none of it. If your site is server-rendered, `raw.html` contains the page. If your site is a client-rendered application, `raw.html` contains a `div` with an id, a few script tags, and perhaps a loading spinner. Both look identical in a browser two seconds later. ## Fetch two, the rendered DOM This is what a browser builds. It downloads the raw HTML, runs the scripts, waits for the data they fetch, and assembles the result into a live document. The inspector in your developer tools shows you this version, and only this version. It is the page as you think of it, and it is the page your analytics, your QA process and your screenshots are all based on. Our crawler captures both. Each page is fetched once with a plain client and once more in a headless browser, and the crawl records the visible text from each. Our own [crawler policy](/bot) describes the full procedure. ## Why the gap swallows everything downstream An agent that fetches raw HTML and finds nothing in it does not report a rendering problem. It reports nothing. From its point of view the page is empty, and every other property the page might have had is empty with it. That is why A7 is upstream of most of the rubric. JSON-LD that a client-side script injects after load is absent from the raw fetch, so check B1 sees no schema and check B2 sees no types. A form that a framework renders during hydration is absent, so check C1 finds no inputs to label. Navigation that a component mounts is absent, so check C2 finds no link graph. Heading structure, landmarks, alt text, the price and availability the commerce check C5 compares against schema: every one of them is measured on what the agent receives. The recommendation text the scanner attaches to a zero on A7 puts it directly: schema an agent never receives may as well not exist. It is the single highest-value fix on most reports where it appears, and the most involved. ## How A7 measures it The check is deliberately simple, so that its output can be reproduced by anyone. The scanner takes the homepage's raw HTML, parses it, strips the contents of `script`, `style`, `noscript`, `template` and `svg` elements, and extracts the visible text with whitespace collapsed to single spaces. It takes the visible text from the headless render of the same page and collapses its whitespace the same way. It then divides the length of the first by the length of the second, capped at 1.0. Scoring follows the thresholds published in the rubric. A ratio at or above 0.8 scores the full five points. A ratio at or above 0.5 but below 0.8 scores three. Below 0.5 scores zero, because an agent using a plain fetch sees essentially nothing. The check is rated High effort, because the fix is usually architectural. Two details are worth knowing. First, `noscript` content is stripped along with scripts, so a fallback block inside `noscript` does not count towards the raw side. Second, the ratio is capped: if the raw HTML somehow carries more text than the render, the score is full marks, not more than full marks. The evidence block on a scan report shows three fields for this check, `rawTextLength`, `renderedTextLength` and `ratio`. When the score is three, the recommendation states what proportion of the visible content only appears after JavaScript runs, which is simply one minus the ratio. Our own report at [/site/agentfriendlyrank.com](/site/agentfriendlyrank.com) shows the block for this site, and [reading your scan report](/blog/reading-your-scan-report) walks through the format. A related point about the rubric's design. Server-rendering frameworks commonly inline a serialised copy of the page data inside a script tag so the client can hydrate without refetching. That payload duplicates the page text and can be a large share of the document. Check B5, the clean text ratio, excludes script and style contents from its denominator precisely so that this pattern is not penalised; the changelog entry for rubric version 1.0.1 on [/methodology](/methodology) explains the correction. A7 rewards server rendering and B5 was adjusted so that it does not punish the same architecture. ## Measure it yourself You do not need our scanner to see the gap. The crude version takes a minute. Fetch the raw HTML and strip the tags. This is rougher than the scanner's parser, because it does not remove script contents before counting, but it is enough to show whether the page is there at all: ```bash curl -sL https://example.com/ \ | sed -e 's/]*>[^<]*<\/script>//g' \ | sed -e 's/<[^>]*>//g' \ | tr -s '[:space:]' ' ' \ | wc -c ``` Then open the same page in a browser, open the developer console, and ask for the rendered text length in the same units: ```javascript document.body.innerText.replace(/\s+/g, " ").trim().length ``` Divide the first number by the second. If the result is near 1.0, your content is in the raw HTML and the agent sees what you see. If it is near zero, your homepage is a shell, and everything you have built on it, structured data included, is on the wrong side of the gap. The even quicker check needs no terminal at all. Open the page, use "view source", and search for a sentence you know appears on the page. If it is not in the source, it is not in the first fetch. ## What fixes it The fix is server rendering or prerendering for the routes that matter commercially, not a script tag that promises to help at runtime, because a runtime patch runs on exactly the side of the gap the agent never reaches. Most frameworks now support rendering on the server or at build time; the work is in moving the routes that matter to that mode and keeping them there through the next deploy. It is the most expensive item on a typical report and the one with the largest score movement, which is why our [implementation service](/services/implementation) scopes it against architecture rather than page count. If a full move is not feasible, prerendering the homepage and the handful of pages an agent is most likely to fetch, pricing, product, docs and contact, moves the check that measures those pages. The scanner measures A7 on the homepage, so that is where to start. ## Check your own site The [free scan](/scan) runs both fetches against your homepage and reports the ratio with the raw and rendered text lengths beside it, so you can see the gap rather than take our word for it. A7 is defined in full under [discovery and access](/methodology#pillar-A) on the methodology page. ### The three tiers of fixes, hours, weeks and architecture Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/three-tiers-of-fixes Every check in the [rubric](/methodology) carries an effort rating: Low, Medium or High. The rating is published alongside the point weight, and the scan report uses it to rank fixes so that cheap, valuable work rises to the top. But "Low" and "High" are labels, and this post is about what sits behind them: what the work actually involves, who does it, and roughly what it costs. The short version is that the three tiers are hours, weeks and architecture. Thirteen checks are hours. Eleven are weeks. Three are architecture, and those three are where most of the money and most of the score movement live. ## Low effort, thirteen checks, hours of work These are the checks where the fix is a file you upload, a header you set, or a line you change in a template. Nobody needs to redesign anything. - **A1** robots.txt exists and parses (2 points) - **A2** major AI crawlers allowed (6) - **A3** XML sitemap valid and referenced (3) - **A4** llms.txt present (4) - **A5** bot user-agent HTTP health (3) - **B4** meta and Open Graph (3) - **B6** alt text coverage (2) - **B7** canonical and duplication hygiene (3) - **D1** transport security (4) - **D2** verifiable identity (4) - **D3** policies discoverable (4) - **D5** identity consistency (3) - **D6** security.txt (2) Between them these thirteen checks are worth 43 points. That is close to half the rubric, available for work that a competent developer can complete in a day or two, and in most cases without a deploy of application code at all. A2 is the single biggest item on the whole rubric at six points, and it is often fixed by editing one file. A5 is the one to check before anything else, because it catches edge and CDN layers that refuse bot user agents regardless of what robots.txt says, and the fix is a setting rather than code. A4 needs a real `llms.txt` with three or more described links; a bare list of URLs scores one point, so the work is writing the descriptions, not creating the file. D6 is a text file at `/.well-known/security.txt` with a future-dated `Expires` field. ```text Contact: mailto:security@example.com Expires: 2027-09-04T00:00:00.000Z Preferred-Languages: en ``` The reason this tier matters is not that any one check is large. It is that the whole tier can be done before a longer engagement starts, which is why we do it first and re-scan before touching anything expensive. ## Medium effort, eleven checks, weeks of work These checks require changes across templates rather than in single files, and the changes need to be right on every page the crawler visits, not just the homepage. That is what moves them from hours to weeks. - **A6** no hard interstitial (2 points) - **B1** JSON-LD present and valid (5) - **B2** correct schema types for the site type (5) - **B3** semantic HTML structure (4) - **B5** clean text ratio (3) - **C1** form usability (5) - **C2** navigable link graph (4) - **C3** interactive element semantics (4) - **C6** on-site search (3) - **D4** freshness signals (4) - **D7** multi-page stability (4) Another 43 points. The pattern here is that each check depends on how your templates were built rather than on what files you serve. B1 wants a parseable JSON-LD block on the homepage and on at least half the crawled pages, and B2 wants the right types for your kind of site, so both are a job of wiring structured data into the layout and the page-type templates, then verifying it across the crawl. B3 wants exactly one `h1`, no skipped heading levels and real landmarks; that is often a component refactor rather than a content edit. C1 and C3 are about forms and buttons being what they claim to be: labelled inputs, correct input types, `button` elements rather than click-handled `div`s. C6 wants a search form that uses GET with a query parameter, so an agent can build the URL rather than drive the form. A6 is the odd one out. Removing a cookie wall or a challenge screen is not technically hard, but it usually involves a conversation with legal or with whoever chose the consent vendor, and that is what makes it Medium rather than Low. This tier is where an engagement typically spends most of its time, and where the [implementation service](/services/implementation) delivers structured data across templates rather than page by page, as reviewable pull requests or documented CMS changes. ## High effort, three checks, architecture - **A7** server-rendered content parity (5 points) - **C4** machine endpoints and agent manifests (5) - **C5** commerce actionability (4, commerce sites only) Only 14 points between them, and for a non-commerce site C5's four points are redistributed across C1 to C4, so the tier is worth ten points to a non-commerce site. But the three checks are in a class of their own. A7 compares the visible text in the raw HTML against the fully rendered DOM. A ratio at or above 0.8 scores full marks; below 0.5 scores zero, because an agent using a plain fetch sees essentially nothing. If your content only exists after hydration, no amount of schema or `llms.txt` will help, because the agent never receives the page you decorated. The fix is server rendering or prerendering for the routes that matter, and depending on the framework that can be a configuration change or a rewrite of the rendering layer. This is the most expensive work on the rubric and also the one with the largest downstream effect, because most of pillars B and C are measured on what the raw fetch returns. C4 asks for any of an `agents.json`, an MCP or WebMCP manifest, a published API documentation link, or a discoverable OpenAPI spec. It is High effort because it involves building something that did not exist rather than fixing something that did. It is also the check where, in the scans we run, the usual result is zero, which is why we cover it [separately](/blog/why-most-sites-score-zero-on-machine-endpoints). C5 is checkout: product pages within two clicks of the homepage, price and availability in schema matching the page, a cart or checkout path returning 200 without login, and a guest or express checkout signal. On most commerce platforms that touches the part of the stack people are most nervous about changing. ## What it costs We scope engagements by architecture rather than by page count, because the tiers above are what determine the work and a fifty-page site with client-rendered templates is a bigger job than a five-thousand-page site with good server rendering. An [agent readiness audit](/services/agent-readiness-audit) runs every check by hand, attaches the captured evidence to each finding, and delivers a remediation plan ranked by impact against effort. It takes five business days and starts from $2,000. [Implementation](/services/implementation) is typically $4,000 to $25,000 over two to eight weeks, again depending on architecture. Both are on the [pricing page](/pricing). We will also say when something is not worth doing. A content site with no forms does not need C1 work, and a site that already renders on the server should not be sold an A7 project. An audit that recommends everything has not done its job. ## Check your own site The [free scan](/scan) lists your failed checks with their effort ratings, so you can see at a glance how much of your gap is hours and how much is architecture. For the checks in each tier, see the rubric's [pillar A](/methodology#pillar-A), [pillar B](/methodology#pillar-B), [pillar C](/methodology#pillar-C) and [pillar D](/methodology#pillar-D) tables. ### What agent readiness actually means, in one page Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/what-agent-readiness-means Agent readiness is whether a piece of software, acting on a person's behalf, can use your website. Not visit it, not rank it, use it: fetch a page, work out what the page is about, do the thing the person asked for, and decide whether what it found is worth trusting. Those are four separate questions, and a site can answer any of them with a no while looking perfectly healthy in a browser and in a search console. We score the four questions out of 100, 25 points each, across 27 checks. The whole rubric is published on [/methodology](/methodology), and this page is the short version of it. ## Four questions, 25 points each The rubric groups its checks into four pillars. Each pillar is a single question, phrased in the rubric exactly as it appears below, and the checks under it are the evidence we collect to answer it. ### A. Discovery and access *Can agents find and reach the content at all?* Seven checks, 25 points. Whether `robots.txt` exists and parses (A1), whether it permits the six major AI crawlers (A2, six points, the heaviest single check in the rubric), whether the XML sitemap is valid and referenced (A3), whether an `llms.txt` is present with described links (A4), whether bot user agents actually receive a 200 rather than a 403 or 429 (A5), whether the page is dominated by a CAPTCHA, challenge or cookie wall (A6), and how much of the visible text is present in the raw HTML before any JavaScript runs (A7). Everything else in the rubric depends on this pillar. A site an agent cannot reach has no schema, no forms and no trust signals, whatever it contains. ### B. Understanding *Once fetched, can a machine parse what this site is and sells?* Seven checks, 25 points. Parseable JSON-LD on the homepage and on at least half the crawled pages (B1), schema types that match the kind of site it is (B2), one `h1`, no skipped heading levels and real landmarks (B3), a title, description and Open Graph tags within sane lengths (B4), a clean text ratio of at least 0.10 (B5), alt text on at least 80% of content images (B6), and canonical hygiene across www, non-www, http and https (B7). Most of this pillar will look familiar to anyone who has done technical SEO. That is not a coincidence, and it is also where the familiarity ends. ### C. Actionability *Can an agent actually do things — navigate, search, fill forms, transact?* Six checks, 25 points. Forms with labelled inputs, correct input types and a real submit button (C1), navigation built on anchor elements rather than click-handled `div`s (C2), buttons that are `button` elements and can be reached by keyboard (C3), any machine endpoint at all, whether an `agents.json`, an MCP manifest, API documentation or an OpenAPI spec (C4), commerce paths that work without a login (C5, commerce sites only, redistributed across C1 to C4 for everyone else), and on-site search that uses GET with a query parameter so an agent can build the URL itself (C6). There is no search engine equivalent of this pillar. Nobody has ever been ranked on whether a program could submit their contact form. ### D. Trust and freshness *Should an agent — or the model behind it — trust and cite this site?* Seven checks, 25 points. Valid TLS, an HSTS header and no mixed content (D1), a contact page plus Organization schema with `sameAs` links (D2), a linked privacy policy and the terms, shipping or returns pages the site type calls for (D3), sitemap `lastmod` values with at least 20% of URLs modified in the last 90 days (D4), a site name that agrees across the title, `og:site_name` and schema (D5), a valid `security.txt` with a future `Expires` field (D6), and every crawled page returning 200 in under four seconds with a consistent template (D7). The pillar's premise is that a model deciding whether to cite a page weighs provenance, and provenance is checkable. None of these signals is about reputation. Each is a file, a header or a field that either exists or does not. ## Why a site can pass every SEO test and fail all four The search engine you have optimised for since the 2000s renders your pages in a headless browser, obeys a `robots.txt` you wrote with its named crawler in mind, and rewards you with a position in a list. Its tests are the right tests for that job. An agent is doing a different job, and its tests fail in places a search console never looks. It usually fetches the raw HTML and does not execute your JavaScript. If your content only exists after hydration, the agent sees a shell. That is check A7, and it sits upstream of every other check in the rubric, because schema an agent never receives may as well not exist. It identifies itself with a user agent string your `robots.txt` may never mention, and a CDN with a default bot-blocking rule can refuse it before `robots.txt` is read at all. That is the gap between A2 and A5, and the reason a site can have a perfect `robots.txt` and still be closed. [The Cloudflare post](/blog/cloudflare-blocks-ai-crawlers-by-default) covers the most common version of it. It then tries to act. A form whose submit is bound to an Enter key handler, a navigation menu made of `div`s with click listeners, a search box that POSTs, a cart that demands a login before it returns a page: each is invisible to a search engine and a dead end for an agent. And when it fails, it does not report an error or drop you a position. It uses a site it could read, and nobody on your side finds out. That is the property that makes this problem different in kind from SEO, and the reason we score the four questions rather than one. [Agent readiness is not SEO, and not GEO either](/blog/agent-readiness-is-not-seo) goes into the overlap and the divergence properly. ## What the score is, and what it is not The score is out of 100 and maps to a grade: 95 to 100 is an A+, 85 to 94 an A, 70 to 84 a B, 55 to 69 a C, 40 to 54 a D, and anything below 40 an F. Every check carries an effort rating, Low, Medium or High, so the fixes can be ranked by cost as well as by points. [The three tiers of fixes](/blog/three-tiers-of-fixes) explains what each rating involves. The rubric is versioned. This page describes version 1.0.1. When a check changes, the version is bumped, the change is logged on [/methodology](/methodology), and scores issued under the old version are never restated. [Why we publish the rubric](/blog/why-we-publish-the-rubric) is the argument for doing it this way. The free scan is deliberately shallow: six pages, a fixed set of probes, no judgement. It is good at telling you that something is wrong and roughly how much it costs you. Every finding it reports carries the evidence it captured, and a finding with no evidence is not published. If you want the same checks run by a person who reads your templates rather than inferring them, that is the [paid audit](/services/agent-readiness-audit). We are scored against this rubric ourselves, and the result is public at [/site/agentfriendlyrank.com](/site/agentfriendlyrank.com), including anything we have not yet fixed. Selling agent readiness from a site nobody had audited would be absurd. ## Check your own site Run the [free scan](/scan) and you will get a score against these four questions, check by check, with the evidence behind each one. The full definitions are on the methodology page, starting with [discovery and access](/methodology#pillar-A) and continuing through [understanding](/methodology#pillar-B), [actionability](/methodology#pillar-C) and [trust and freshness](/methodology#pillar-D). ### Why most sites score zero on machine endpoints, and why that is the opportunity Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/why-most-sites-score-zero-on-machine-endpoints Check C4 in the [rubric](/methodology#pillar-C) is worth five points and asks a simple question: does this site publish anything a program can use directly, rather than having to scrape? Any one of an `agents.json`, an MCP or WebMCP manifest, a published link to API documentation, or a discoverable OpenAPI spec earns credit. More than one earns the full five. When we scan sites, the usual result on C4 is zero. Not a partial, not a near miss, zero. That is unusual on a rubric where most checks fail by degrees, and it is the reason C4 separates the leaders: it is the one check where doing the work at all puts you in a small group. ## What the check is actually asking for The four things C4 accepts are different in kind, which is worth understanding before deciding which to build. An `agents.json` is a manifest at a well-known location that describes the site to an agent: what it is, how to contact the operator, where the documentation lives, how to search it, how the operator would like to be crawled, and which endpoints exist. It is a description, not an interface. An OpenAPI spec describes an HTTP API precisely enough that a client can be generated from it. It is a settled standard with a decade of tooling behind it, and a great many organisations already have one for internal use. Publishing a read-only subset at a discoverable URL is the smallest step on this list for anyone who already runs an API. A published API documentation link is the human-readable counterpart. The check wants it to be linked, meaning findable from the site rather than sitting on a subdomain nothing points to. An MCP or WebMCP manifest exposes capabilities as tools a model can call. MCP servers exist and are consumed by client software today; WebMCP is a proposal for doing the same from inside a web page, and it is still moving. ## Why almost nobody has done it There is no mystery here. Four reasons account for nearly every zero we see. Nobody's checklist asks for it. Twenty years of SEO produced a well-understood list of things a site should have, and every audit tool checks that list. Machine endpoints are not on it, because search engines never needed them. A site can pass every SEO test ever written and still have nothing on C4; the [difference between agent readiness and SEO](/blog/agent-readiness-is-not-seo) is largely this. It needs an engineer, not a marketer. Most of the rubric can be moved by whoever owns the templates or the CMS. C4 is the one check that involves publishing an interface, which means a developer, a decision about what to expose, and a maintenance commitment. That is why it carries a High effort rating. The standards are unsettled. `agents.json` has no single governing specification. WebMCP is a proposal. Even where MCP is being consumed, the conventions for discovering a server from a domain are still forming. A sensible engineering lead, asked to publish something against a moving target, will often wait, and waiting is a defensible choice. There is no visible cost to not doing it. A site without an OpenAPI spec looks identical to one with it, in a browser and in analytics. The agent that could not act on the site does not report an error. It uses a site it could act on, and nobody on your side finds out. ## Why that makes it the cheapest differentiation Five points is not a large share of 100. The value of C4 is not the points; it is the distribution. On most of the rubric, the sites you compete with are somewhere on a curve. Everyone has a robots.txt, most have some structured data, a fair number have alt text. Moving from partial to full credit on B1 puts you a little ahead of a lot of sites. Moving from zero to five on C4 puts you ahead of nearly every site we have scanned on that check, which is a different kind of gap from a few points on a check everyone half-passes. It is also, unusually for a High effort check, bounded. Fixing A7 on a client-rendered site can mean re-architecting the rendering layer for every commercial route. Publishing a read-only API with an OpenAPI document is a scoped piece of work with a clear end, and for organisations that already have an API it can be a matter of writing the descriptions and choosing a URL. ## What we published, and which parts are a bet It would be indefensible for us to score C4 and not publish anything ourselves, so we did all four, and they are worth reading as a worked example rather than as a template. [`/agents.json`](/agents.json) is our manifest. It carries the site's name and legal operator, contact addresses for general, security and privacy enquiries, links to `llms.txt`, `llms-full.txt` and the methodology with the current rubric version, a search URL template an agent can construct without touching a form, a crawling section that documents our own bot's user agent, limits and opt-out, a list of endpoints, and links to the privacy, terms and data-processing policies. The endpoints list only what actually exists. It was deliberately empty until the API shipped, because a manifest that advertises things that do not exist is worse than none. ```bash curl -s https://agentfriendlyrank.com/agents.json | jq '.endpoints[].type' ``` [`/api/v1/openapi.json`](/api/v1/openapi.json) is an OpenAPI 3.1 document for the read-only API: the rubric, the leaderboard, per-site reports, comparisons and scan status, with no key required. It is hand-written rather than generated, because the value of the document to an agent is in the descriptions it reads to decide what to call, and no generator writes those for you. The path list is unit-tested against the route files so the two cannot drift. `/mcp` is an MCP server over Streamable HTTP, no authentication, exposing the same data as five tools: `get_site_report`, `get_leaderboard`, `compare_sites`, `get_scan_status` and `get_rubric`. [`/docs`](/docs) is the human-readable documentation for both, and [`/llms.txt`](/llms.txt) points at all of them under a "Notes for agents" heading, so that a program which reads only the text file still finds the machine surfaces. Now the honest part. OpenAPI is settled; publishing one is not a bet. MCP has real clients today, but the convention for discovering a server from a bare domain is not fixed, which is why our `agents.json` and `llms.txt` both point at it explicitly. `agents.json` itself is the least settled of the four: our file declares `schemaVersion: "0.1"` because there is no authoritative version to declare, and the field names are ours. WebMCP we have not built, because it is a proposal and we would rather not build to a draft and call it done. When any of this settles, the rubric will change, the [changelog](/methodology#changelog) will say what changed, and historical scores will stay as they were. ## What to build first If you already run an API, publish an OpenAPI document for a read-only subset at a discoverable URL and link the documentation from your site. That is one thing, it is a settled standard, and it already earns C4 credit. If you do not, an `agents.json` that describes the site honestly and lists nothing that does not exist is a small piece of work. Pair it with a documentation page and you have more than one surface, which is the full five points. Treat MCP as the third step, once there is something worth exposing as tools. Treat WebMCP as something to watch. The [implementation service](/services/implementation) builds any of these, and we say plainly which parts are a bet before we start. ## Check your own site Run a [free scan](/scan) and open the evidence block on C4; it shows exactly which of the four surfaces the crawler looked for and what it found. The full definition is in [pillar C of the methodology](/methodology#pillar-C). ### Why we score out of 100 and publish the whole rubric Foundations · Published 4 September 2026 · https://agentfriendlyrank.com/blog/why-we-publish-the-rubric A score you cannot audit is a score you should not trust. That sentence sits at the top of our [methodology page](/methodology) and it is the reason the page exists at all. Every check we run, its point weight, its threshold and the effort we think the fix takes is published there, and every scan we issue records which version of that rubric scored it. This post is the argument for doing it that way. It is not a complicated argument, but it runs against how most "AI visibility" products work, so it is worth setting out plainly. ## What a proprietary score hides The typical AI visibility score is a single number, a colour and a button. The number is produced by a model of some kind, the inputs are not disclosed, and the thresholds are not disclosed either. You cannot tell whether a 62 means you are missing structured data or whether the vendor's crawler was blocked at the edge and scored the resulting error page. You cannot tell whether the number would be different tomorrow because your site changed or because the vendor's weighting did. That opacity is not an accident. A score nobody can check is a score nobody can argue with, and an audit business that cannot be argued with sells more audits. It also means the score can drift with the vendor's commercial interests, and nobody outside the vendor would know. We think the right response is not a better black box. It is no box. ## What publishing costs us Publishing the rubric means you can read check A2 and see that we test six named crawlers, score proportionally, and give zero for a blanket disallow. You can read B5 and see the exact ratio thresholds. You can read C4 and see that one machine endpoint earns partial credit and more than one earns the full five points. It also means you can disagree with us, specifically. You might think four points for `llms.txt` is generous for a convention that is still settling. You might think five points for server-rendered parity is not enough given how much depends on it. Those are real arguments, and we would rather have them in the open than have you quietly distrust a number you cannot interrogate. Most audit businesses depend on you not being able to name the check you disagree with. Ours depends on you being able to. The other cost is that the rubric is a single definition, in one file, feeding both the scan engine and the published page. The methodology page is rendered from the same structure that scores every report, so what we publish and what we measure cannot disagree. That constraint is inconvenient at times. It is also the whole point. ## Why the rubric is versioned Standards in this area are moving. `llms.txt` adoption, MCP and WebMCP, the various agentic commerce proposals: all of them will change what a reasonable check looks like, and a rubric that never changes would be measuring 2025 forever. So the rubric carries a version, and every scan stores the version it was scored under. When the rubric changes, the change is dated and explained in the [changelog](/methodology#changelog), and scores already issued are left alone. A site that scored 71 under one version does not become 68 because we tightened a threshold six weeks later. If you want to know what the site would score today, you re-scan it, and the new report says which version it used. This matters more than it sounds. The leaderboard and the badge only mean anything if a score is a fixed statement about a site on a date under a named rubric. A number that can be retroactively rewritten is not evidence; it is an opinion with a timestamp. ### What changed between v1.0.0 and v1.0.1 The rubric has already changed once, and the change is a useful example of why versioning is not a formality. Check B5, clean text ratio, measures visible text as a proportion of HTML bytes. It is there to penalise div soup and megabyte pages. Under v1.0.0 the denominator was every byte of the document. When we audited our own site we found that our homepage's serialised server-rendering payload, which duplicates the page text so the browser can hydrate it, was 60% of the document. Measured against total bytes, B5 was penalising precisely the server-rendered architecture that check A7 rewards. It was measuring framework choice, not markup quality. Version 1.0.1 excludes the contents of `script` and `style` elements from the denominator. The changelog entry says so, says why, and says that scores issued under v1.0.0 are not restated. That is the whole protocol: explain the delta, do not touch the history. ## Why we score ourselves in public We sell agent readiness. The first thing a sensible prospect does after reading a cold email from us is scan us, so [our own report](/site/agentfriendlyrank.com) is public, scored by the same engine, with the same evidence blocks, including anything we have not yet fixed. Internally we treat a drop in our own score as a broken build rather than a backlog item. That rule has practical consequences for how this site is built: server components only, no content that exists solely after hydration, cookieless analytics so there is no consent banner to count as an interstitial, no third-party embeds, and nothing sitting in front of the origin that might answer a bot with a challenge page. Every one of those is a rubric check in disguise. Scoring ourselves is also how we found the B5 problem above, and [reading your scan report](/blog/reading-your-scan-report) walks through the same evidence blocks on any report. A rubric you only ever point at other people is a rubric you never have to defend. ## Where the 100 comes from The rubric asks four questions, and each is worth 25 points. Can agents find and reach the content at all (pillar A, discovery and access)? Once fetched, can a machine parse what the site is and sells (pillar B, understanding)? Can an agent actually do things, navigate, search, fill forms, transact (pillar C, actionability)? Should an agent, or the model behind it, trust and cite this site (pillar D, trust and freshness)? Twenty-seven checks sit under those four questions. Every one is automatable from an unauthenticated crawl and returns evidence we captured rather than an opinion. The total maps to a grade: A+ for 95–100, A for 85–94, B for 70–84, C for 55–69, D for 40–54, F for 0–39. Grade B or better earns a badge that links back to the report it came from, because a badge that cannot be traced to its evidence would be exactly the kind of number we are arguing against. Equal weighting across the four pillars is a deliberate choice, not a measured one. We do not have the aggregate data yet to say that discovery matters more than trust, and until we do, we would rather be transparently even-handed than falsely precise. ## Check your own site Run a [free scan](/scan) and read the result against the [published rubric](/methodology), starting with [pillar A](/methodology#pillar-A), since nothing else can be measured if agents cannot get in. If a check scored you wrongly, the methodology page tells you how to dispute it, and we will publish the correction. --- ## Crawler policy Our crawler identifies itself as `AgentFriendlyRankBot/1.0 (+https://agentfriendlyrank.com/bot)` and never impersonates a browser. Hard limits: 6 pages and 6 file probes per audit, 1 concurrent request per host, 10 second timeout, 1 re-scan per domain per day. We obey robots.txt for our own user agent. To block us entirely, add `User-agent: AgentFriendlyRankBot` / `Disallow: /`. ## Contact - General: hello@agentfriendlyrank.com - Data protection officer: dpo@agentfriendlyrank.com - Security: security@agentfriendlyrank.com - Postal: Socio360, 2nd Floor, #829, 26th Main, HSR Layout, Bengaluru 560102, India - GST: 29DVCPS4841F1ZA