Rubric v1.1.0
How we score agent readiness
The whole rubric, published. 27 checks across four pillars worth 25 points each. Every check is automatable from an unauthenticated crawl and returns evidence we captured, never an opinion.
We publish this for a simple reason: a score you cannot audit is a score you should not trust.
Discovery & access
25 pointsCan agents find and reach the content at all?
| ID | Check | Points | Fix effort | How it is evaluated |
|---|---|---|---|---|
| A1 | robots.txt exists and parses | 2 | Low | Returns 200 with valid directives. A 404 scores partial credit — the site is open by default, but the omission is sloppy. |
| A2 | Major AI crawlers allowed | 6 | Low | Allow/deny tested for thirteen documented tokens, weighted by what each is for rather than counted alike: three points for user-initiated agent fetches (ChatGPT-User, Claude-User, Perplexity-User), two for search and citation indexing (OAI-SearchBot, Claude-SearchBot, PerplexityBot), one for model training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent). Each class is scored proportionally within itself. Declining training alone — an increasingly common and entirely defensible licensing position — costs at most one point; blocking everything scores zero. |
| A3 | XML sitemap valid and referenced | 3 | Low | One point for existing, one for parsing with at least one URL, one for being referenced from robots.txt. |
| A4 | llms.txt present | 4 | Low | An llms.txt at the root opening with an H1 and carrying three or more described links scores three points; an llms-full.txt adds one. Described links without an H1 score two, matching what Lighthouse's Agentic Browsing audit requires. An empty or boilerplate file scores one. |
| A5 | Bot user-agent HTTP health | 3 | Low | Every probed bot user agent — GPTBot, ClaudeBot and PerplexityBot, plus the user-initiated ChatGPT-User and Claude-User — receives a 200 rather than a 403 or 429, plus one point for a median TTFB under 1.5 seconds. This is where edge and CDN blocking surfaces, regardless of what robots.txt says. |
| A6 | No hard interstitial | 2 | Medium | The rendered page is not dominated by a CAPTCHA, challenge screen or cookie wall — detected from known challenge markup and from any overlay covering more than half the viewport. |
| A7 | Server-rendered content parity | 5 | High | Visible text in the raw HTML compared against the fully rendered DOM. A ratio at or above 0.8 scores full marks; below 0.5 scores zero, because an agent using a plain fetch sees essentially nothing. |
Understanding
25 pointsOnce fetched, can a machine parse what this site is and sells?
| ID | Check | Points | Fix effort | How it is evaluated |
|---|---|---|---|---|
| B1 | JSON-LD present and valid | 5 | Medium | At least one parseable application/ld+json block on the homepage, and on half or more of the crawled pages. Invalid JSON scores zero for that page. |
| B2 | Correct schema types for the site type | 5 | Medium | Organization or WebSite sitewide, plus type-specific markup: Product and Offer with price and availability for commerce; Article, FAQPage or SoftwareApplication for SaaS and content sites. |
| B3 | Semantic HTML structure | 4 | Medium | Exactly one h1 per page, no skipped heading levels, and real landmarks — main, nav and footer, or their ARIA equivalents. |
| B4 | Meta and Open Graph | 3 | Low | A unique title of 15–70 characters, a meta description of 50–160, and og:title, og:description and og:image all present. |
| B5 | Clean text ratio | 3 | Medium | Visible text as a proportion of HTML bytes, excluding the contents of script and style elements. At or above 0.10 scores full marks, 0.05–0.10 scores two. This penalises div soup and megabyte pages. Script contents are excluded deliberately: server-rendering frameworks inline a serialised payload that duplicates the page text, and counting it would penalise the server-rendered architecture check A7 rewards. |
| B6 | Alt text coverage | 2 | Low | Content images — icons under 50px excluded — carrying non-empty alt text. 80% or better scores full marks. |
| B7 | Canonical and duplication hygiene | 3 | Low | A self-consistent rel=canonical, plus www/non-www and http/https all resolving to one canonical host. |
Actionability
25 pointsCan an agent actually do things — navigate, search, fill forms, transact?
| ID | Check | Points | Fix effort | How it is evaluated |
|---|---|---|---|---|
| C1 | Form usability | 5 | Medium | Every input carries a label or aria-label, input types are correct (email, tel, number), and submit is a real button. A site with no forms at all scores partial credit. |
| C2 | Navigable link graph | 4 | Medium | Navigation uses real anchor elements rather than click-handled divs, fewer than 5% of links use contextless text like 'click here', and there are breadcrumbs or a clear URL hierarchy. |
| C3 | Interactive element semantics | 4 | Medium | Buttons are button elements rather than clickable divs, interactive elements are focusable with sane tab order, and no keyboard traps appear under render. |
| C4 | Machine endpoints and agent manifests | 5 | High | Any of an agents.json, an MCP manifest or server card, a WebMCP tool registration via navigator.modelContext, a published API documentation link, or a discoverable OpenAPI spec. More than one scores full marks. Most sites score zero here today — which is precisely why it separates the leaders. |
| C5 | Commerce actionability | 4 | High | Commerce sites only. Product pages within two clicks of the homepage, price and availability in schema matching the page, a cart or checkout path returning 200 without login, and a guest or express checkout signal. Redistributed across C1–C4 for non-commerce sites. |
| C6 | On-site search | 3 | Medium | A search form exists, and it uses GET with a query parameter so an agent can construct search URLs directly rather than having to drive a form. |
Trust & freshness
25 pointsShould an agent — or the model behind it — trust and cite this site?
| ID | Check | Points | Fix effort | How it is evaluated |
|---|---|---|---|---|
| D1 | Transport security | 4 | Low | Valid TLS, an HSTS header, and no mixed content under render. |
| D2 | Verifiable identity | 4 | Low | A contact page or discoverable contact details, plus Organization schema with sameAs links to at least one social or registry profile. |
| D3 | Policies discoverable | 4 | Low | A linked privacy policy, plus shipping and returns pages for commerce sites or a terms page otherwise. |
| D4 | Freshness signals | 4 | Medium | Sitemap lastmod present, at least 20% of URLs modified in the last 90 days, and visible dates on articles or a changelog. |
| D5 | Identity consistency | 3 | Low | The site name agrees across the title element, og:site_name and the schema name, and both favicon and og:image resolve. |
| D6 | security.txt | 2 | Low | A valid /.well-known/security.txt per RFC 9116, including a future-dated Expires field. |
| D7 | Multi-page stability | 4 | Medium | Every crawled page returns 200, no page exceeds 4 seconds TTFB, and template and navigation stay consistent across pages. |
Grading
| Grade | Score |
|---|---|
| A+ | 95–100 |
| A | 85–94 |
| B | 70–84 |
| C | 55–69 |
| D | 40–54 |
| F | 0–39 |
How fixes are prioritised
Every failed check carries an impact (roughly its point weight) and an effort: low for a file you can upload, medium for template and schema work, high for rendering architecture or checkout. Reports rank the top five by impact descending and effort ascending, so the cheap, high-value work is obvious.
Versioning and disputes
Every scan stores its rubric version. Standards are moving: llms.txt adoption, WebMCP, agentic commerce protocols. So the rubric will change, but historical scores are never rewritten. If you think a check scored your site wrongly, tell us and we will publish the correction.
Signals we watch but do not score
A check earns points when the standard behind it has settled enough that scoring it will not need restating next quarter, and when we have shipped it on this site ourselves. Until both are true it belongs here, with the reason stated, rather than silently absent.
Markdown content negotiation
The same URL returns a clean markdown representation when asked for one with Accept: text/markdown, with Content-Type: text/markdown and Vary: Accept on the response, and often a parallel .md URL as well. Cloudflare converts at the edge as an opt-in beta on its paid plans (February 2026); Vercel serves its own documentation this way and publishes the pattern; Cloudflare reports that Claude Code and OpenCode already send the header.
Not scored: The strongest candidate for the next scored check, and the probe is clear: request the homepage with Accept: text/markdown and credit a text/markdown response that carries Vary: Accept. Markdown served on user-agent sniffing alone would earn nothing — it is invisible to any client that does not spoof, and it is the dynamic-rendering pattern that check A7 exists to discourage. It is not going in the rubric until this site serves markdown itself.
Content Signals and IETF AIPREF
Cloudflare's Content Signals Policy (search / ai-input / ai-train) is deployed across millions of managed robots.txt files; the IETF AIPREF working group's Content-Usage header and its robots.txt binding remain Internet-Drafts.
Not scored: Declaring a usage preference is a licensing decision, not a readiness one — a site can state ai-train=no and still be exemplary for agents. Both syntaxes are parsed and reported verbatim in A2's evidence, and neither moves the score. Revisited if AIPREF reaches RFC.
MCP server cards and .well-known discovery
SEP-2127 (an open draft since January 2026, now an experimental extension) reserves GET <mcp-url>/server-card, while early adopters and the proposal thread itself publish cards under /.well-known/mcp/ with more than one filename in use.
Not scored: Two incompatible locations and an unmerged draft. C4 already credits an MCP reference; pinning a path now would score sites against a convention that has not settled.
WebMCP
A W3C Community Group draft with a Chrome origin trial and a zero-weighted Lighthouse audit. Real deployment is close to nil.
Not scored: Credited in C4 when detectable, but not required. Nothing that needs an origin trial token belongs in a check a small business is expected to pass.
Agentic commerce protocols
OpenAI's Agentic Commerce Protocol, Google's Universal Commerce Protocol and x402 are all live enough to matter to merchants and not yet convergent.
Not scored: C5 measures whether a human-shaped checkout is machine-traversable, which is the question every merchant can act on today. A protocol check waits for one of the three to win.
Rubric changelog
- v1.1.0
First review of the rubric against the ecosystem since it was published, and the changes are corrections rather than additions. A2 now weights crawler tokens by purpose instead of counting thirteen of them alike: three points for user-initiated agent fetches, two for search and citation indexing, one for model training. Every major operator separated those tokens during 2026 — Anthropic split ClaudeBot, Claude-User and Claude-SearchBot in February — and the flat count meant a site running Cloudflare's managed default of allow-search, block-training lost a third of the check for a deliberate licensing position that leaves it perfectly reachable. That measured copyright policy and called it agent readiness, which is the same error v1.0.1 corrected in B5. The scored token list grew from six to thirteen in the same pass; it had fallen behind the list this site's own robots.txt names. A5 now probes ChatGPT-User and Claude-User alongside the three indexers, because from 15 September 2026 Cloudflare's defaults block the agent category on ad-bearing pages and nothing in the rubric tested that path. A4 requires an H1 for full marks, matching the llms.txt audit Lighthouse 13.3 shipped in May 2026. C4 credits a WebMCP registration. A2's evidence now reports any Content-Signal or Content-Usage declaration in robots.txt verbatim, unscored. Scores issued under earlier versions are not restated.
- v1.0.1
B5 (clean text ratio) now excludes the contents of script and style elements from the denominator. Found by auditing our own site: modern server-rendering frameworks inline a serialised payload duplicating the page text, which on our homepage was 60% of the document. Measured against total bytes, the check penalised precisely the server-rendered architecture that check A7 rewards — so it was measuring framework choice rather than markup quality. Scores issued under v1.0.0 are not restated.
- v1.0.0
Initial published rubric. Four pillars, 25 points each, 27 checks.