A publisher is the one kind of site where blocking an AI crawler protects something. The text is the product, and keeping it out of a vendor's training data is a decision about an asset. It is also the site type most likely to have blocked the wrong thing, because "block the AI bots" arrived as one switch and the six tokens check A2 scores do three different jobs. The real decision is not AI or no AI. It is whether to be training data, whether to be a citable source, and whether to be readable when a person asks about one article, and those are three separate groups in robots.txt. This post takes that decision first, then Article markup, dates under D4, and what a paywall does and does not do under A6.
The decision, token by token
A2 is six points, one for each token allowed at the root. The tokens sort into three jobs.
| Job | Tokens | What a block gives up |
|---|---|---|
| Training crawl | GPTBot, ClaudeBot, Google-Extended | Your articles' contribution to that vendor's future models |
| Search index | OAI-SearchBot, PerplexityBot | Being fetched, indexed and cited as a source in that product |
| Fetch for a reader | Claude-User | The one article a person just asked an assistant about |
Being cited without being scraped means blocking the first row and allowing the other two. OpenAI documents GPTBot and OAI-SearchBot as separate crawlers for exactly this purpose, and states that the search crawler is not used for training; blocking GPTBot but not OAI-SearchBot is that file line by line. The publisher row of the six AI crawlers scores it: three of six on A2, with the evidence listing exactly which three are blocked and the recommendation saying that if the block is deliberate there is nothing to do.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: *
Allow: /
Two boundaries. Claude-User is a reader, not a crawler: a person looking at your article asked for a summary, and the fetch is that request. Blocking it refuses the reader, not the vendor. And robots.txt is a request, not a firewall. A CDN that refuses these user agents at the edge overrides the file, usually refuses the search crawlers along with the training ones, and shows up under A5 rather than A2.
Article schema, and how a publisher is typed
Site type is decided from the homepage's raw HTML. A publisher is typed content when the homepage JSON-LD contains an Article or a BlogPosting. A homepage that is a list of headlines with no schema of its own is typed general instead. The difference matters for B2: a content site earns its three type-specific points for Article, BlogPosting or FAQPage, while a general site earns them for Article, FAQPage, Service or SoftwareApplication. BlogPosting alone is enough only if the site was typed content. Emitting Article on every article earns the points under either type, and an Article for the lead story in the homepage's JSON-LD, beside the WebSite block, is what makes the type content rather than general.
An Article block needs headline, datePublished, dateModified, author and publisher, with the publisher an Organization carrying sameAs links so D2 can corroborate it and D5 can match its name to the title element. The two dates do double duty under D4. Emit the block server-side, on the article template; a block injected by a tag manager after load is invisible to a plain fetch and to B1's parser.
Dates, and the 90-day window
D4 is four points. One for a sitemap with any lastmod at all, two for at least 20% of the sitemap's URLs having a lastmod within the last 90 days, and one for a machine-readable date on any crawled page, meaning a time element with a datetime=" attribute or the literal text "datePublished" or "dateModified", which is what those keys look like inside an Article block. Freshness signals goes through the computation and the reasons a build-time stamp is worse than no date.
The Article block above clears the visible-date point, provided a dated page is among the six crawled; a date on the homepage makes that certain. The 90-day ratio is where a publisher's shape matters. A weekly publication clears 20% on new URLs alone in any sitemap of up to about 65 entries. A large archive does not, because the ratio is against every URL in the file the crawler read, and the file it reads is the first one declared in robots.txt. If that file is a sitemap index, its entries are child sitemap locations with no lastmod at all, so as of v1.0.1 the first three points are lost however well the child files are dated. The remedy is ordering: declare the sitemap that lists recent articles with real dates first, and the index after it. XML sitemaps for agents has the file. And keep lastmod honest: it is the date the page changed, per the sitemaps protocol, not the date the site was deployed.
Paywalls and interstitials under A6
A6 is two points and the mechanism, as the engine implements it, is a set of markup patterns tested against the rendered homepage HTML, or the raw HTML when no render was captured. The patterns are a Cloudflare challenge, a reCAPTCHA or hCaptcha script, an element id beginning onetrust or the strings osano, cookieyes, cookie-consent, cookie-banner or cc-window anywhere in the markup, the text "Just a moment...", and the sentence "Please enable JavaScript to continue" or "to view". Any match other than the cookie pattern scores zero; a cookie pattern alone scores one of two.
A paywall matches none of those patterns. A metered page that renders the article body in the HTML and draws a subscription prompt over it after a few visits is not an interstitial by this check, and the agent reading the raw HTML gets the article. That is the model to keep. What costs a publisher under A6 is the consent banner, which the cookie pattern catches on the homepage, and any bot-challenge product placed in front of the whole site, which is a zero and a closed door for every crawler in the table above. Cookie walls are interstitials sets out the cookieless route.
The paywall that does cost you is the one that removes the body from the HTML and fetches it after a subscription check. A6 runs on the homepage only and does not see it. What sees it is the assistant a reader asked about the article, which receives a headline and nothing else. A7 measures the same gap on the homepage; on an article page you measure it yourself, and a hard paywall that serves the body only to a session is the one design that fails both being cited and being read.
Fixes in order of effort
| Order | Fix | Checks | Effort |
|---|---|---|---|
| 1 | Rewrite robots.txt per token, blocking the training crawlers only if that is the decision, and allowing OAI-SearchBot, PerplexityBot and Claude-User by name | A2 | Minutes |
| 2 | Declare the sitemap with real article dates first in robots.txt | A3, D4 | Minutes |
| 3 | Add datePublished and dateModified to the Article block and a time element with datetime to the article template |
D4 | Under an hour |
| 4 | Add Article on every article and a WebSite block on the homepage, with an Organization publisher carrying sameAs |
B2, D2, D5 | Template work |
| 5 | Confirm the edge returns 200 to the crawlers robots.txt allows | A5 | Minutes to test, one setting to fix |
| 6 | Replace the consent banner with cookieless analytics | A6 | An afternoon, or an argument |
| 7 | Serve the article body in the HTML for metered pages, with the gate drawn over it rather than in place of it | A7 | Architecture |
The first three are configuration. The fourth is the one that decides whether an answer can cite you by name and date. The last is the one that decides whether there is anything to cite.
Check your own site
The free scan lists which of the six tokens your robots.txt blocks, the schema types it found across the crawl, the withLastmod and modifiedLast90Days counts under D4, and any interstitial markers under A6. The content leaderboard shows other publishers on the same checks, and the definitions are on the methodology page under discovery and access, understanding and trust and freshness.