Agent readiness, one check at a time

How to write an llms.txt that earns its place

The llms.txt convention, why A4 gives three points for three described links and one for a bare list, and a complete example file for a small SaaS company.

Checks A4

Google-Extended, what it controls and what it does not

Google-Extended is a robots.txt control token, not a crawler. What it switches off, what it leaves alone, three common misconfigurations, and what A2 records.

Checks A2

Your robots.txt is perfect and you are still blocking agents

A vendor-neutral tour of the edge layer, bot rulesets, bot scores, rate limits, JavaScript challenges and slow origins, and how each shows up in the A5 probes.

Checks A5

ClaudeBot and Claude-User are different things

Anthropic's crawler and its user-triggered fetcher share a prefix and get blocked together by accident. The robots.txt for each intent, and its A2 score.

Checks A2

Blocking GPTBot but not OAI-SearchBot, the distinction that matters

OpenAI documents three tokens so training, indexing and user fetches are decided separately. The robots.txt that opts out of training only, and its A2 score.

Checks A2

The six AI crawlers and what each one is for, a decision matrix

Which of the six A2 tokens a shop, a SaaS product, a publisher and a services firm should allow, what each combination scores, and what each block gives up.

Checks A2

robots.txt for the agent era, a practical rewrite

A line-by-line rewrite of a typical legacy robots.txt. What checks A1, A2 and A3 read from it, the REP rules a parser applies, and a finished file to adapt.

Checks A1, A2, A3

Agent readiness is not SEO, and not GEO either

Where SEO, generative engine optimisation and agent readiness overlap, where they diverge, and why ranking and being usable for a task are different problems.

Cloudflare blocks AI crawlers by default, before it reads your robots.txt

Cloudflare's default bot rules turn GPTBot, ClaudeBot and PerplexityBot away before robots.txt is read. How to check in thirty seconds, and what to change.

Checks A2, A5

Who is actually fetching your site, a field guide to AI user agents

The six crawler tokens check A2 scores, what each one is for, training, search indexing or a fetch on a user's behalf, and what blocking each one costs.

Checks A2, A5

Reading your scan report, a check-by-check walkthrough

What each section of an AgentFriendlyRank report means, what the evidence blocks show, how fixes are ranked, and how to decide what to do first.

The two fetches, what an agent sees versus what your browser sees

Most serious agent readiness findings live between the raw HTML and the rendered DOM. How check A7 measures the gap, and how to measure it yourself with curl.

Checks A7

The three tiers of fixes, hours, weeks and architecture

What the rubric's Low, Medium and High effort ratings mean in practice, which of the 27 checks fall into each tier, and what the work costs.

What agent readiness actually means, in one page

The four questions the rubric asks of a site, find, understand, act and trust, and why passing every SEO test says nothing about any of them.

Why most sites score zero on machine endpoints, and why that is the opportunity

Check C4 asks for an agents.json, an MCP manifest, API docs or an OpenAPI spec. Almost nobody publishes one, which makes it the cheapest differentiation.

Checks C4

Why we score out of 100 and publish the whole rubric

The case for a public, versioned scoring standard over proprietary AI visibility scores nobody can check, and why we never rewrite a historical score.