6 min read
Time to first byte for bots, 1.5 seconds is the bar
Check A5 loses a point when the median bot fetch takes 1.5 seconds or more. Where the time goes, how to measure it with curl, and the fixes in order of effort.
Checks A5
Blog category
Discovery & access
Pillar A: robots.txt, AI crawlers, edge blocking, sitemaps, llms.txt and server-rendered parity.
- 6 min read
Server-rendered parity, measuring the gap yourself
A Node script that computes check A7's raw-to-rendered text ratio, how to run it across your key routes, and what the number says about your rendering mode.
Checks A7
- 7 min read
CAPTCHA challenges and agents, what a challenge page costs you
A challenge on every request is a closed door to an agent. How check A6 detects challenge markup, and how to scope challenges to login, checkout and signup.
Checks A6
- 6 min read
Cookie walls are interstitials, the consent banner problem
Why check A6 treats a consent dialog as an interstitial, the markup it looks for, why a cookie-only wall scores one of two, and how going cookieless removes it.
Checks A6
- 6 min read
XML sitemaps for agents, the three points you are probably dropping
How check A3 scores a sitemap, exists, parses, referenced from robots.txt, why the first sitemap declared is load-bearing for D4, and how ours is built.
Checks A3, D4
- 6 min read
llms-full.txt, and when to publish the whole site as one file
What an llms-full.txt is for, how check A4 scores it, what belongs in it, how large is too large, and why ours comes from the same registry as the sitemap.
Checks A4
- 5 min read
How to write an llms.txt that earns its place
The llms.txt convention, why A4 gives three points for three described links and one for a bare list, and a complete example file for a small SaaS company.
Checks A4
- 6 min read
Google-Extended, what it controls and what it does not
Google-Extended is a robots.txt control token, not a crawler. What it switches off, what it leaves alone, three common misconfigurations, and what A2 records.
Checks A2
- 6 min read
Your robots.txt is perfect and you are still blocking agents
A vendor-neutral tour of the edge layer, bot rulesets, bot scores, rate limits, JavaScript challenges and slow origins, and how each shows up in the A5 probes.
Checks A5
- 5 min read
ClaudeBot and Claude-User are different things
Anthropic's crawler and its user-triggered fetcher share a prefix and get blocked together by accident. The robots.txt for each intent, and its A2 score.
Checks A2
- 5 min read
Blocking GPTBot but not OAI-SearchBot, the distinction that matters
OpenAI documents three tokens so training, indexing and user fetches are decided separately. The robots.txt that opts out of training only, and its A2 score.
Checks A2
- 6 min read
The six AI crawlers and what each one is for, a decision matrix
Which of the six A2 tokens a shop, a SaaS product, a publisher and a services firm should allow, what each combination scores, and what each block gives up.
Checks A2
- 6 min read
robots.txt for the agent era, a practical rewrite
A line-by-line rewrite of a typical legacy robots.txt. What checks A1, A2 and A3 read from it, the REP rules a parser applies, and a finished file to adapt.
Checks A1, A2, A3
- 8 min read
Cloudflare blocks AI crawlers by default, before it reads your robots.txt
Cloudflare's default bot rules turn GPTBot, ClaudeBot and PerplexityBot away before robots.txt is read. How to check in thirty seconds, and what to change.
Checks A2, A5