Time to first byte for bots, 1.5 seconds is the bar

Check A5 awards two of its three points for status codes, when all three crawler user agents the scanner sends receive a 200, and the third for time. The scanner records how long each probe took, takes the median, and awards the point only if the median is under 1,500 milliseconds. Your robots.txt is perfect and you are still blocking agents names the slow origin as a soft block and gives the causes a paragraph. This post goes through them properly: what the scanner actually times, where the milliseconds go, how to see the breakdown with curl, and what to change, cheapest first.

What the number is

The crawler fetches your homepage once under each of three user agent strings, GPTBot, ClaudeBot and PerplexityBot, one after another, with a ten second timeout and redirects followed. For each it records a status and a field called ttfbMs. The evidence block on the report lists the three probes, the medianTtfbMs, and the edgeBlockingDespiteRobots flag.

The field's name is slightly generous. The timer starts when the request is issued and stops when the whole response body has been read, so what is recorded is the time to the last byte of the HTML, not the first. A homepage that starts responding quickly but delivers a large document slowly is measured on the whole document, and any redirect on the way, from http to https or from the bare host to www, sits inside the same timer.

The bar is strict rather than inclusive: a median of 1,499 ms earns the point and 1,500 does not. With three probes the median is the middle value, so one slow probe is forgiven and two are not. A probe that times out is recorded with a status of 0, which also costs the two status points.

One detail of sequencing helps diagnosis. By the time the bot probes run, the crawler has already fetched robots.txt, the sitemap, four text files and up to six pages under its own user agent, as documented at /bot, so the homepage is warm for anything that caches it. If the bot probes are still slow, whatever is warm for one user agent is cold for another.

Where the time goes

A request has a fixed sequence of phases, and each has a characteristic way of being slow.

DNS, connection, TLS and distance

Before a byte of HTTP is sent, the client resolves the name, opens a TCP connection and completes a TLS handshake. Each is at least one round trip, and the handshake is one or two depending on the protocol version. The crawlers that matter run from cloud regions chosen by their operators, not by where your customers are, so when the fetch comes from another continent every round trip pays the distance, and three or four of them before the request is even sent can consume a third of the budget. A long certificate chain, or a certificate whose revocation status the client must fetch separately because the server does not staple it, adds to the same phase.

The cache that bots miss

The most common cause of a slow bot fetch on an otherwise fast site is a cache that excludes bots. Either the origin sends a Vary: User-Agent header, so the cache keeps a copy per user agent string and a crawler seen once never hits; or the edge bypasses cache for requests it classifies as automated; or the HTML is not cached at all and only the assets are, so the browser feels fast on its later requests while every document fetch does the full origin round trip. MDN documents the Vary and Cache-Control headers that govern this.

Serverless cold starts

On a serverless platform a route that has not been called for a while has no running instance, and the first request pays to start one, loading the runtime, the framework and your code before rendering anything. That is exactly the request a crawler makes when it arrives at a site nobody has visited recently. The median forgives one cold probe; if the platform scales the instance down again between probes, it does not.

Origin work per request

A server-rendered homepage that queries a database, calls a CMS or asks a personalisation service on every request is as slow as the slowest of those calls, every time. Frameworks that fetch data at request time make this easy to do without noticing, because in development the calls are local and fast.

Body size

Because the timer runs to the last byte, the size of the HTML counts. A document that inlines a large serialised data payload for hydration takes proportionally longer over a distant connection. The two fetches explains why that payload exists; check B5 was adjusted in rubric version 1.0.1 so as not to penalise it, but A5's timer still has to download it.

Measure it with curl

curl can report the time at which each phase completed, measured from the start of the request. Run this under a browser user agent and then under each crawler's, three times each, and compare.

fmt='%{http_code}  dns %{time_namelookup}  tcp %{time_connect}  tls %{time_appconnect}  first-byte %{time_starttransfer}  total %{time_total}  redirects %{num_redirects}  bytes %{size_download}\n'

for ua in \
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/128.0 Safari/537.36" \
  "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
do
  for i in 1 2 3; do
    curl -sL -o /dev/null -A "$ua" -w "$fmt" https://example.com/
  done
done

The times are cumulative, so the gap between neighbours is the cost of that phase: tls minus tcp is the handshake, first-byte minus tls is roughly the origin's think time plus one round trip, and total minus first-byte is the body transfer. total is the figure to compare with the scanner's ttfbMs, because both run to the last byte, and a redirects value above zero means part of the total is spent on hops you can remove.

Then read the differences. A fast browser row and slow crawler rows in first-byte is the cache excluding bots. Every row slow in first-byte on the first run and fast on the second is a cold start. Every row consistently slow in first-byte is origin work. Large dns, tcp and tls values on every row are distance and handshake cost. A large gap between first-byte and total is body size. Run the loop from a cloud instance in a region other than your origin's; your desk is near your servers in every way that matters and will flatter every phase.

Fixes, in order of effort

1. Remove redirect hops. Make the canonical URL answer directly and have the alternatives redirect to it in one step. Configuration, usually minutes.
2. Stop varying the cache on user agent. Drop Vary: User-Agent unless you genuinely serve different HTML by device, and if you do, key the cache on the device class rather than the raw string.
3. Let bots use the cache. Find the rule that bypasses cache for automated traffic and exempt the crawlers you allow, in the same rule where you exempt them from blocking.
4. Cache the document, not just the assets. Give public routes a Cache-Control policy the edge can hold, with a short freshness window and stale-while-revalidate so a crawler gets the last copy while a fresh one is made.
5. Keep the origin warm, or make it static. Use a minimum instance count where the platform offers one. Better, render the homepage at build time or on a schedule, so there is nothing to wake.
6. Move per-request work out of the request. Fetch at build time, revalidate on an interval, or read from a cache the request never has to fill. The one item that touches application code, and the largest effect when origin work is the cause.
7. Tune the handshake. TLS 1.3, a short certificate chain, stapled revocation status and HTTP/2 each save a round trip or part of one. Last, because the others are worth more.

None of this needs a bot-blocking layer in front of the origin; the layer that speeds browsers up is frequently the one that turns crawlers away, and the Cloudflare post is the usual instance. Our own site runs with nothing in front of the origin, a choice recorded on our live report.

Check your own site

The free scan records the three probes with their status and ttfbMs, and the medianTtfbMs beside them, under A5, so you can see which probe was slow rather than infer it. A5 is defined in full under discovery and access, and our crawler's own timeouts and politeness limits are published at /bot.