A sitemap is worth three points on our rubric, and each point is a separate fact: the file exists, it parses to at least one URL, and robots.txt says where it is. Sites drop them one at a time, usually without noticing, because a browser never asks for the file. Then a second check, D4, reads the same file for lastmod values, and which sitemap it reads depends on a detail almost nobody thinks about: the order of the Sitemap: lines in robots.txt. This post goes through the three points, the ordering problem, and how our own sitemap is generated so that neither can go wrong quietly.
What the crawler fetches
Before any scoring, the crawler has to decide which file to read. It fetches robots.txt, and if that file declares one or more sitemaps it fetches the first one declared. If robots.txt declares nothing, it tries /sitemap.xml on the origin it was asked to scan. If the declared file fails, either because it does not return 200 or because it parses to zero URLs, the crawler falls back to /sitemap.xml on the origin as well. The fallback exists for a specific reason: robots.txt declares sitemaps as absolute URLs, which normally point at the production host, so a scan of a staging origin would otherwise follow the declaration off-site and report a missing sitemap for a site that has one.
The result is one file. The crawler does not follow a sitemap index into its children and does not merge several declared sitemaps. Whatever that one file contains is what A3 and D4 see.
Point one, it exists
The first point is awarded when the sitemap request returns a 200. The evidence block records this as exists.
The common way to lose it is to have a sitemap at a path the crawler never tries. A platform that publishes at /sitemap_index.xml or /wp-sitemap.xml and a robots.txt that does not mention it leaves the crawler fetching /sitemap.xml, getting a 404, and scoring zero on all three points, because the other two depend on this one. The fix is either of the two things the crawler understands: put a file at /sitemap.xml, or declare the real path in robots.txt.
The other way is a file that exists but does not return 200: a 403 from an edge rule, or a 503 from a generator that timed out. The sitemap is fetched with the same crawler identity as everything else, so an edge that refuses crawlers, as in the Cloudflare post, costs points in two checks rather than one.
Point two, it parses
The second point requires at least one URL to come out of the parser. The evidence block records the count as urlCount.
The parser is deliberately simple. It looks for <url> elements and takes the <loc> from each, with the <lastmod> if there is one. A file with a <urlset> root and one valid <url> entry scores the point. A file that returns 200 but contains an HTML error page, an empty <urlset>, or a namespace declaration with nothing inside it does not, and the report says so: sitemap parsed to zero URLs.
A sitemap index is the awkward case. The protocol allows a <sitemapindex> root whose entries are <sitemap> elements pointing at child files rather than pages. When the parser finds no <url> entries it falls back to reading those <sitemap> entries, so an index does score the parse point. What it does not carry is any per-page lastmod, because an index lists files, not pages. That is where D4 comes in, below.
The two shapes:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://www.example.com/</loc><lastmod>2026-09-04</lastmod></url>
<url><loc>https://www.example.com/pricing</loc><lastmod>2026-08-21</lastmod></url>
</urlset>
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap><loc>https://www.example.com/sitemap-pages.xml</loc></sitemap>
<sitemap><loc>https://www.example.com/sitemap-products.xml</loc></sitemap>
</sitemapindex>
Both parse. Only the first gives the freshness check anything to read.
Point three, it is referenced
The third point is awarded when robots.txt returns 200 and contains at least one Sitemap: directive. The evidence field is referencedInRobots.
The directive takes an absolute URL, sits outside any user-agent group, and can appear anywhere in the file. A robots.txt that does not exist loses this point regardless of how good the sitemap is, which is one of the reasons the robots.txt rewrite argues that a 404 is sloppier than an empty file. The recommendation attached to a missing reference puts the reason plainly: a sitemap referenced from robots.txt is how a crawler discovers pages that are not linked from the homepage.
That is the whole of A3: three facts, each checkable with curl, which is why the rubric rates the check's effort as Low.
Why the first sitemap listed is load-bearing
D4, freshness signals, is worth four points and reads the same file. One point for any lastmod value at all. Two points when at least 20% of the URLs in the file have a lastmod within the last 90 days of the scan. One point for visible dates in the crawled pages, meaning a <time> element with a datetime attribute or a datePublished or dateModified field in the page's JSON-LD. The evidence block records sitemapUrls, withLastmod, modifiedLast90Days and visibleDates.
Two details in that definition decide most outcomes.
The first is that the crawler reads the first sitemap declared in robots.txt. If that is an index, the URLs it parses are child sitemap locations with no lastmod, so withLastmod is zero and three of the four points are gone, however carefully the child files are dated. The remedy is ordering: list the file that contains real page entries with real dates first, and the index or the remaining chunks after it.
The second is the denominator. The 20% is measured against every URL in the file, not against the ones that carry a date. A sitemap with 8,000 product pages last touched two years ago and 40 living pages fails the window even if all 40 changed this week. If the first-declared file is the one with the pages that actually change, the ratio reflects the site as it is.
There is a tempting shortcut, which is to stamp every URL with the build date. The scanner cannot tell a true date from a false one, so it would score. It would also make the value meaningless to any consumer comparing lastmod against what actually changed. A sitemap claiming every page changed today is a freshness signal that says nothing, and we treat it the way we treat an invented score.
How ours is generated
Our sitemap at /sitemap.xml is a route handler rather than a static file. It builds a <urlset> from the same route registry that feeds llms.txt and llms-full.txt, so a page cannot be added to the site and left out of the sitemap. Static pages carry the registry's content date, edited by hand when content changes, never the deploy date. Blog posts carry their own publish date, or a revision date if one has been set. The file is regenerated hourly, so a scheduled post joins it on its day without a deploy. The llms-full.txt post describes the same registry from the other side.
Report pages are a different problem. Every scanned site gets a page at /site/{domain}, and the number grows with every scan. They live in sibling files at /sitemaps/sites-{n}.xml, each capped at the protocol's limit of 50,000 URLs, each carrying the scan date as lastmod, and each listed in robots.txt after /sitemap.xml. The registry sitemap stays first, on purpose and with a comment in the code saying so, because our own crawler reads the first declared sitemap and D4 needs real page dates there.
One more decision is worth copying. If the database that lists the report pages is unreachable, robots.txt declares the registry sitemap alone, and a chunk file returns a 503 with a Retry-After header rather than an empty 200. An empty sitemap is not neutral; a consumer reads it as a statement that the pages it used to list are gone.
Check your own site
Fetch your robots.txt, take the first Sitemap: URL, fetch that, and look for <url> entries with <lastmod> values. That is the whole of what the crawler does. The free scan does it for you and records exists, urlCount and referencedInRobots under A3, with the freshness counts under D4. The definitions are on the methodology page under discovery and access and trust and freshness.