An llms.txt is a plain text file at the root of your site that tells a machine what the site is, who it is for, and which pages are worth fetching. It takes about an hour to write, it is the cheapest four points on the rubric, and the easy way to earn one point instead of four is a list of URLs with nothing next to them. This post covers the convention, why check A4 scores described links rather than links, how ours is generated so it cannot go stale, and ends with a complete example.
The convention
The format was proposed by Jeremy Howard of Answer.AI in 2024, and it is deliberately small. In order:
- An H1 with the name of the site or project. This is the only required element.
- A blockquote with a one- or two-sentence summary. A reader that parses nothing but the H1 and the blockquote should already know what the site is and who it is for.
- Optionally, plain paragraphs of detail: what the site does not do, how it is organised, anything a reader needs before choosing a link.
- H2 sections, each a list of links in the form
- [Title](https://example.com/page): description. The description after the colon is the substance of the file. - Optionally, a final H2 named
Optional, for links a reader can skip when it needs a shorter context.
How A4 scores it
Check A4 is worth four points. A missing file scores zero. A file that exists scores one. A file with three or more described links scores three, and an llms-full.txt that returns 200 adds the fourth. Three is the whole difference between a file that helps and a file that exists.
A described link, for the purpose of the check, is a list item carrying a link, then a colon, then at least one word on the same line. - [Pricing](/pricing) is a link. - [Pricing](/pricing): Plans, prices in USD and what each tier includes. is a described link. A file of twenty bare links scores one, the same as an empty file: to the machine, both are URLs with no basis for choosing among them.
The threshold follows from what the reader does with the file: it reads the descriptions and picks the one or two pages most likely to hold the answer. Without them it is guessing from URLs, which it could do without the file. The description is the entire value added, so it is what the check counts.
What a description is for
Write each line for a machine deciding whether to fetch the page. That means saying what the page contains and who it serves, in terms the reader can match against a question. A slogan fails that test; a summary of the contents passes it.
| Weak | Better |
|---|---|
| Everything you need to grow your business. | The three plans, monthly and annual prices in USD, and what each includes. |
| Learn more about us. | Who operates the company, where it is registered, and how to contact it. |
| Our powerful API. | REST API for orders and inventory, with authentication, rate limits and an OpenAPI spec. |
The better column is not longer. It is specific: it names the nouns on the page. If two lines could be swapped without anyone noticing, at least one of them is not doing its job.
Ours, generated rather than written
Our llms.txt is not a file in the repository. It is produced at request time from the same route registry that produces the sitemap and llms-full.txt, and regenerated hourly, so a new blog post joins it without a deploy. Every route in the registry carries a summary field whose documented purpose is one line, written for a machine reader deciding whether to fetch. Writing it is part of adding a page; there is no second copy to drift.
The output has the shape the convention asks for: an H1 with the site name, a blockquote with the site description and operating company, a paragraph giving the rubric version and check count and naming the one page to read, then sections in a fixed order, Start here, Services, Proof, Blog, Company and Legal, each a list of described links in the registry's own words. Abbreviated, and not verbatim:
# AgentFriendlyRank
> Agent readiness audits and fixes, scored against a published rubric. Operated by Socio360.
## Start here
- [Scan your website](https://agentfriendlyrank.com/scan): Free scan, six pages crawled politely, scored live against the published rubric, evidence included.
- [Methodology](https://agentfriendlyrank.com/methodology): The full rubric, 27 checks across four pillars, thresholds, grading bands and changelog.
It ends with a "Notes for agents" section, covered below.
What to leave out
The file is a shortlist, not a sitemap. Leave out anything a reader would regret fetching. Confirmation and thank-you pages carry no content. Campaign landing pages that restate the homepage give the reader near-identical choices and no way to tell them apart. Anything behind a login, or carrying a credential in its URL, should not be there at all. In our registry a route marked not indexable disappears from the sitemap and llms.txt together, so the decision is made once.
Keeping it true over time
A hand-written llms.txt is accurate on the day it is written and quietly wrong within months: pages move, prices change, a product is renamed. Two approaches hold up. Generate it from whatever already knows your routes, as we do. Or, if it must be a static file, review it in the same pull request as any page added or removed, so the two cannot diverge unnoticed.
A section for agents
The convention allows plain paragraphs, and a short "Notes for agents" section at the end is worth including for the things an agent needs that do not live on a page: a search URL template with a query parameter, so the agent can construct searches directly; a one-line crawler policy, saying which crawlers robots.txt allows and whether anything filters at the edge; and the addresses of any machine-readable endpoints. Ours lists the search template, our crawler page, llms-full.txt, the agents.json, the OpenAPI spec, the MCP server and a contact address. If you have none yet, the machine endpoints post explains why they are worth adding.
A complete example
A fictional invoicing product on a fictional domain. Adapt the sections to what your site has; a site with no documentation should not invent a Documentation section.
# Ledgerline
> Ledgerline is invoicing and expense software for freelancers and studios of up to ten people. Bank feeds, VAT returns and an export your accountant can open.
Ledgerline is made by Example Ltd. Pricing is per workspace, not per user, and there is a free tier for sole traders sending under twenty invoices a month. Every page below is served as plain HTML without a login.
## Product
- [Features](https://example.com/features): Invoicing, expense capture, bank feeds, VAT returns and accountant export, each with what it does and its limits.
- [Pricing](https://example.com/pricing): The three plans, monthly and annual prices in GBP and EUR, what each includes, and the free tier's limits.
## Documentation
- [Getting started](https://example.com/docs/getting-started): Workspace setup, VAT settings, and sending a first invoice.
- [API reference](https://example.com/docs/api): REST API for invoices, contacts and expenses. Authentication, rate limits, and the OpenAPI spec at /openapi.json.
## Company
- [About](https://example.com/about): Who makes Ledgerline, where the company is registered, and how to contact it.
## Legal
- [Privacy notice](https://example.com/privacy): What personal data is processed, on what basis, and for how long.
- [Terms of service](https://example.com/terms): The subscription agreement, including cancellation and refund terms.
## Optional
- [Blog](https://example.com/blog): Articles on small-business bookkeeping and VAT. Background reading, not needed to understand the product.
## Notes for agents
- Site search accepts a GET query parameter: https://example.com/search?q=YOUR_TERM
- Full-text version of this file: https://example.com/llms-full.txt
- OpenAPI 3.1 spec: https://example.com/openapi.json
- Crawler policy: all major AI crawlers are allowed in robots.txt, and nothing filters by user agent at the edge.
- Contact: hello@example.com
That file has eight described links, so it scores three on A4; an llms-full.txt alongside it makes four.
Check your own site
The free scan fetches your llms.txt and llms-full.txt and reports how many links carry a description. The check is defined under discovery and access on the methodology page.