llms.txt: what it is, what reads it, and how to generate one

llms.txt is a Markdown file published at the root of a domain — /llms.txt — that gives large language models a short, curated index of a site: an H1 with the site's name, a blockquote summary, optional prose, and H2-grouped lists of links that each carry a one-line description. It was proposed on 3 September 2024 by Jeremy Howard of Answer.AI, and revised to v2 in August 2026. It is not a crawl directive, it is not markup, and — this is the part most articles bury — it is not a ranking factor: Google has said outright that Search ignores it. What it is: a reading aid, fetched on demand by an agent that needs to understand your site without crawling forty pages of navigation chrome first.

That makes the honest recommendation narrower than the hype and broader than the backlash. Publishing one costs an afternoon if you generate it and nothing thereafter; expecting it to move rankings is a category error; the payoff, where there is one, is that a coding agent or a documentation tool pointed at your site gets a clean map instead of a scrape. This page covers the spec, the evidence, and a generation method that survives contact with a growing site.

What the spec actually says

The llmstxt.org spec is short. A conforming file contains these sections, in this order: an optional byte-order mark; an H1 with the name of the project or site — the only required section; a blockquote with a short summary carrying the key information needed to understand the rest of the file; zero or more Markdown sections of any type except headings, giving more detail about the project and how to interpret the files listed; and zero or more sections delimited by H2 headings, each containing a "file list" of URLs. Each entry in a file list is a required Markdown hyperlink [name](url), optionally followed by a : and notes about the file.

The file may sit at the site root or at any path within it, covering the pages beneath that path — where several apply, the most specific one wins. The spec draws its own line against robots.txt in one sentence: robots.txt tells automated tools what access is acceptable, while llms.txt information is used on demand, when an agent needs it. Everything else about the format follows from that distinction.

# Example Project

> Example Project is a … — one clean definitional sentence, then the
> two or three facts a reader needs before the links make sense.

Optional prose. No headings allowed in this block.

## Docs

- [Quickstart](https://example.com/quickstart): install and first run, in five minutes
- [API reference](https://example.com/api): every endpoint, with request and response shapes

## Optional

- [Changelog](https://example.com/changelog): release notes, by version

What changed in v2 (August 2026)

The v2 revision is worth reading in full, because it quietly repositions the whole proposal. Four changes matter:

• Discoverability via link relations. The commonest request after two years of adoption was: given a page, how does an agent find its Markdown version, or the llms.txt covering it, without guessing? v2 answers with standard relations — an alternate link with type text/markdown for the page's Markdown version, and a describedby link for the llms.txt that covers it — as HTML <link> elements or an HTTP Link: header. If you publish Markdown versions of pages, this is the single most useful thing you can add this year.
• Both Markdown URL forms are legal. v1 specified .md appended to the full page URL (page.html.md); publishing tools shipped extension replacement (page.md) anyway, and v2 blesses both.
• Subpath semantics are defined. A file covers the pages under its path, and the most specific file applies — which is what lets a site that only controls a path, such as a GitHub Pages project site, participate at all.
• The Optional section lost its mechanical meaning. v1 came with context-expansion tooling that used ## Optional to decide what to omit. v2 drops that tooling and states the expectation directly instead: agents view or search the file, then follow the relevant links, which should point at LLM-friendly content. Optional sections are still allowed as a convention for secondary links — they just no longer instruct anything.

Read the direction of travel: the author's own framing moved from "language models might read websites" to "coding agents use these reliably." The spec is settling into being an agent-facing convention, not a search one — which is exactly what the measurements say too.

The honest part: almost nothing fetches it

Google Search does not use llms.txt, and says so in its own documentation. Its guidance for generative AI features in Search (updated 10 July 2026) states that you do not need to create new machine-readable files, AI text files, markup, or Markdown to appear in Search including its AI features, because Search itself does not use them — and that maintaining an llms.txt for other systems will neither help nor hurt your visibility, since Search ignores it. This is consistent with what Google staff have said in public since 2025: Gary Illyes confirmed at Search Central Live APAC in July 2025 that Google had no plans to support it, and John Mueller has compared the file to the keywords meta tag, a site-controlled field search engines stopped trusting a decade ago.

The request logs agree. Ahrefs checked 137,210 domains that had traffic in May 2026. Roughly 38,000 of them served a valid llms.txt — about 28% adoption, which the authors themselves call an upper bound because their audience skews technical and SEO-aware. Of those files, 97% received zero requests that month. Among the requests that did arrive, AI bots accounted for 19.5%, while 77% came from non-AI sources: SEO audit tools, unidentified crawlers, tech profilers. The authors add the caveat that matters most — a request is the generous measure, and whether a bot acts on what it fetched is another question entirely.

As of August 2026 we have not found documentation from any major AI provider stating that its crawlers fetch your llms.txt, and Google has explicitly said Search does not. Note the irony that the labs publish one for their own docs: docs.anthropic.com, developers.openai.com, docs.perplexity.ai, plus Cursor, Cloudflare, Stripe, GitHub and Svelte all serve one. That is not hypocrisy — it is the tell. These files exist for developers and coding agents pulling clean documentation into a session, not for crawlers ranking a page.

One more contradiction worth knowing before someone quotes half of it at you: Chrome audits the file that Google Search ignores. Lighthouse now ships an experimental Agentic Browsing category — Chrome 150 and later — that checks for a machine-readable summary at the domain root alongside WebMCP and agent-accessibility audits, and reports a pass ratio rather than a 0–100 score. Both statements are true at once, because they are about different jobs: Search ranking versus agent readiness.

So who is it actually for?

• Coding agents reading your docs. The demonstrated use case, and the one the v2 revision leans into: a developer points Claude Code, Cursor or Codex at your docs, and the agent gets a curated map plus clean Markdown instead of scraping rendered HTML.
• Your own agents and skills. An internal agent that needs to know what your site contains is a first-class consumer of your own index, and you control both ends.
• The discipline it forces. Writing one makes you produce a single canonical sentence that defines what you do, plus a one-line description per page. Both are useful well beyond the file — they are what an answer engine lifts when it summarizes you anywhere.
• Not: rankings, AI Overviews placement, or citation frequency in ChatGPT. No published evidence supports any of these, and Google denies the first outright.

How to generate one from your sitemap

The failure mode for llms.txt is not writing it — it is the file quietly rotting three months later while the site moves on. A hand-maintained index is worse than none, because it confidently describes a site that no longer exists. So derive it, from the inventory you already have:

• 1 — Start from the inventory that builds your sitemap. Read the same list of pages your sitemap generator reads. One source of truth means llms.txt and sitemap.xml cannot disagree about what the site contains.
• 2 — Pull each link's title and note from the page itself. Take the link text from the page's <title> and the one-line note from its <meta name="description">. Both are already written to stand alone, and reading them live means a retitled page updates the index for free.
• 3 — Hand-write only the header block. The H1 and the blockquote are the one part worth writing by hand. Open the blockquote with a clean definitional sentence — Product X is a … — because that is the sentence an agent or answer engine can lift verbatim.
• 4 — Group the links by job, not by URL structure. Editorial H2 sections — start here, set up, compare, use cases, writing — tell a reader which links matter for which task. A directory-shaped dump does not.
• 5 — Make missing coverage loud. Any page in the inventory that no section claims should be appended to a catch-all section and reported as a warning at build time. A new page must never silently fail to appear.
• 6 — Keep it to one canonical language and link the sitemap for the rest. llms.txt is a curated context file, not a URL dump. Index the canonical language set and point at sitemap.xml for the full multilingual inventory.
• 7 — Regenerate on every content change, and never hand-edit the file. Wire the generator into the same command that rebuilds the sitemap, and treat the .txt as build output. A hand-edited index drifts within weeks, which is the failure mode that makes llms.txt worthless.

A worked example: clize.ai/llms.txt is produced exactly this way. A generator reads the same page manifest that produces the sitemap, pulls every link's title and one-line summary live from each page's <title> and <meta name="description">, groups the English canonical set under hand-chosen H2 headings, and warns if a page in the manifest landed in no section — in which case it is appended to a ## More block rather than dropped. The only hand-written part is the header, and its first sentence is deliberately definitional: Clize is a CLI and MCP server that gives AI coding agents real-world actions: domains, email, deploys, payments, and media generation. The same sentence appears on the site and in the docs, so anything summarizing us has one consistent claim to lift.

llms.txt vs robots.txt vs sitemap.xml

Two of these are addressed to machines that index you. The third is addressed to a machine that is reading you right now. That is why llms.txt is curated and short while a sitemap is exhaustive and long — and why "put every URL in llms.txt" is the most common way to make the file useless.

Common mistakes

• Hand-maintaining it. The index and the site diverge, and the file starts lying. Generate it.
• No H1, or no summary. The H1 is the only section the spec requires, and the blockquote is what makes the links interpretable. Lighthouse's audit flags a file that is missing a heading, too short, or link-free — a decent proxy for "an agent will get nothing from this."
• Dumping the sitemap into it. Every URL, no descriptions, no grouping. That is a sitemap with worse syntax.
• Marketing copy in the notes. The one-line note should say what the page contains, not why the reader should be excited. Adjectives cost context and buy nothing.
• Serving HTML at /llms.txt. If your 404 handler returns a styled page with a 200 status, agents get a wall of markup where Markdown should be. Check the status code and the content type.
• Expecting rankings. Publish it because agents read your docs, not because you were told it helps SEO.

Should you publish one?

If you ship developer documentation, yes — this is the case with actual evidence behind it, and your readers' coding agents will use it. If you run a marketing site, publish one only if you can generate it; the upkeep, not the file, is the cost. Either way, measure the right thing: grep your server logs for requests to /llms.txt and see who actually fetches it, rather than watching rankings for a change that Google has told you will not come. If you want the same honesty applied to the tools that claim to measure AI visibility, we compared them in GEO tools, grouped by what they actually do.