llms.txt: What It Is and Why Your Site Needs One

llms.txt is a plain markdown file at the root of your domain that hands AI systems a short, curated index of your most important pages, each with a one-line note on what it covers. Publish one. Just don't expect it to win you citations, because the 2026 server-log evidence says almost nothing fetches it, and Google has stated plainly that no AI system reads it. The honest reasons to add the file are narrower than most vendors suggest. This guide covers what the format is, what the data shows, how to write one properly in about twenty minutes, and what it will never do for you.

What llms.txt actually is

Jeremy Howard of Answer.AI proposed the convention on September 3, 2024, alongside a spec at llmstxt.org. The problem he described is real enough. A modern web page buries its answer under navigation, scripts, cookie banners, and ad slots, so a language model reading that markup spends most of its attention on furniture. Even a clean page rarely helps, because a whole site will not fit into one context window.

His fix borrows the shape of robots.txt and sitemap.xml. One file, one predictable location, written for machines rather than people. The spec asks for very little. An H1 carrying the site or project name is the only required element. Below it sits a blockquote that sums up what the site is and who it serves. After that you may add any markdown except headings, then as many H2 sections as you need, each holding a list of links. Every link can carry a colon and a short note explaining what sits behind it.

One convention matters more than the others. A section titled Optional marks links an agent may skip when it needs a shorter context. That gives the reader permission to stop early, which is the closest thing this format has to telling a model what you consider secondary.

The spec suggests a second habit too: publish clean markdown copies of your pages, either at page.md or page.html.md. The index points, and the markdown copies deliver.

That is the whole format. No schema, no validator, no registry, no governing body. It remains a proposal rather than a ratified standard, and two years on, not one major model provider has confirmed that its retrieval stack reads it.

What the evidence says about who reads it

Here the sales pitch and the server logs part company.

Ahrefs studied 137,000 domains. Roughly 28 percent of them, about 38,000 sites, served a valid llms.txt. Of those, 97 percent received zero requests for the file during May 2026. Not fewer requests than expected. None: no bots, no humans, nothing. Ryan Law, who runs content at Ahrefs, answered the question of whether you should bother with "In my opinion, no, not yet," and noted there is no evidence the file improves AI retrieval, lifts traffic, or makes models more accurate about you.

SE Ranking ran a different test on a wider sample. Across nearly 300,000 domains, in research published November 7, 2025, it found the file on 10.13 percent. Adoption barely shifted with site size: 9.88 percent among sites under 100 monthly visits, 10.54 percent in the 1,001 to 5,000 band, and 8.27 percent above 100,001. The modelling result is the one worth sitting with. When the team removed llms.txt as a variable from their XGBoost model of citation frequency, the model became more accurate. The signal was not weak. It was noise.

Google has been blunt. In June 2025 John Mueller wrote, "FWIW no AI system currently uses llms.txt," and likened the file to the old keywords meta tag. That comparison has teeth, since the keywords tag turned into a spam magnet and then into nothing. At Search Central Live in July 2025, Gary Illyes said Google does not support the convention and has no plans to.

Our own audit found the same gap between effort and effect. When we checked fifty B2B sites on August 27, 2026, 68 percent of them served an llms.txt, while a third of those same homepages carried no structured data at all. Those teams had done the speculative thing and skipped the proven one.

So why publish one at all

Three reasons survive the evidence. Not one of them is "to get cited."

Coding agents and documentation tools really do read it. This is the single audience that turns up in logs. Cursor, Claude Code, GitHub Copilot, and Windsurf look for the file when they pull documentation, because a curated map is the cheapest way to find the right page without spending context. Mintlify generates llms.txt automatically for every documentation site it hosts. If developers, technical buyers, or partners size you up through an assistant, that is a live channel worth feeding.

Writing the file forces the discipline that does move citations. Summarizing each page in one sentence is uncomfortable work. You find the four pages that say nearly the same thing, the service page that never quite states what the service is, and the one sentence a buyer actually needs. Fixing those problems changes how models read your HTML, and your HTML is where retrieval really happens.

The cost is roughly twenty minutes and the downside sits near zero. Conventions do sometimes win adoption later. Twenty minutes buys you that option cheaply. Paying a monthly fee for a managed llms.txt service, or counting the file as your AI search strategy, buys you nothing.

How to write an llms.txt in about twenty minutes

Pick the pages a model should quote

Start from buyer questions rather than your sitemap. Write down the eight to fifteen questions a prospect asks before they talk to you, then name the page that answers each one. That list is your file. Most sites need twenty to forty links, not four hundred.

Leave out anything that answers nothing: tag archives, pagination, thin category pages, the careers page, your privacy policy.

Write the H1 and the blockquote

The H1 is your company or product name. The blockquote is the hardest sentence in the file, and the one a model is most likely to repeat. Say what you do, who you do it for, and what makes the offer specific. Two sentences at most.

Vague: "We deliver innovative marketing solutions for growing brands." Specific: "Fractional CMO and AI SEO services for companies between two and fifty million in revenue that need senior marketing leadership without a full-time hire."

Group the links and write the descriptions

Use H2 sections that match how a buyer thinks: Services, Pricing, Case Studies, Articles. Then write one clear sentence per link. The description carries more weight than the URL, because it tells a model what the page settles.

Compare these two entries.

- [Fractional CMO](https://example.com/fractional-cmo): Our fractional CMO page

- [Fractional CMO](https://example.com/fractional-cmo): What a fractional CMO owns, typical monthly retainer ranges in 2026, and when to hire one instead of an agency

The first tells a model nothing it could not guess. The second gives it three facts and a reason to quote the page.

Add an Optional section

Put secondary material under a heading named Optional: press releases, older posts, company background. Agents that need a tighter context will skip it. You get to decide what gets dropped first instead of leaving that choice to a crawler.

Publish it at the root and keep it out of the index

The file belongs at yourdomain.com/llms.txt, served as text/plain, reachable without a redirect. Google has said it makes sense to noindex the file, which is sound advice: it is a machine artifact, and it has no business competing with your real pages in search results.

Put a date on it

Add a last-updated line, then diary a quarterly review. A file listing three pages you killed last spring is worse than no file, because the one audience that does read it will follow those links.

What llms.txt will not do

It does not control access. Access rules live in robots.txt, which governs what a crawler may fetch, while llms.txt blocks nothing at all. Anyone treating it as a consent or licensing mechanism has misread the spec.

It is not a ranking factor, in classic search or in AI answers. Both studies above tested exactly that claim and came back empty.

It will not rescue a page that answers nothing. A model quotes the passage that settles a question. Pointing at a vague page simply gets you to a vague page faster.

It should not mirror your sitemap. A dump of every URL rebuilds the problem the format set out to solve, and it throws away the only real value on offer, which is your judgment about what matters.

How to check whether yours gets read

Pull thirty days of access logs and count the hits on /llms.txt by user agent. You are looking for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and the coding assistants. Expect a thin result, and expect SEO audit tools to outnumber the AI bots. That is the normal picture in 2026, and knowing it beats assuming the file is working.

Then spend the rest of the hour on the things that do carry weight: schema markup, clear entity signals, and passages written to be quoted. Our breakdown of how models choose who to recommend covers the four factors that actually move an answer, and the GEO playbook puts them in sequence.

The bottom line

Publish an llms.txt. Spend twenty minutes on it, write the descriptions with real care, keep it current, and treat it as a cheap option on a convention that may yet get adopted rather than a lever you are pulling today. The exercise alone will sharpen how you describe your own pages.

Then be honest with yourself about what you just did. You spent twenty minutes. Nobody got cited because of it. The work that earns citations is slower and less tidy: fixing pages so they answer one question cleanly, building the third-party evidence models trust, and keeping your entity data consistent everywhere it appears. A file at your domain root is not a substitute for any of that, and any agency selling it as one is telling you more about their own priorities than about AI search.