What llms.txt is and whether it's worth shipping

The pitch is reasonable. A modern web page is mostly navigation, cookie banners, scripts and layout. The actual information is a small fraction of the bytes. An agent that wants to answer a question about your product has to fetch all of it and throw most of it away. llms.txt is a curated index — here is what this site is, here are the pages that matter, in markdown, already clean.

That's a good idea. Whether it's a good idea that anything currently acts on is a separate question, and the answer today is mostly no.

What the format actually specifies

The specification is short. It defines a markdown file served at /llms.txt with a fixed order:

Each entry in a file list is a markdown hyperlink — [name](url) — optionally followed by a colon and a note describing the file. The Optional H2 is the one piece of reserved vocabulary: it signals secondary material an agent can drop when it's short on context window.

You'll also see llms-full.txt in the wild. That is not part of the original specification. It's a convention that grew up around it — a single file containing the full text of the linked pages rather than links to them — and different platforms implement it differently.

An example llms.txt

Here is a complete, spec-shaped file for a small services business:

# Rivet & Co.

> Rivet & Co. is a commercial HVAC contractor serving the Portland metro area.
> We handle rooftop unit replacement, preventive maintenance contracts, and
> emergency repair for buildings between 5,000 and 200,000 square feet.

We publish detailed guides on equipment selection, maintenance scheduling and
building code compliance. Pricing is quoted per site; we do not publish rate cards.

## Services

- [Rooftop unit replacement](https://example.com/services/rtu-replacement): scope, timeline and permitting for full RTU swaps
- [Preventive maintenance contracts](https://example.com/services/maintenance): what quarterly and semiannual plans cover
- [Emergency repair](https://example.com/services/emergency): 24-hour response coverage and dispatch areas

## Guides

- [How often commercial HVAC needs servicing](https://example.com/guides/service-intervals)
- [Signs a rooftop unit needs replacing rather than repairing](https://example.com/guides/repair-vs-replace)
- [Oregon commercial HVAC code requirements](https://example.com/guides/oregon-code)

## Company

- [About and service area](https://example.com/about)
- [Contact and dispatch](https://example.com/contact)

## Optional

- [Case study archive](https://example.com/work)
- [Equipment brands we service](https://example.com/brands)

That's the whole exercise. Write it by hand, keep it under a couple hundred lines, and point at pages that genuinely answer questions rather than at your entire sitemap. A file that lists 400 URLs is a sitemap with worse formatting and defeats the point.

Does anything actually read it?

This is where honest coverage of llms.txt tends to stop short, so let's be specific.

Google: no. John Mueller, speaking for Google Search, described llms.txt as "purely speculative for now" and pointed out that the file has existed for years without any AI system using it. His suggested threshold was blunt — when an AI platform that actually sends you customers tells you it needs the file, build it then. Google's own documented lever for AI content use is Google-Extended, a robots.txt token that controls whether your content trains future Gemini models. Google states directly that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." Neither mechanism has anything to do with llms.txt.

The other major engines: no public commitment. OpenAI, Anthropic and Perplexity all publish documentation for their crawlers and the robots.txt tokens that govern them. None of them documents llms.txt as an input to retrieval or citation.

Documentation platforms: yes, meaningfully. This is the part that works. Mintlify automatically generates llms.txt and llms-full.txt for sites it hosts, and a large number of developer documentation sites — including several published by AI companies themselves — now serve these files. Coding assistants and agent frameworks that ingest documentation can and do fetch them. If your product has technical docs that developers point AI tools at, this is a real, present-tense use case rather than a speculative one.

So the adoption picture is lopsided: near-total in developer documentation, near-zero in consumer AI search.

So should you ship one?

Yes, with correctly sized expectations.

The argument for is that it costs an afternoon, it is a static file that cannot break anything, it forces you to write down what your site is actually for, and if adoption ever arrives you are already there. The argument against is that some people will ship it, tell their boss they've "done AI SEO," and stop — which is worse than never having heard of it.

The work that demonstrably influences whether an AI engine names you is elsewhere: being crawlable and fast, answering the question in the first sixty words of the page, marking up your entities in structured data, attributing your statistics, and having genuine depth on the topic. That's the substance of answer engine optimization. llms.txt is a nice-to-have you do after those, not instead of them.

Anyone selling llms.txt implementation as a standalone service is selling you a text file.

How to ship it without wasting the effort

Write it by hand rather than generating it from your sitemap. Keep the blockquote summary tight and factual — this is the part most likely to be read verbatim by anything that does fetch the file. Link only to pages you would be happy to have quoted. Put anything promotional under ## Optional. Serve it as text/plain or text/markdown at the root, make sure it returns a 200, and don't block it in robots.txt.

Then add it to whatever list you use for content that goes stale. A llms.txt pointing at three URLs that 404 is worse than no file at all, because it's the one artifact on your site explicitly addressed to machines that don't check.

Frequently asked questions

Does Google use llms.txt?

No. Google Search representative John Mueller has described llms.txt as "purely speculative for now" and noted that the file has existed for years without any AI system adopting it. Google's documented controls for AI-related content use are robots.txt tokens — Google-Extended governs whether content is used to train future Gemini models, and Google states that this token does not affect inclusion or ranking in Google Search. Neither AI Overviews nor AI Mode is documented as reading llms.txt. If a page appears in Google's AI features, it is because Googlebot crawled and indexed the page normally.

Is llms.txt an official standard?

No. It is a proposal published by Jeremy Howard of Answer.AI in September 2024, hosted at llmstxt.org. It has no standards body behind it, no formal review process, and no adopting engines that have committed to honoring it. That is different from robots.txt, which has decades of universal implementation and is now a published IETF specification. Calling llms.txt a standard is a marketing choice rather than an accurate description. It is a well-designed convention with meaningful adoption in one niche — developer documentation — and negligible adoption everywhere else.

What is the difference between llms.txt and robots.txt?

They solve opposite problems. robots.txt is a permission file: it tells crawlers which paths they may and may not fetch, and every major crawler respects it. llms.txt is a discovery file: it tells an AI agent what the site is about and where the substantive content lives, in clean markdown. robots.txt restricts; llms.txt recommends. One is universally honored and directly consequential — blocking the wrong bot in robots.txt can remove you from an engine entirely. The other is optional, advisory, and currently ignored by the large consumer AI search products.

What is llms-full.txt?

llms-full.txt is a convention that grew up alongside llms.txt rather than part of the original specification. Where llms.txt links to your important pages, llms-full.txt concatenates the actual text of those pages into a single file, so an agent can ingest the whole corpus in one fetch instead of many. Documentation platforms that auto-generate these files typically produce both. It is most useful for technical documentation, where an assistant genuinely benefits from having a full API reference in context. For a marketing site of a few dozen pages, it adds little over the linked version.

Will llms.txt help me get cited by ChatGPT or Perplexity?

There is no evidence that it will, and neither company documents it as an input. Both operate their own crawlers with published user agents and their own robots.txt tokens, and both retrieve from indexes built by crawling pages normally. What demonstrably affects citation is whether your page can be fetched, whether it answers the question in a self-contained passage near the top, whether its structure and markup are machine-readable, and whether the claims on it are attributed. Ship llms.txt because it is cheap and tidy. Do not ship it expecting citations to move.