llms.txt file: what it is and how to create one

The llms.txt file is a proposed standard, published by Answer.AI co-founder Jeremy Howard in September 2024, that offers large language models a clean Markdown map of the pages you most want them to read. The idea is simple: raw HTML is noisy, context windows are small, so you hand the model a curated menu instead of making it dig through your whole site. This guide breaks down what the file actually is, gives you a template you can copy today, and does the thing most tutorials skip: it tells you honestly, with real data, where llms.txt helps and where it does nothing at all.

What is an llms.txt file?

An llms.txt file is a single Markdown document, hosted at the root of your domain, that lists your most important URLs with short descriptions so an AI model can find and understand your best content quickly.

Think of it as a curated table of contents written for machines. A typical file opens with an H1 (your site or product name), a one-line summary in a blockquote, then H2 sections that group links by theme, each link followed by a plain-language note on what the page covers. The published specification lives at llmstxt.org and also defines an optional llms-full.txt variant that inlines the full Markdown of each page rather than just linking to it.

Crucially, llms.txt is descriptive, not restrictive. It does not block anything, grant access, or change how your pages render. It only says: here is what matters, and here is what each piece is about. That makes it closer to a hand-picked sitemap than to any access-control file.

Who created llms.txt and why?

Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed llms.txt on September 3, 2024. The problem he set out to solve was practical: LLM context windows are too small to swallow an entire website, and converting a page full of navigation, ads, and JavaScript into clean text is slow and error-prone.

The original use case was developer documentation. When a coding assistant needs your API reference, pointing it at a tidy Markdown file is far more reliable than asking it to parse a rendered docs site. That heritage matters, because it explains why llms.txt caught on fastest in developer tooling and only later got repositioned as a general generative engine optimization tactic. The name deliberately echoes robots.txt, but as we will see, the resemblance is mostly cosmetic.

How is llms.txt different from robots.txt and sitemap.xml?

Short answer: robots.txt controls access, sitemap.xml lists everything for discovery, and llms.txt curates a small set of your best pages for comprehension. They solve three different problems and can all coexist.

The key distinction is enforcement. If you want to keep an AI bot out, that is a robots.txt job, covered in our guide to managing GPTBot and other AI crawlers. If you want the full list of pages crawled and indexed, that is your XML sitemap. llms.txt sits on top as an optional roadmap, and unlike the other two it carries no guarantee that anyone reads it. For a refresher on the access layer, see our robots.txt configuration guide.

Do the AI engines actually read llms.txt?

Mostly no, and this is the part that most tutorials gloss over. As of 2026, no major AI company (OpenAI, Google, Anthropic, Meta, or Mistral) has publicly committed to reading or acting on llms.txt in their production search and answer systems.

The public statements are consistent. OpenAI's own crawler documentation tells site owners to control GPTBot with robots.txt and never mentions llms.txt. Google's John Mueller compared llms.txt to the long-ignored keywords meta tag, framing it as something a site owner claims about itself rather than something Google verifies. The usage data backs this up: an Ahrefs study of 137,000 domains found that 28% now publish an llms.txt file, yet 97% of those files never received a single request as of May 2026.

Where llms.txt genuinely gets fetched is developer tooling: IDE agents like Cursor and Claude Code, MCP servers, and in-product documentation assistants routinely pull it in to build context. So the file is not useless, but its real traction is in agentic developer workflows, not in the AI answer engines most marketers care about.

How do you create an llms.txt file?

You write a plain Markdown file, name it llms.txt, and save it. The structure is deliberately minimal, so a working file takes about ten minutes to assemble by hand.

Follow this order:

• H1: the name of your site or product, one line.
• Blockquote summary: a single sentence describing what you do.
• H2 sections: group your links by theme (Docs, Guides, Products, Company).
• Links with descriptions: each entry is a Markdown link followed by a short note on what the page answers. Reuse existing meta descriptions if you have good ones.
• Optional section: put lower-priority links under a final H2 titled Optional so a model knows they can be skipped.

A minimal template looks like this:

• # Example Co
• > Example Co builds accounting software for freelancers.
• ## Docs
• - [Getting started](https://example.com/docs/start.md): first-run setup and account creation
• - [API reference](https://example.com/docs/api.md): endpoints, auth, and rate limits
• ## Guides
• - [Invoicing basics](https://example.com/guides/invoices.md): how to create and send an invoice
• ## Optional
• - [Changelog](https://example.com/changelog.md): release history

Lead with your canonical, most-cited pages. Keep descriptions concrete and answer-shaped, because a model scanning the file is deciding what to fetch next. If your stack can output clean Markdown versions of each page (the .md links above), all the better, since that is the format the standard was designed around.

Where do you place llms.txt and how do you validate it?

Place the file at the root of your domain so it resolves at yourdomain.com/llms.txt, exactly like robots.txt. Serve it as plain text or Markdown with a 200 status code, and make sure your robots.txt does not accidentally block the path.

To validate, load the URL in a browser and confirm it renders as raw text, not an HTML error page. Then check three things: every link returns a 200, the Markdown is well-formed (each entry is a proper list item under a heading), and the file stays reasonably short. The whole point is a low-noise summary, so a bloated file that lists hundreds of URLs defeats the purpose and starts to duplicate your sitemap. Keep it to the pages you would genuinely want quoted. If you maintain it, review it whenever your site structure changes so the links do not rot.

Does llms.txt actually help your AI visibility?

The honest answer, based on the available evidence, is that there is no measured citation benefit yet. This is the information most llms.txt guides leave out, so it is worth stating plainly.

A SE Ranking analysis of 300,000 domains found a 10.13% adoption rate and, more tellingly, no measurable lift: having an llms.txt file did not make a domain more likely to be cited by AI models. Their machine-learning model actually predicted citations more accurately once llms.txt was removed as a variable, meaning the file behaved as noise rather than signal in their data. That is not proof it can never help, but it is a strong reason not to expect it to move your standing in the AI engines today.

What does drive AI citations is well-documented and unrelated to this file: pages that already rank, clean extractable passages, sourced statistics, and consistent entity signals. If you want the tactics that genuinely earn quotes, our guide on how to get cited by AI covers them. Treat llms.txt as complementary housekeeping, not as the mechanism that gets you into an answer.

Should you create an llms.txt file in 2026?

For most sites, yes, because the cost is near zero and it future-proofs you if adoption grows, but do it with clear expectations. Use this decision framework:

• Create one now if you publish developer documentation or an API, since agentic dev tools already fetch it and your users benefit directly.
• Create a lean one if you have a content or product site and can maintain it in minutes, treating it as a cheap hedge on the standard being adopted later.
• Skip it or deprioritize if you would have to build heavy tooling to generate and sync it, or if you would be tempted to treat it as a substitute for the fundamentals.

The trap to avoid is opportunity cost. An llms.txt file is a five-minute task that some vendors dress up as a growth strategy. Ship it if it is cheap, then put your real effort into technical SEO and citable content. Disclosure: our team also builds Sorank, so we watch this space closely, and our current read is that llms.txt is worth doing and not worth overthinking.

Frequently asked questions

Related guides