llms.txt explained: what it is and how to create one

llms.txt is one of the most talked-about and least understood files on the web right now. The pitch is simple: drop a plain-text Markdown file at your domain root, and AI models like ChatGPT, Claude, and Perplexity will read a clean, curated summary of your site instead of fighting through cluttered HTML. The reality is more interesting than the pitch, because the file is advisory. Nothing forces an AI crawler to fetch it, and the 2025-2026 evidence on whether they actually do is mixed at best.

This guide covers what llms.txt is, how to write one in a few minutes, and the part most articles skip: a straight, evidence-based answer to whether it works. The short version is that llms.txt is cheap to publish and occasionally useful, but it is not a ranking button, and the only way to know if it helps is to measure whether AI crawlers request it.

What llms.txt is

llms.txt is a proposed standard for a Markdown file, served at yourdomain.com/llms.txt, that gives large language models a curated map of your most important content. It was proposed by Jeremy Howard of Answer.AI on September 3, 2024, and published at llmstxt.org. The goal is to help models “use a website at inference time” by handing them clean, linkable context rather than making them parse navigation menus, cookie banners, and ad markup.

The proposal got its first revision in August 2026. In the v2 changelog, Howard writes that “Thousands of sites now publish an llms.txt file, documentation platforms generate one automatically, and coding agents use them reliably.” The revision is about how agents find and consume the file: they “view or search the llms.txt to find what they need, then follow the relevant links.” It is still a community proposal, not an IETF or W3C standard. The changes that matter for publishers are covered below.

The file solves a specific problem. LLMs work within a limited context window, and raw HTML burns tokens on markup that has nothing to do with your actual content. A well-structured llms.txt points a model straight to your documentation, product pages, or policies in a format it can ingest cheaply. Think of it as a reading list you hand to an AI assistant, curated by you rather than guessed at by a crawler.

It helps to place llms.txt next to the two files it is often compared with. Each does a different job:

File	Purpose	Who reads it
robots.txt	Grants or denies crawler access to paths	Search and AI crawlers
sitemap.xml	Lists every indexable URL for discovery	Search engines
llms.txt	Highlights and summarizes your best content	AI models (voluntarily)

The critical difference is enforcement. robots.txt is an instruction that well-behaved crawlers obey, and sitemap.xml is consumed by search engines that have committed to reading it. llms.txt is neither. It is a suggestion that a model can read, ignore, or never request in the first place. That single fact drives everything else in this article.

What goes in the file

An llms.txt file follows a strict but tiny Markdown structure, and only the first line is required. The spec at llmstxt.org defines the format in order:

• An H1 with the name of the project or site. This is the only mandatory element.
• A blockquote with a short summary of what the site is and who it serves.
• Zero or more Markdown sections with detail, guidance, or context.
• Zero or more H2 headings, each introducing a list of links in [name](url) format with optional notes.
• An optional section titled “Optional” for secondary links.

Version 2 of the proposal changed three things worth knowing:

• The Optional section is now just a convention. v1 described a tool, llms_txt2ctx, that expanded the file into one context and dropped the “Optional” links when space was tight. v2 removes that tooling, “and with it goes the special meaning of the Optional section.” You can still use it for secondary links, but nothing is guaranteed to treat it specially.
• Pages can point to their Markdown and to their llms.txt. v2 uses standard link relations: rel="alternate" type="text/markdown" for a page’s Markdown version and rel="describedby" for the llms.txt file that covers it, as an HTML <link> element or an HTTP Link: header. Both page.html.md and page.md URL forms are allowed.
• Subpath files have defined scope. A file such as /docs/llms.txt “covers the pages under its path, and the most specific file applies,” which lets a docs section or a project site publish its own.

There is also a companion convention. llms-full.txt compiles your core content into one large Markdown file so a model can load everything at once. Older guides also mention generated llms-ctx.txt variants, which came from the context-expansion tooling v2 dropped. For most sites, a lean llms.txt with good links is the right starting point, not a full content dump.

How to create an llms.txt file

Creating an llms.txt file takes minutes: write the Markdown, save it as llms.txt, and upload it to your web root so it resolves at https://yourdomain.com/llms.txt. There is no registration, no schema validation service, and no submission step. Here is a complete, copy-paste example you can adapt:

# Lume

> Lume shows site owners the non-human half of their web traffic and
> cryptographically verifies which AI agents and crawlers are authentic.

Lume is a server-side analytics platform for AI-agent, bot, and human
traffic. It identifies crawlers by user agent, network, and signature,
then verifies whether each claimed agent is genuine.

## Docs

- [Getting started](https://example.com/docs/quickstart): Install the
  SDK and start capturing agent traffic in minutes.
- [Agent verification](https://example.com/docs/verify): How Lume
  confirms an agent is who it claims to be.
- [API reference](https://example.com/docs/api): Endpoints, auth, and
  response schemas.

## Guides

- [Reading your AI crawler report](https://example.com/guides/report)
- [Blocking vs. allowing agents](https://example.com/guides/policy)

## Optional

- [Changelog](https://example.com/changelog)
- [Brand assets](https://example.com/brand)

A few practical rules keep the file useful. Link only to canonical, high-value pages, because a bloated file defeats the purpose. Keep summaries short and factual, since models read them literally. Point to Markdown or clean HTML versions of pages where you have them, as several AI coding tools prefer Markdown responses. And treat the file as living documentation: if your docs move, update llms.txt in the same change, or you will point models at stale or 404 URLs. That last mistake is common enough that it has caused AI coding assistants to hallucinate broken links from outdated files.

Two smaller details are worth getting right. First, make /llms.txt return either the file or a clean 404, never a server error: Chrome’s Lighthouse llms.txt audit “flags the pages if a server error occurs,” while a 404 is marked Not Applicable “as providing the file is optional at the moment.” Second, if you follow v2, add a discovery link to your page templates so agents that land on a page can find the file:

<link rel="describedby" href="/llms.txt">

Does llms.txt actually work?

This is the question that matters, and the honest answer as of late 2026 is: mostly not yet, with narrow exceptions. The file is trivial to create, which is exactly why so many sites have one and so few see any benefit. The evidence falls into three buckets.

Major AI companies have not committed to it. OpenAI, Anthropic, and Google have not formally adopted llms.txt as a signal their production crawlers parse. Google has been the most vocal. Search Advocate John Mueller compared llms.txt to the long-dead keywords meta tag, noting that “none of the AI services have said they’re using llms.txt, and you can tell from server logs that they don’t even check for it.” In June 2025 he was blunter on Bluesky, as reported by Search Engine Roundtable: “The consumer LLMs / chatbots (the ones that SEOs want traffic from) will fetch your pages - for training and grounding, but none of them fetch the llms.txt file.” Gary Illyes reinforced in 2025 that Google does not support the file and has no plans to.

Google has since put this in writing. Its guide to optimizing for generative AI features lists llms.txt under things you can ignore and says maintaining one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” The Chrome team, separately, ships an llms.txt check inside Lighthouse’s experimental agentic-browsing audits, whose documentation says that without the file “agents may spend more time crawling the site to understand its high-level structure.” Those two positions are not a contradiction. Asked about it by Lily Ray, Mueller said, as reported by Ahrefs, that llms.txt is “not done for search” but a “temporary crutch, perhaps to save some tokens” for AI coding tools reading developer docs. On Google’s Search Off the Record podcast he added the structural problem, as reported by Search Engine Journal: the file is self-reported, so an LLM “by design, can’t trust what is here as a way of differentiating between different websites.” He did leave one opening: “If someone is already on your website, maybe some kind of automated system is helpful.”

The server-log data is brutal. Ahrefs studied 137,000 domains and found that while 28% published a valid llms.txt file (an upper bound, since Ahrefs’ customers skew technical), 97% of those files received zero requests during May 2026. The details are more telling than the headline:

• No AI bot goes looking for it. Requests for llms.txt files that did not exist were 98% human, and the AI bot share of those 404s was zero.
• The readers are not the ones you want. Across the roughly 1,100 domains whose files were fetched at all (about 22,000 requests), named AI tools made 19.5% of fetches. GPTBot led and Anthropic’s coding agent, Claude-Code, came second, ahead of every AI search and assistant bot. OAI-SearchBot, PerplexityBot, and Claude’s search crawler combined made “only a couple of hundred fetches across thousands of sites,” and Slackbot fetched llms.txt more often than PerplexityBot did.
• Not even Lighthouse moves the needle. Chrome’s Lighthouse audit made just 22 of those requests, roughly 1 in 1,000.

When Mueller says the bots do not check for the file, the logs largely agree.

Citation data agrees. SE Ranking analyzed nearly 300,000 domains and found 10.13% had an llms.txt file, and that “Both statistical analysis and machine learning showed no effect of LLMs.txt on how often a domain is cited by LLMs.” Removing the llms.txt variable from its model actually improved the model’s accuracy. Adoption keeps climbing anyway: Originality.ai’s tracker, which scans more than 3 million sites, counted 4,088 llms.txt instances in June 2025 and 36,120 by May 2026, an 8.8x increase.

There is a real, narrower use case. The one group of readers that does show up in the logs is coding tools. In the Ahrefs data above, Anthropic’s coding agent, Claude-Code, was the second most frequent reader of llms.txt files, ahead of every AI search and assistant bot. That fits what the file was designed for: llms.txt earns its keep for developer-facing docs that AI coding tools pull into context on demand, and it does close to nothing for a marketing site hoping to rank in AI answers.

Put those together and the verdict is clear. llms.txt is not a magic ranking button, and no major search or answer engine treats it as one. It is a low-cost, low-risk file that helps a specific audience (AI tools reasoning over your documentation) and is widely ignored by the crawlers that feed AI Overviews and chatbots. If you want to influence how you show up in generative results, the durable levers are original content and clean, crawlable pages, which is the core of generative engine optimization and the practical steps to optimize your site for AI search. llms.txt is a small complement to that work, not a substitute for it.

How to know if it is working on your site

The only reliable way to judge llms.txt is to measure whether AI crawlers actually request it, because industry averages tell you nothing about your specific domain. This is where server-side observation beats speculation. If your logs show GPTBot, ClaudeBot, or PerplexityBot fetching /llms.txt, the file is doing something for you. If they only ever pull HTML, you have your answer, and you can stop worrying about it.

This is exactly the gap Lume was built to close. Lume shows you the non-human half of your traffic server-side and identifies which AI crawlers hit your site and what they request, so you can see whether a given bot ever touches /llms.txt versus crawling pages directly. It also verifies whether an agent claiming to be GPTBot or ClaudeBot is authentic rather than a spoofed user agent, which matters because a user-agent string is just a header anyone can send, and a fake GPTBot fetch would otherwise look like proof that OpenAI reads your file. Pairing a published llms.txt with real agent analytics turns the question from “should I have this file” into “is this file being read,” which is the only version of the question worth answering. The same visibility lets you connect crawler behavior to outcomes when you track AI referral traffic arriving from those tools.

Should you create one anyway?

For most sites, yes, publish a lean llms.txt, because the downside is nearly zero and the file occasionally helps. It costs a few minutes, it will not hurt your rankings, and if you run documentation that AI coding assistants consume, it can measurably improve the answers those tools give. Just calibrate your expectations. Publish a tight file that links to your genuinely best pages, keep it current, and do not expect it to move you up in ChatGPT or Google AI Overviews on its own.

Skip the maximalist approach. You do not need a giant llms-full.txt dump, a file crammed with every URL, or a paid “AI visibility” service built around it. Publish the minimal version, then watch your logs. The moment you can see which agents fetch it, you replace a year of speculation with evidence from your own traffic, and you can decide whether to invest further based on what the crawlers on your domain actually do.

Frequently asked questions

What is an llms.txt file?

An llms.txt file is a Markdown document served at your domain root that gives AI models a curated summary and link list of your most important content. It is a proposed standard, published in September 2024, designed to help LLMs read your site efficiently within a limited context window.

Where does llms.txt go on my site?

It goes at the root of your domain, so it resolves at https://yourdomain.com/llms.txt. This mirrors the placement of robots.txt and sitemap.xml. Upload it to your web root or configure your framework to serve it at that path.

Does Google use llms.txt for rankings?

No. Google representatives John Mueller and Gary Illyes have both stated Google does not use llms.txt, with Mueller comparing it to the discredited keywords meta tag. Google’s own guide to generative AI features in Search now says the file “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Chrome’s separate Lighthouse audit checks for the file for AI agents, not for Search.

Is llms.txt the same as robots.txt?

No. robots.txt controls whether crawlers may access paths and is obeyed by well-behaved bots. llms.txt is purely advisory and only summarizes your best content for models that choose to read it. Neither replaces the other, and llms.txt does not block or grant access to anything.

Do AI crawlers actually read llms.txt?

Rarely, based on current data. A 137,000-domain study found 97% of published files received zero requests in May 2026, no AI bot requested a missing file, and log analysis shows the major citation-driving crawlers usually skip the file and read HTML directly. The clearest exceptions are AI coding tools reasoning over developer documentation. The only way to know for your site is to check your own server logs or use agent analytics and AI-referral tracking to see which bots request it.

What should I put in llms.txt?

Start with an H1 site name, a one-line blockquote summary, and H2 sections linking to your best documentation, guides, or product pages in Markdown link format. Keep it short, link only to canonical pages, and use an “Optional” section for lower-priority links. Since the August 2026 v2 revision, that section is a convention for readers rather than an instruction tools are expected to act on.

What changed in llms.txt v2?

The August 2026 revision added standard link relations so agents can find a page’s Markdown version (rel="alternate" type="text/markdown") and the llms.txt that covers it (rel="describedby"), defined that a subpath file covers the pages under its path, accepted both page.html.md and page.md URLs, and dropped the llms_txt2ctx expansion tool along with the special meaning of the “Optional” section.