llms.txt: when it works and when it's just decoration

· , Founder, modulla · EN

llms.txt is a Markdown-formatted text file in a domain's root directory that gives language models a condensed site map. The premise is reasonable: instead of parsing 2,600 kilobytes of HTML filled with navigation and scripts, an agent fetches a few kilobytes of clean content.

Around this file, however, a promise has grown that the data does not support. It is sold as a GEO standard, a way to make AI search engines cite a brand more often. In that role, llms.txt does not work, and both Google and server logs from studies covering hundreds of thousands of domains say so.

There is, however, a second use case where the file actually delivers, and the same logs confirm it. This article separates those two cases, because confusing them costs implementation time with no return.

What the llms.txt file is: definition and standard

The standard was proposed in September 2024 by Jeremy Howard, creator of the fast.ai library. The idea is simple: provide machines with the equivalent of a table of contents that points to what is valuable on the site, instead of forcing them to discover the site's architecture on their own through repeated server requests.

The standard defines two related files:

Both files must be located in the domain's root directory (e.g. yourdomain.com/llms.txt) and served with MIME type text/plain, UTF-8 encoding, and HTTP status 200 OK.

The technical requirements are simple, but they must be met precisely, because an agent fetching the file has no way to work around a configuration error.

What the 2026 data shows

The broadest available study was conducted by Ahrefs in May 2026 on 137,210 domains from its Web Analytics, examining every request to the /llms.txt path and classifying them by user agent. The results are unambiguous.

SEO tools checking for the file's presence generate roughly twenty times more traffic to it than the AI systems that were supposed to use it. It is an industry measuring itself.

A second study, by EZY Research, approached it from the site owner's perspective: 83 sites with llms.txt deployed, 12 weeks of server log monitoring. OpenAI's bots fetched the file seven times.

It is worth honestly noting the discrepancy in adoption figures. Ahrefs reports 28%, while SE Ranking, on a sample of 300,000 domains, reports 10.13%. The samples and file qualification methods differ. As for read frequency, however, the studies agree.

Our own logs: eight fetches in eight days

Someone else's study can always be challenged on sample selection, so we checked our own domain. The llms.txt file for modulla.ai sits at modulla.ai/llms.txt. We pulled every request to that path from Cloudflare between 3 and 10 August 2026, together with the user agent.

Eight requests across eight days. Not one came from an AI crawler: no GPTBot, no ClaudeBot, no PerplexityBot, no Google-Extended. Four requests came from ordinary browsers, meaning people who looked at the file. Two came from the mcpsee-observatory scanner, which catalogues whether the file exists. The last two were our own checks via curl.

It is the same proportion Ahrefs reports, only on a single site and in our own logs: the file is read by tools measuring its existence and by people out of curiosity, not by models building answers.

Two honest caveats about these numbers. The window is short, because Cloudflare accepts at most one day per query for detailed data, and further back than a week our plan returns nothing we could distinguish from an absence of traffic. And no fetches today settles nothing about the future: this measures the present state, it does not forecast.

Google's position: the file is ignored

Google stated this directly in its Search Central documentation for AI features. The llms.txt files and markup designed specifically for language models are bypassed by the search engine. The same documentation dismisses two other popular tactics: splitting content into small fragments to suit alleged model preferences, and tools promising optimization for AI-generated answers.

The mechanism Google actually uses is RAG on the search index. The model expands a query with related questions, retrieves matching pages from the same index used for regular results, and bases its answer on those pages. There is no separate channel or format for AI. A page is either in the index or it is not.

The practical conclusion: if the goal of the implementation was to appear in AI Overviews, llms.txt does not achieve that goal, and there is no basis for believing it will.

Two use cases that must be distinguished

The conversation around llms.txt is confused because it conflates two different mechanisms under one name. Separating them clarifies the implementation decision.

Use caseHow it was supposed to workActual state
Visibility in AI search enginesAn indexing crawler reads the file and cites the brand more often in responsesNo evidence. 1.1% of requests; Google deliberately ignores the file
Documentation for coding agentsAn agent fetches the file on demand when a developer is working with your APIWorks, and visible in logs. The primary real consumer of the file

The difference comes down to who reaches for the file and when. In the first case, a crawler would visit it on its own initiative while building an index, and the studies find no such behavior. In the second case, the file is fetched by an agent at a specific moment when a person has asked it to work with a given piece of documentation. That is an on-demand request, not indexing.

How to configure an llms.txt file: structure and technical requirements

Configuring the file takes between 20 and 60 minutes with a manual approach. The structure is strictly defined by the specification:

Required structural elements

  1. H1 heading (#): the project or brand name. Required, it is the only line that all parsers treat as an entity identifier.
  2. Blockquote (>): 1 to 3 sentences describing the site's mission and scope of activity. Functions as an "elevator pitch" for the algorithm.
  3. H2 sections (##): link categories, including Services, Documentation, FAQ, Pricing, Policies.
  4. Annotated link list: format Title: Description. The description after the colon is critical, as it helps the agent decide whether a given resource is worth fetching without loading the page first.

The Optional section: a signal for models with limited context

The specification reserves the ## Optional section for secondary resources. Links placed there may be skipped by agents operating under token context limit pressure. It is an elegant prioritization mechanism: you tell the AI what is absolutely essential and what can wait.

Server technical requirements

This standard solves a problem that sitemap.xml cannot: it not only indexes pages but explains to the algorithm which ones are worth reading.

Who actually uses llms.txt

Technical products with API documentation. A check from August 10, 2026 shows that the file is published by, among others:

This is not cargo-culting. Users of these products work today in Cursor, Claude Code, and similar tools. When a developer asks an agent how to use an API, the agent fetches the documentation in Markdown instead of wading through HTML with navigation, cookie banners, and chat widgets. The Ahrefs study confirms this from the log side: among bots actually fetching files, coding agent infrastructure is visible, including Claude Code, alongside GPTBot at 4.51%.

For a company with a technical product, llms.txt is therefore part of the developer experience, much like a good README or SDK examples. The return comes from someone integrating your API faster, not from a search ranking.

When llms.txt is just decoration

A services page, a company brochure site, an e-commerce store, a corporate blog. If nobody is building anything on your API, no coding agent has a reason to reach for your documentation, because it does not exist. What remains is the AI search engine path, and that Google has explicitly closed off.

The file costs a few minutes and zero maintenance, so its presence harms no one. What does cause harm is planning a budget around it and expecting a visibility boost. The same time invested in content based on original data works toward citations in a meaningful way, because models draw from the search index, and content, not a helper file, is what enters the index.

For the record: we publish llms.txt on modulla.ai ourselves and treat it as exactly that, decoration, because modulla.ai does not offer API documentation. We keep it because it costs nothing, but we are not counting on it to deliver anything.

Common mistakes when implementing llms.txt

The biggest mistake is the "sitemap" approach: listing every URL on the site instead of selecting 10 to 20 of the most important pages. This is not a content discovery tool, it is a content prioritization tool. Other typical problems:

These mistakes matter where the file is actually being read, that is, with technical documentation. An outdated file is then worse than no file at all, because the agent will suggest a function to the developer that no longer exists.

FAQ: llms.txt in practice

Does llms.txt affect visibility in Google and AI Overviews?

No. Google states in its Search Central documentation that it ignores llms.txt files and markup created specifically for language models. AI-generated answers are built from the same index as regular search results, so presence in them is determined by a page's indexability and content quality.

Does implementing llms.txt make sense for my company?

It depends on whether you offer technical documentation. If you have an API, an SDK, or a product that developers integrate, the file makes life easier for coding agents and the implementation pays off. If you run a services site or an online store, you can add the file, but do not plan for results around it.

How do I check whether anything is fetching my file?

Analyze server logs or CDN-level analytics, breaking down requests to /llms.txt by user agent. That is the only method that gives an answer about your domain specifically, rather than the market average. Pasting the file's URL into a chat proves nothing, because in that case you are the one instructing the model to fetch it.

Do I need to update the file with every change on the site?

For technical documentation, regenerate it in your CI/CD pipeline alongside the documentation itself, because an outdated file does more harm than no file: the agent will suggest a non-existent function to the developer. For a small site, manual updates are sufficient.

Sources