Table of Contents
TL;DR
- An SE Ranking study of 300,000 domains in 2026 found about ten percent had an llms.txt file. A Limy analysis of more than half a billion AI-bot events recorded only around four hundred direct fetches of those files.
- No major AI engine — OpenAI, Google, Anthropic, or Perplexity — has publicly confirmed reading /llms.txt as a ranking or citation signal in 2026.
- Setup takes five minutes. The file is short, plain text, and costs nothing. The honest framing is that it may matter later if any engine starts reading it.
- The mistake is letting the file replace the real work. Clean HTML, crawler permissions, named entities, and clear answer blocks still do the citation work.
- The practical answer for a small site is calm. If your weekly hour is otherwise booked on the real fixes, write the five-minute file. If real fixes are still pending, skip the file and fix those first.
An SE Ranking study of 300,000 domains in 2026 found that about ten percent of websites had set up an llms.txt file. A separate Limy analysis covered more than half a billion AI-bot events across a different sample. The bots fetched the /llms.txt path roughly four hundred times in the whole sample.
Widely set up. Barely read.
The honest answer to the worth-it question follows from those two numbers. Setup is up. Use is barely measurable.
That is the calm answer most AEO posts will not give you in 2026, because the file is easy to oversell. Almost no major AI engine has publicly confirmed using llms.txt as a ranking signal, and Limy’s read of more than half a billion AI-bot events showed almost no direct fetches. The probable practical answer today is that it does not change your citations — but the cost of setting it up is so small that it may still be worth doing as a small bet on next year.
What is llms.txt and what is it supposed to do?
A small plain-text file placed at the root of a website, at /llms.txt. The format is short and the instructions are voluntary.
It is a proposed standard, not an enforced one. The file describes the site’s purpose, names the pages that matter most, and points AI crawlers at a curated reading list of the content you most want them to pick up.
The idea is reasonable. An AI engine reading the file would, in theory, pick sources more accurately. A small site could highlight its three best pages and not get lost behind a hundred shallow ones.
The file is short. It is honest. It is voluntary.
The whole standard is built on the assumption that the major AI crawlers will start reading the file. In 2026 that assumption has not yet been confirmed by any of them.
Has any AI engine actually confirmed reading llms.txt to pick sources?
No major AI engine has publicly confirmed using llms.txt as a ranking or citation signal in 2026.
Google’s 2026 AI search guidance page does not mention the file. The page covers crawler access, structured data, query fan-out, and AI Mode. It says nothing about /llms.txt.
OpenAI’s published crawler documentation does not mention the file. Anthropic’s does not. Perplexity’s does not.
The standard exists. The adopters are unilateral. The reading party has not yet arrived.
That gap is the single most important fact in the worth-it question. A standard with one half of the handshake but not the other is a hope, not a mechanic.
What did the 2026 adoption numbers actually show?
The adoption side of the standard moved fast. The reading side did not.
SE Ranking scanned roughly 300,000 domains in 2026 and found around ten percent had set up an llms.txt file. The frequency rose steadily through the year as practitioners watched competitors publish the file and copied the move.
Limy then analysed more than half a billion AI-bot events across a different sample. The bots fetched the /llms.txt path roughly four hundred times in the whole sample.
Widely set up. Barely read.
The gap is not a fluke. It is the natural shape of a standard whose publication side is much easier than its reading side. Writing a file is unilateral. Reading it is a product decision that every AI vendor has to make and announce separately.
That is the honest reading of the 2026 numbers. Setup is genuinely common. Use is barely measurable.
Why does the file still cost almost nothing to set up?
Because it is a short text file.
Five minutes of writing. No plugin to install. No code change.
No new dependency. No new failure mode. The longest part of the work is deciding which two or three pages you actually want a future AI crawler to pick up first.
The honest cost is small but real. If you change your site’s topic clusters — add a service, retire one, rebrand the consulting — the file needs a one-paragraph edit to stay accurate. A stale llms.txt is worse than no llms.txt, because it sends the wrong signal in the future case where a crawler does start reading it.
Set it once. Glance at it twice a year. That is the entire maintenance plan.
Compared to most AEO work, the file is a rounding-error cost.
When does adding llms.txt actually become a mistake?
When it replaces the work that earns citations.
A reader’s instinct after reading a fresh llms.txt post is that the file is the optimization. It is not. The optimization is still clean HTML, clear short answer blocks, named entities, and crawler access. The file is at most a hedge sitting on top of those.
If your homepage renders only inside a JavaScript bundle, llms.txt does not help. If your robots.txt or your Cloudflare configuration is silently blocking the AI crawlers, llms.txt does not help. If your H2s do not name real reader questions, llms.txt does not help.
The mistake is the substitution. Adding the file should be a small five-minute action at the end of the real audit — not the headline of the work.
A second mistake is the training-crawler confusion. The llms file is not a tool for deciding whether to allow training crawlers. The two questions are separate.
The training-crawler choice is in robots.txt. The llms file is a hint to retrieval crawlers, not a training-crawler permission.
Other questions worth answering
How does the proposed crawler hint differ from robots permission rules?
Robots.txt tells a crawler whether it may visit a page. The proposed crawler hint tells a crawler which pages matter most once it visits. Per the SE Ranking llms.txt adoption study (2026), the new format spread quickly among practitioners.
Robots.txt is a gate. The new hint is a curated reading list.
Who originally specified the crawler-hint standard, and when was it published?
Jeremy Howard published the specification in September 2024. The proposal sat alongside a fuller variant from day one. Howard’s design choice was deliberate: keep the format plain text, keep it short, keep it voluntary. The standard is roughly two years old in mid-2026.
What is the full-text variant of the crawler-hint format, and when would a small business consider it?
It is a flatter version named llms-full that packs the whole resource into one plain-text document. Per the SE Ranking llms.txt adoption study (2026), the format saw rising practitioner uptake but uneven payoff. The trade-off is token budget. An LLM reading the full variant sees more pages but spends more context.
A small WordPress resource with a handful of core pages rarely needs the full variant.
Can WordPress plugins generate the crawler hint for you automatically?
Yes. Yoast and Rank Math, the major WordPress SEO plugins, can auto-generate the hint format. Per the Limy AI-bot crawler-log analysis (2026), the bots fetched these locations only about four hundred times across half a billion events.
The auto-generated version is usually fine for a small business. The honest work is reviewing what the plugin chose to feature.
How should you decide whether to add llms.txt to your site?
Treat it as a low-stakes hedge.
If your weekly AEO hour is otherwise booked on real work — crawler permissions confirmed, pages rendering as static HTML, answer blocks pulling the lede up front, freshness moving — write the five-minute file once this month. The downside is almost nothing. The upside is a small future option if one of the major engines starts reading it.
If your real work is not yet done, skip the file. Fix the crawler block first. Fix the JavaScript-only render first.
Tighten the answer blocks first. The file does not move citations today, and putting it at the top of the list trades real lift for a hedge.
If you read this and still feel uncertain about whether your own site is in the booked or the not-yet category, that is normal. You can contact me here, and we can walk through your site for fifteen minutes and decide together. I do not charge for this kind of call, and there is no pitch waiting at the end.
I will tell you whether the five-minute file is worth doing this month on your site, or whether one of two real fixes deserves the hour first. That is usually enough to keep the AEO work calm and on the right thing.