Your SEO platform flagged a missing llms.txt file and called it a site issue. It isn’t one. Somewhere in the last year, a small Markdown file at the root of your domain turned into the thing every audit tool nags you about and every other LinkedIn post swears you need. The pitch is tidy. Robots.txt tells crawlers where they can go, sitemaps tell them what exists, so llms.txt must be the file that gets your brand into ChatGPT. That logic is clean. The crawler data says it’s also wrong.
What llms.txt actually promises
llms.txt is a proposal from Jeremy Howard, published in September 2024. The idea is reasonable. Raw HTML is a mess of navigation, cookie banners, pop-ups, and scripts, and a language model has to strip all of that away before it can read your actual content. An llms.txt file hands the model a clean, curated Markdown overview of your site so it can skip the cleanup. That’s the whole design. It helps a model use your site at inference time. It was never built to change your ranking inside an AI answer box.
Somewhere between the proposal and the pitch deck, that got lost. Tools started treating llms.txt like the next robots.txt. It reads like progress. It’s mostly a category error.
The crawler data is in, and it isn’t close
OtterlyAI ran the experiment everyone kept theorizing about. They put a correctly built llms.txt at the root of a site and watched the server logs for 90 days. Out of more than 62,000 AI bot visits, 84 touched the llms.txt file. That’s about a tenth of a percent. An average content page on the same site pulled roughly 265 bot visits in the same window, so the file the whole industry keeps nagging you about performed three times worse than a normal page, and no better than a stray PDF. The logs tell the story the theorizing couldn’t.
It’s not one study, either. Search Engine Land tracked llms.txt across ten sites and found two with AI traffic gains, neither traceable to the file. Google has said plainly that Search doesn’t use llms.txt, and John Mueller compared it to the old keywords meta tag, which is about the most polite way an engineer can call something dead. Adoption keeps climbing, north of a hundred thousand files indexed this spring, and the bots keep not reading them. Publishing a file isn’t the same as being read. The gap between those two is the whole story.
Where llms.txt actually earns its place
This is where the honest answer splits from the hot take. llms.txt isn’t useless. It’s just aimed at a different target than the one marketers keep firing at.
If you run documentation, an API, or a product that other people’s AI tools plug into, llms.txt is a genuinely good idea. Stripe, Vercel, Cloudflare, and Anthropic all ship one, and none of them are chasing a ranking. They’re making it cheap for a third-party AI assistant to pull a clean version of their docs instead of scraping and de-noising a page on every request. When a tool has to hit a search API at ten to fourteen dollars per thousand queries, a pre-built Markdown snapshot saves real money and returns better answers. That’s an integration decision. It’s an engineering courtesy to the developers building on top of you.
So the test is simple. If people build AI products on top of your content, llms.txt does something real. If you just want Perplexity to mention your brand, it does nothing. Same file, two very different jobs.
What actually moves your AI visibility
The uncomfortable part is that the things that get you cited by an AI engine are the same things that have always worked, which is why nobody’s selling them as a shortcut. Write content with a real point of view. Structure it so a machine can tell what the page is about. Mark it up with clean, honest schema. Earn references from places that already carry authority. We’ve made this argument before, from the strategy side: if your team can’t agree on what you stand for, neither can ChatGPT.
None of that fits in a file you drop at your root and forget. It’s slower, it’s harder, and it’s the actual work. A tidy llms.txt on a thin, forgettable site is a clean label on an empty jar.
If you want a site that AI engines can actually read and want to quote, that’s a content and build problem, not a config file. That part we can help with. Add the llms.txt too, if you like. Just don’t file it under strategy.