Does llms.txt work? What Google, Chrome and real server logs say
Not for AI search, so far. Google says its Search ignores llms.txt, and an Ahrefs study of 137,210 sites found 97% of llms.txt files got zero requests in a month. Where it does get read is by AI agents and coding tools, mostly on documentation sites. Publish one if it's cheap for you, but don't expect citations from it.

- Google Search doesn't use it: Google's guide to generative AI features says you don't need llms.txt or other AI text files, "as Google Search itself doesn't use them", and that keeping one "will neither harm nor help" your rankings.
- Chrome checks for it anyway: Lighthouse's experimental Agentic Browsing audits include an llms.txt check. A missing file is marked "Not Applicable", not a failure; only a server error is flagged.
- Most files are never fetched: Ahrefs found 28% of the 137,210 domains it monitors publish an llms.txt, and 97% of those files received no requests at all in May 2026.
- AI search bots barely look: AI retrieval bots such as OAI-SearchBot and PerplexityBot made 1.1% of the llms.txt requests Ahrefs measured. AI agents and their tools made 10.5%, with Claude Code the second-biggest AI fetcher after GPTBot.
- No AI bot hunts for a missing file: requests for llms.txt files that didn't exist came almost entirely from humans, and none from AI bots.
- The AI labs use it for their own docs: OpenAI, Anthropic and Perplexity all publish llms.txt files for their developer documentation.
What the companies actually say
The confusion around llms.txt comes from mixed signals. Here is what each source says in its own words, checked on 13 November 2026.
| Who | What they say | What it means for you |
|---|---|---|
| Google Search | "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." | No effect on AI Overviews, AI Mode or rankings, good or bad |
| Google Chrome (Lighthouse) | "Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content." A 404 is marked Not Applicable "as providing the file is optional at the moment." | A crawl-efficiency argument for agents, not a ranking one. No penalty for not having one |
| llmstxt.org (the proposal) | Version 2, updated in August 2026, describes the file as information "to help agents use a website", and gives coding agents reading documentation as the first example | It was designed for agents, not for search rankings |
| OpenAI, Anthropic, Perplexity | Each publishes an llms.txt for its developer docs. We found no page from any of them saying their search crawlers use llms.txt to choose what to cite | They use it as publishers, not as a promise about their bots |
Google's two positions aren't really in conflict. Google Search doesn't read the file. The Chrome team is measuring how ready a site is for browsing agents, and a short index file can save an agent some work. Those are different jobs.
What the server logs show
The best public data we've found is the Ahrefs llms.txt study, first published in June 2026 and updated on 6 November 2026. Ahrefs looked at every request to /llms.txt across 137,210 domains in May 2026.
| Who fetched llms.txt (Ahrefs, May 2026) | Share of requests |
|---|---|
| SEO audit tools (including Lighthouse, Ahrefs' own crawlers and others) | 21.7% |
| Other and unidentified bots | 14.9% |
| General web crawlers, such as Googlebot | 13.1% |
| Tech profiling tools, such as BuiltWith | 11.6% |
| AI agents and agent infrastructure, such as Claude Code | 10.5% |
| GEO and AEO scoring tools | 5.8% |
| AI training crawlers, such as GPTBot and ClaudeBot | 5.3% |
| llms.txt scanners and directories | 3.6% |
| Link-preview bots, such as Slackbot | 2.9% |
| Research bots | 2.7% |
| AI assistants, such as ChatGPT-User | 2.5% |
| AI retrieval bots, such as OAI-SearchBot and PerplexityBot | 1.1% |
| Humans | 4% |
Three things stand out.
- Almost nobody reads the file. 97% of valid llms.txt files got no requests in the month. The other 3%, about 1,100 domains, received all of the traffic, roughly 22,000 requests.
- The bots that decide AI search citations barely show up. AI retrieval bots made 1.1% of requests. Ahrefs notes that Slackbot alone fetched llms.txt files more often than PerplexityBot.
- Agents are the real audience. AI agents and the tools built for them were the biggest AI group at 10.5%. That fits what Google's John Mueller told Lily Ray, as reported by Ahrefs: that llms.txt is "not done for search" and is a "temporary crutch, perhaps to save some tokens" for AI coding tools reading developer documentation.
Ahrefs also warns that a fetch is not proof of use. A bot can download the file and do nothing with it, so every figure above is a ceiling.
Why most llms.txt files go unread
Ahrefs found that AI bots don't go looking for llms.txt. On sites without the file, the requests for /llms.txt came almost entirely from humans, most likely SEOs checking competitors, and none from AI bots. Its conclusion is that agents fetch llms.txt "when directed, not speculatively".
The sites that do get agents to read their file tell them where it is. On 13 November 2026 we checked the developer docs of the AI companies themselves:
- OpenAI's developer docs carry the line "For the complete documentation index, see llms.txt".
- Perplexity's docs show a banner: "For AI agents: see the complete llms.txt documentation index."
When we checked our own site, we found the same gap most sites have. Our llms.txt loads, has an H1 and 13 Markdown links, so it passes Lighthouse's basic checks. But our homepage HTML doesn't link to it, so an agent that lands on the homepage has no pointer to the file. That's a gap worth closing, on our site and probably on yours.
Who should publish an llms.txt?
| Your site | Is it worth it? | Why |
|---|---|---|
| Software, API or developer documentation | Yes | Coding agents like Claude Code are the main readers Ahrefs found. This is the use the proposal was written for |
| SaaS or a product people set up with an AI assistant's help | Probably | Agents acting for users are the AI group most likely to fetch it. Keep it accurate and link to it |
| Online shop, local business or service site | Optional | Low effort if your platform makes one for you, but no evidence it brings AI search visibility or sales |
| A site where you'd have to build and maintain it by hand | Low priority | Spend the time on the things that do affect AI answers first |
If you do publish one, keep the effort small and the file honest:
- Let your platform do it if it can. The llms.txt proposal lists Wix, Yoast SEO and AIOSEO among the tools that can generate one automatically.
- Link to it. Add a plain link to
/llms.txtfrom your docs, footer or help pages, so agents have a reason to fetch it. - Treat it like code. Ahrefs found a research bot called
prompt-injection-surveyamong the fetchers. Keep the file to plain links and descriptions, link only to pages you control, and check what any plugin generates for you. - Make sure it can actually be fetched. Lighthouse fails the check on a server error, and a firewall can stop bots that robots.txt allows. We covered how that happened to one French site in our post on firewalls blocking AI bots.
How to check if anything reads yours
You don't need a study to know your own numbers. If your host gives you access logs, search them for requests to /llms.txt and look at the user agents:
grep "llms.txt" access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
This counts requests by user agent, assuming the common "combined" log format. Look for three things:
- Any requests at all. If the count is zero after a few weeks, the file isn't being read.
- Who is asking. GPTBot, ClaudeBot or Claude-Code mean AI tools. Mostly SEO tools and scanners means the SEO industry is checking your file, not AI.
- The status code. A 404 means someone asked for a file you don't have. A 403 or 5xx means your server or firewall is turning visitors away.
If your site is on a hosted platform without raw logs, your CDN's analytics (Cloudflare, for example) may show the same data.
What matters more than llms.txt
If your goal is to be cited by ChatGPT, Perplexity or Google's AI features, the evidence points elsewhere:
- Let the AI search bots in. Check your robots.txt and firewall with our guide to checking which AI crawlers can access your website.
- Put the answers in your pages as text. Google says its AI features use the same core SEO practices, with content available in textual form.
- Keep facts consistent. Prices, hours, policies and product details should match across your pages, structured data and business profiles.
For how llms.txt fits next to the other files, see robots.txt vs llms.txt vs sitemap.xml, or start with our plain-English definition of llms.txt.
Check your site
The free Web2Agents AI check looks at your homepage, robots.txt, sitemap.xml and llms.txt, and tests 12 AI bots. If you decide you want an llms.txt, you can build one with the free robots.txt and llms.txt generator, or get files written for your own pages in the €19 AI-Ready Kit.
FAQ
Does llms.txt help you rank in Google's AI Overviews?
No. Google says Google Search, including its generative AI features, doesn't use llms.txt, and that having one will neither help nor harm your visibility.
Does ChatGPT read llms.txt?
OpenAI hasn't said that ChatGPT search uses llms.txt. In the Ahrefs study, OpenAI's training crawler GPTBot was the biggest AI fetcher of llms.txt files, while its search crawler OAI-SearchBot fetched them rarely. A fetch doesn't show the file was used.
Why does Lighthouse check for llms.txt?
Chrome's Lighthouse has an experimental Agentic Browsing category that measures how easy a site is for AI agents to use. Its documentation says that without the file, agents may spend more time crawling to understand a site. A missing file is marked Not Applicable, not failed.
Is it bad to have an llms.txt file?
No, as long as it's accurate and loads properly. Google says it won't hurt your rankings. Keep it to plain links and descriptions, because agents may trust what it says.
How do I know if bots read my llms.txt?
Search your server or CDN logs for requests to /llms.txt and look at the user agents and status codes. If there are no requests after a few weeks, nothing is reading it.
Sources
- Google's guide to optimizing for generative AI features on Google Search, Google Search Central
- AI features and your website, Google Search Central
- llms.txt audit, Chrome for Developers
- How the Agentic Browsing category is scored, Chrome for Developers
- We analyzed 137K sites: 97% of llms.txt files never get read, Ahrefs
- The /llms.txt file, v2, Jeremy Howard, llmstxt.org
- OpenAI Developers llms.txt, OpenAI
- Anthropic developer documentation llms.txt, Anthropic
- Perplexity docs llms.txt, Perplexity


















