llms.txt & AI discovery files
Definition

What is llms-full.txt, and does your website need one?

Short answer

llms-full.txt is a single plain-text or Markdown file, usually at /llms-full.txt, that holds the full text of a site's key pages so an AI tool can load it all from one link. It's a convention from developer documentation sites, not part of the llms.txt proposal, and Google Search ignores it. It's useful for docs that people feed to coding assistants. For a shop, a local business or a service site, it's optional and low priority.

Cover: What is llms-full.txt? Definition article on the single-file version of llms.txt, who publishes it and whether your website needs one.
On this page

    Key facts

    • It isn't in the proposal: the llms.txt proposal at llmstxt.org, updated to version 2 in August 2026, defines /llms.txt and Markdown versions of pages (page.md). It doesn't define a file called llms-full.txt.
    • The proposal now points the other way: version 2 says agents should read llms.txt and "then follow the relevant links", and that "the file itself stays small enough to fit in context". The tool that expanded llms.txt into one big file "is no longer part of the proposal".
    • Google Search doesn't use it: Google says you don't need "AI text files" to appear in Google Search, including its AI features, "as Google Search itself doesn't use them".
    • The big AI companies publish one for their docs: on 21 November 2026, Anthropic's, OpenAI's and Perplexity's developer docs all served an llms-full.txt.
    • Real files are huge: we measured Cloudflare's developer docs file at 70.6 MB (about 6.7 million words) and Anthropic's at 54.7 MB (about 6 million words). Perplexity's file opens with a warning that it "is very large and may be truncated".
    • Small sites make small files: our own llms-full.txt for web2agents.com is about 207 KB, roughly 35,000 words.

    What llms-full.txt is

    An llms.txt file is a short map: a title, a one-line summary and a list of links to your important pages. An llms-full.txt file is the full version. Instead of links, it contains the text of those pages, one after another, in one file.

    The idea is simple. Some AI tools let you paste a link and load what's behind it. A developer who wants an AI coding assistant to know a library can give it one URL instead of fifty. Cloudflare's docs describe exactly that use: in editors such as Zed or Windsurf, "you can use llms-full.txt files to provide comprehensive documentation context for indexing."

    A typical file looks like this:

    # Example Bakery - full content
    
    > Family bakery in Porto. Sourdough, pastries and custom cakes. Open Tuesday to Sunday, 7:00 to 19:00.
    
    This file holds the main text of each public page of https://www.example.com/ in plain text.
    The short index is at https://www.example.com/llms.txt.
    
    ---
    
    ## Custom cakes
    
    URL: https://www.example.com/custom-cakes
    
    We make custom cakes for birthdays and weddings. Order at least 5 days ahead.
    Prices start at €35 for a 6-person cake...
    
    ---
    
    ## Delivery and collection
    
    URL: https://www.example.com/delivery
    
    We deliver within Porto on Saturdays. Collection is free from the shop...
    

    There's no official format. Most files follow the llms.txt layout at the top (an H1 and a short summary in a blockquote), then one section per page with its URL and its text.

    llms.txt vs llms-full.txt vs Markdown pages

    llms.txt llms-full.txt Markdown page versions (page.md)
    What it holds A short index of links with one-line descriptions The full text of many pages in one file The clean text of one page
    Where it lives /llms.txt (or /docs/llms.txt for a section) Usually /llms-full.txt Same URL as the page, with .md
    Defined by the llms.txt proposal? Yes No Yes
    Typical size A few KB From a few hundred KB to tens of MB Same as the page text
    Who reads it Mostly AI agents and coding tools, when pointed to it Coding tools and people pasting it into an AI chat Agents that follow links from llms.txt
    Used by Google Search? No No No special treatment

    Where llms-full.txt came from

    llms.txt was proposed by Jeremy Howard, co-founder of Answer.AI, on 3 September 2024. The first version also described a small tool, llms_txt2ctx, for expanding an llms.txt into one context file for an AI model. Separately, documentation sites began publishing their own all-in-one text file under the name llms-full.txt.

    The name stuck, but it never became part of the proposal. When llmstxt.org published version 2 in August 2026, it went further. Its Changes page says version 2 "states the expectation directly: agents view or search the llms.txt to find what they need, then follow the relevant links", and that "the context-expansion tooling is no longer part of the proposal". The page also says context windows "are still too small for most websites in their entirety".

    So the current proposal's answer to "how does an agent read my whole site?" is: it doesn't. It reads the index and fetches the pages it needs, ideally as clean .md versions.

    Who actually publishes one

    We checked a set of well-known documentation sites on 21 November 2026.

    Site llms-full.txt Size we measured What stands out
    Cloudflare developer docs Yes 70.6 MB, about 6.7 million words Also offers smaller per-product files, such as one for Workers (5.4 MB)
    Anthropic (Claude) developer docs Yes 54.7 MB, about 6 million words Its llms.txt ends with "For more comprehensive documentation, see llms-full.txt"
    Vercel docs Yes 13 MB, about 1.5 million words
    OpenAI developer docs Yes 10.2 MB, about 1.1 million words Opens with links to smaller per-product llms.txt indexes
    Perplexity docs Yes 3.5 MB, about 446,000 words Opens with: "This file is very large and may be truncated"
    Model Context Protocol docs Yes 2.6 MB, about 331,000 words Same truncation warning
    Svelte docs Yes 1.3 MB, about 173,000 words Also offers medium (915 KB) and small (53 KB) versions
    Stripe docs Not at that URL Returned a 404

    Two patterns are clear. Almost every example is software documentation, the use the llms.txt proposal itself calls the heaviest. And the bigger the docs, the more these sites split the file up or warn that it won't load in full. A 6-million-word file is far too long for an AI tool to read in one go, so it ends up being searched or cut short, much like a normal website.

    Does llms-full.txt help with AI search?

    There's no evidence that it does.

    • Google: its guide to generative AI features in Search says you don't need AI text files and that keeping them "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
    • OpenAI, Anthropic and Perplexity: they publish llms-full.txt for their own developer docs. We found no page from any of them saying their search products use llms-full.txt to decide what to cite.
    • Usage data: the largest public server-log study we know of looked only at /llms.txt. It found most llms.txt files are never requested, and that AI agents fetch the file when something points them to it. The little data published on llms-full.txt comes from documentation vendors measuring their own customers' sites, and it counts visits, not citations. A fetch doesn't show the file was used.

    We went through that llms.txt evidence in Does llms.txt work?. The short version applies here too: these files are for AI tools that are pointed at them, not a lever for AI search rankings.

    Does your website need one?

    Your site Worth it? Why
    Software, API or developer documentation Yes Developers paste docs into coding assistants. This is where every big example comes from
    SaaS or product with a help centre Probably Useful when customers or support staff load your help content into an AI chat. Keep it to help and product pages
    Online shop Low priority Your product data belongs in pages, structured data and feeds, which is what Google and ChatGPT shopping read
    Local business or service site Optional Cheap if your site is small or a tool makes it for you. Don't expect visits or citations from it

    If you have to choose where to spend an hour, spend it first on letting AI search crawlers in, on putting the answers people look for in your pages as text, and on a short, accurate llms.txt.

    How to build a small, safe llms-full.txt

    1. Start from your llms.txt. Use the same pages, in the same order. If a page isn't worth listing in llms.txt, it doesn't belong in the full file.
    2. Use the page text only. Leave out menus, cookie banners, footers and repeated calls to action. Keep headings, prices, opening hours, policies and FAQs.
    3. Add the URL above each page's text. That lets a tool, or a person, check the source and see where a quote came from.
    4. Keep it small, or split it. If the file runs past a few hundred KB, publish one per section (for example /docs/llms-full.txt and /help/llms-full.txt), the way Cloudflare does per product.
    5. Serve it as text. It should load with a 200 status, as text/plain or text/markdown in UTF-8, without a login, and robots.txt shouldn't block the path.
    6. Link to it from llms.txt. Add it to the ## Optional section of your llms.txt, for example - [Full text for AI assistants](https://www.example.com/llms-full.txt): The main page content of this site in plain text.
    7. Rebuild it when pages change. An old copy with last year's prices is worse than no copy. Regenerate it whenever you publish, or at least monthly.
    8. Publish only what's already public. Anything in the file can be quoted. Never include internal notes, staff details or text you wouldn't put on the site, and check what a plugin generates before you publish it.

    For how these files fit next to robots.txt and your sitemap, see robots.txt vs llms.txt vs sitemap.xml, or start with our plain-English definition of llms.txt.

    Common mistakes

    • Treating it as an SEO file. Google ignores it, and no AI search company says it uses it for citations.
    • Dumping the whole site. Thousands of tag pages, archives and old posts make the file bigger without making it more useful.
    • Letting it go stale. The file doesn't update itself when your pages change, unless your platform rebuilds it.
    • Making it the only place a fact lives. If something matters, such as a price, a policy or an address, it has to be on the page too.
    • Not checking it loads. Open /llms-full.txt in a browser. A login wall, a 404 or a firewall challenge means no tool can read it.

    Check your site

    The free Web2Agents AI check scores your homepage out of 30, tests 12 AI bots against your robots.txt and checks your robots.txt, sitemap.xml and llms.txt, with results on screen. You can build a robots.txt and llms.txt with the free generator. The €19 AI-Ready Kit audits up to 25 pages and includes an llms.txt and an llms-full.txt written from your own pages, plus robots.txt and schema snippets.

    {"@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [{"@type": "Question", "name": "Is llms-full.txt an official standard?", "acceptedAnswer": {"@type": "Answer", "text": "No. Neither llms.txt nor llms-full.txt is a formal web standard. llms.txt is a proposal published at llmstxt.org, and llms-full.txt isn't mentioned in its current version. It's a convention that documentation sites adopted."}}, {"@type": "Question", "name": "What's the difference between llms.txt and llms-full.txt?", "acceptedAnswer": {"@type": "Answer", "text": "llms.txt is a short index of links with one-line descriptions. llms-full.txt holds the full text of the pages in a single file, so it's much larger."}}, {"@type": "Question", "name": "Does Google use llms-full.txt?", "acceptedAnswer": {"@type": "Answer", "text": "No. Google says Google Search, including its AI features, doesn't use AI text files such as llms.txt, and that having them will neither help nor harm your visibility."}}, {"@type": "Question", "name": "Where should llms-full.txt go?", "acceptedAnswer": {"@type": "Answer", "text": "It usually sits at the root, at /llms-full.txt, with a link to it from the site's llms.txt. Some large sites, such as Cloudflare's docs, also publish one per section, for example /workers/llms-full.txt."}}, {"@type": "Question", "name": "How big should llms-full.txt be?", "acceptedAnswer": {"@type": "Answer", "text": "There's no rule. Ours, for a small website, is about 207 KB. If yours grows into many megabytes, split it by section, because AI tools may only read part of a very large file."}}]}

    FAQ

    Is llms-full.txt an official standard?

    No. Neither llms.txt nor llms-full.txt is a formal web standard. llms.txt is a proposal published at llmstxt.org, and llms-full.txt isn't mentioned in its current version. It's a convention that documentation sites adopted.

    What's the difference between llms.txt and llms-full.txt?

    llms.txt is a short index of links with one-line descriptions. llms-full.txt holds the full text of the pages in a single file, so it's much larger.

    Does Google use llms-full.txt?

    No. Google says Google Search, including its AI features, doesn't use AI text files such as llms.txt, and that having them will neither help nor harm your visibility.

    Where should llms-full.txt go?

    It usually sits at the root, at /llms-full.txt, with a link to it from the site's llms.txt. Some large sites, such as Cloudflare's docs, also publish one per section, for example /workers/llms-full.txt.

    How big should llms-full.txt be?

    There's no rule. Ours, for a small website, is about 207 KB. If yours grows into many megabytes, split it by section, because AI tools may only read part of a very large file.

    Sources

    Newsletter

    New guides on AI visibility, in your inbox.

    Practical, sourced articles on AI crawlers, llms.txt, structured data and AI search. A new one every day.

    Unsubscribe anytime. See our Privacy Policy.