robots.txt vs llms.txt vs sitemap.xml: what each file does for AI

The comparison at a glance

robots.txt: the gatekeeper

robots.txt is the oldest of the three and the only one that sets rules. It is now an internet standard, RFC 9309, published in September 2022.

What it does for AI: the main AI companies document their bots' robots.txt names, so this file is where you decide who gets in. You can allow AI search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot so you can be cited, and disallow training tokens such as GPTBot, ClaudeBot or Google-Extended. OpenAI's crawler documentation says these settings are independent of each other.

What it can't do:

• It isn't a lock. RFC 9309 says its rules "are not a form of access authorization".
• It doesn't keep pages out of Google's index. Google's robots.txt introduction recommends noindex or password protection for that.
• User-triggered fetchers, such as ChatGPT-User or Perplexity-User, may not follow it, according to OpenAI and Perplexity.

Watch out for: each crawler follows only the single most specific group that matches it. Google's robots.txt specification says "Other groups are ignored", so a bot with its own group won't read your User-agent: * rules.

sitemap.xml: the index

A sitemap is an XML list of the URLs you want search engines to crawl, defined by the Sitemaps protocol.

What it does for AI: it helps pages get discovered and indexed. That matters for AI search, because Google says a page must be indexed and eligible for a snippet in Google Search to appear as a supporting link in AI Overviews or AI Mode (AI features and your website).

Useful facts:

• One sitemap can hold up to 50,000 URLs and be up to 50 MB. Larger sites use a sitemap index.
• You can point crawlers to it from robots.txt with a line like Sitemap: https://www.yourdomain.com/sitemap.xml.
• Google says it ignores <priority> and <changefreq>, and uses <lastmod> only if it is consistently and verifiably accurate (Build and submit a sitemap). So keep lastmod honest and skip the rest.

What it can't do: a sitemap doesn't guarantee indexing, and it says nothing about what the pages mean. It is a list, not a summary.

llms.txt: the guide for AI agents

llms.txt is a Markdown file proposed by Jeremy Howard in 2024 on llmstxt.org. It holds your site's name, a short summary and lists of links to your key pages, written so a language model can read it quickly.

What it does for AI: when an agent does look at it, it gets a clean, small overview instead of having to parse your whole site. The proposal itself notes it is used most heavily for software documentation, where coding agents follow it to find API references.

How it differs from a sitemap, according to the spec: a sitemap lists every indexable page, often far too much to fit in an AI model's context. llms.txt is curated, short, and can include links to external resources.

What it can't do:

• It has no access rules. It doesn't block or allow anything, and it doesn't opt you out of training.
• It isn't required by Google. Google says you don't need AI text files or special markup to appear in AI Overviews or AI Mode.
• It isn't widely read yet. An Ahrefs study of 137,210 domains found 97% of published llms.txt files received zero requests in May 2026.

Which one matters most?

If you're short on time, work in this order:

1. robots.txt first. One wrong line can hide your site from AI search entirely. Check that search engines and AI search bots are allowed.
2. sitemap.xml second. Make sure it exists, lists your important pages, has honest lastmod dates and is referenced in robots.txt.
3. llms.txt third. Add it once the basics are in place. It's quick and low-risk, but don't expect it to move your AI visibility on its own.

None of these files replaces good content. AI assistants cite pages that clearly answer the question being asked. The files only make sure those pages can be found and read.

Check all three in one go

The free Web2Agents scan checks your robots.txt and sitemap and tells you whether AI bots are blocked. To create a robots.txt and llms.txt from scratch, use the free .txt generator. The AI-Ready Kit gives you ready-to-upload robots.txt, llms.txt, llms-full.txt and schema files written for your site.