How to add robots.txt, llms.txt and schema to a WordPress site

Before you start

You need three things:

1. An Administrator account, so you can change plugin settings.
2. File editing allowed. If wp-config.php contains define( 'DISALLOW_FILE_EDIT', true );, Yoast hides the File editor. You'll then need FTP or your host's file manager.
3. One SEO plugin. Yoast SEO and Rank Math each create their own robots.txt rules, llms.txt and schema. Running both is a recipe for duplicate or conflicting output.

If you don't have your files written yet, the free Web2Agents .txt generator builds a robots.txt and llms.txt you can paste straight in.

Step 1: Set your robots.txt rules

robots.txt tells crawlers, including AI search bots, which parts of your site they may access. One wrong line can hide your site from AI answers, so start here.

Know which file you're editing

A fresh WordPress install has no robots.txt file on disk. The do_robots() function builds one on the fly:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Yoast's robots.txt guide explains that WordPress only generates this virtual file when the site root doesn't contain a physical one. So there are really two cases:

Two details worth knowing:

• Rank Math edits a virtual file. According to Rank Math's General Settings guide, changes in its editor have no effect if a real robots.txt file already exists in your root folder. Its troubleshooting guide says to remove that file by FTP or cPanel first.
• Developers can use a filter. The robots_txt filter in do_robots() lets a theme or plugin change the virtual file. It does nothing once a physical file exists.

What to put in it

A sensible starting point for a site that wants to be cited by AI assistants but opt out of AI training looks like this. Keep /wp-admin/ blocked in every group:

# AI search and user-triggered fetchers: allowed, so you can be cited
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.yourdomain.com/sitemap_index.xml

Use your real sitemap address. With Yoast it's /sitemap_index.xml. Without an SEO plugin, WordPress's own sitemap is at /wp-sitemap.xml.

If you're happy for your content to be used in AI training, leave out the middle group. That's a business choice, not a technical one. For the reasoning behind each bot, see our guide on allowing AI search bots while blocking AI training.

Why repeat /wp-admin/? A crawler follows only the group that matches its name most specifically. So OAI-SearchBot reads its own group and ignores the User-agent: * rules.

Two WordPress traps

• "Discourage search engines" isn't a robots.txt rule. The checkbox in Settings > Reading has added a noindex robots meta tag since WordPress 5.3, according to the Reading Settings documentation. Before 5.3 it served Disallow: /. Either way, it asks search engines to stay away. Make sure it's off on a live site.
• Cloudflare can add rules on top. If your site runs through Cloudflare and you turn on its managed robots.txt, Cloudflare's robots.txt setting docs say it prepends its own directives to your file. Its example blocks GPTBot, ClaudeBot, Google-Extended and other AI crawlers. That may be what you want, but check that the final file still matches your intent.

Step 2: Create your llms.txt

llms.txt is a short Markdown file that gives AI agents a curated map of your key pages, following the llms.txt proposal. On WordPress you have three ways to make one. Pick one only.

Option A: Yoast SEO

According to Yoast's llms.txt help article:

1. Go to Yoast SEO > Settings > Site features.
2. Under AI tools, turn on llms.txt.
3. Click Save changes.
4. Optional: under Customize llms.txt file, choose Manual page selection and pick your key pages.

Yoast's functional specification explains how the file is built:

• It's a real file in your site root, refreshed weekly by a scheduled action.
• Automatic mode lists the 5 most recently updated posts and pages, giving priority to cornerstone content. Posts are only included if they were published in the last 12 months.
• It also lists the 5 categories or tags with the most content.
• It won't overwrite an existing llms.txt file. Delete the old file if you want Yoast to take over.
• It isn't available on multisite installs.

For a business site, manual page selection is usually the better choice. Your five latest blog posts are rarely the pages you most want an AI assistant to read. Your services, pricing, about and contact pages usually are.

Option B: Rank Math

According to Rank Math's llms.txt guide:

1. Go to Rank Math SEO > Dashboard and turn on the LLMS Txt module.
2. Open Rank Math SEO > General Settings > Edit llms.txt.
3. Choose the post types (such as Pages) and taxonomies to include.
4. Set the posts/terms limit. The default is 100.
5. Use Additional Content for any extra links or text.
6. Click Save Changes, then use the preview link.

The file is empty until you configure it, and posts set to noindex are left out. If you don't see the module, Rank Math says to update the plugin and switch to Advanced Mode.

A limit of 100 can turn llms.txt into a second sitemap. Keep it short: your key pages, each with a one-line description.

Option C: Upload your own file

If you'd rather write it yourself, upload an llms.txt file to your site root by FTP or your host's file manager. Yoast's spec says to disable its feature first, so the two don't compete. A minimal file looks like this (the bakery and links are made up):

# Forno Azul

> Forno Azul is a family bakery in Porto. We bake sourdough bread and pastries daily and deliver to cafés across the city.

## Products

- [Bread](https://www.fornoazul.example/bread): Sourdough loaves, daily menu and prices
- [Wholesale](https://www.fornoazul.example/wholesale): Delivery for cafés and restaurants, minimum order

## About

- [Our bakery](https://www.fornoazul.example/about): Address, opening hours and contact

Keep expectations honest. Google's AI optimization guide says that for Google Search you can ignore "unnecessary AI text files (like llms.txt)". An Ahrefs study of 137,210 domains found 97% of published llms.txt files received zero requests in May 2026. On WordPress it takes a toggle, so it's cheap to have. Just don't do it before robots.txt and your content are right. More in our explainer what is llms.txt.

Step 3: Check and complete your schema

Schema markup tells search engines and AI tools what a page is about: who runs the site, what a product costs, when an article was published. On WordPress your SEO plugin already outputs most of it. Your job is to fill in the details and avoid duplicates.

Yoast SEO

According to Yoast's schema documentation, every page gets one connected JSON-LD graph:

• Organization (or Person, if the site represents an individual), WebSite and WebPage on every page.
• Article on single posts, linked to the WebPage.
• FAQPage and HowTo when you use Yoast's FAQ or How-to blocks.

Yoast's Organization spec requires a name and a logo. If either is missing, the Organization piece isn't output. So fill in your organization name and logo in Yoast's settings, and add your real social profiles. That's the five-minute fix most sites skip.

Rank Math

Rank Math's Article schema guide says Article is the default schema for posts and pages. You can change it per post type in Titles & Meta. For your business details, Rank Math's Organization schema guide walks through the Local SEO settings: name, logo, address, phone and business type.

Adding your own JSON-LD

If you need something your plugin doesn't cover, such as Product for a service with a fixed price, add one JSON-LD script with a code snippets plugin or your child theme's header. Don't paste a second Organization block if your SEO plugin already outputs one. Two descriptions of the same business that disagree are worse than one.

What schema won't do: Google says structured data isn't required for generative AI search and there's no special schema.org markup to add (AI optimization guide). It still helps rich results. Think of schema as accurate labelling, not a ranking trick.

Step 4: Check everything on the live site

After you save, check each file in a private browser window:

1. robots.txt: open https://yourdomain.com/robots.txt. You should see your own rules, not the three default WordPress lines. If you use Cloudflare, look for any managed block at the top.
2. llms.txt: open https://yourdomain.com/llms.txt. A 404 usually means the feature is off, Rank Math has nothing configured yet, or a cache is serving an old response.
3. Schema: paste a page URL into the Schema Markup Validator to catch errors, and into Google's Rich Results Test to see which rich results the page qualifies for.
4. Caches: purge your caching plugin and CDN after changes.
5. Google: Google says it generally caches robots.txt for up to 24 hours, so give it a day before worrying.

Common WordPress mistakes

Get your WordPress site ready for AI

Run the free AI visibility check to see whether your robots.txt blocks AI bots and how your homepage scores. To write the files yourself, use the free .txt generator. The AI-Ready Kit delivers a robots.txt, llms.txt, llms-full.txt and JSON-LD schema written for your site, with step-by-step install notes for WordPress.