How do you check which AI crawlers can access your website?

Why robots.txt alone doesn't tell you

Most guides stop at robots.txt. That tells you what you are asking bots to do. It doesn't tell you what happens when a bot knocks. The official standard, RFC 9309, is clear that its rules "are not a form of access authorization".

Between the bot and your pages there are usually two or three more layers: a CDN, a web application firewall (WAF) and your host. Any of them can return a 403 error, a CAPTCHA or a JavaScript challenge to a bot. A bot that hits a challenge page can't read your content, even if robots.txt says Allow: /.

This matters because the main AI companies tell site owners to allow their search bots at both levels. OpenAI's crawler documentation recommends "allowing OAI-SearchBot in your site’s robots.txt file and allowing requests from our published IP ranges". Perplexity says something similar for PerplexityBot and has a section on configuring WAFs.

The reverse also happens. Some sites block a bot at the firewall and say nothing in robots.txt. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it stops the bot from reading your robots.txt. If you want to opt out, say it in robots.txt.

The bots worth checking

You don't need to test every bot on the internet. Start with the ones that decide whether AI assistants can find, read and cite you.

If you want to decide which of these to allow, our guide on allowing AI search bots while blocking AI training covers the policy. This guide is about checking that your policy actually works.

Step 1: Read your robots.txt rules

Open https://yourdomain.com/robots.txt in a browser. Then answer three questions for each bot in the table above.

1. Is there a group with its name? For example User-agent: GPTBot. If so, only that group counts for that bot.
2. If not, what does User-agent: * say? Bots without a named group follow the wildcard group.
3. Is there a Disallow: / that catches it? One line can hide a whole site.

The trap here is the "one group per bot" rule. Google's robots.txt specification says a crawler follows only the most specific group that matches it, and "other groups are ignored". So if you block /admin/ under User-agent: * and then add a group for OAI-SearchBot, that bot no longer sees the /admin/ rule.

You can check this with a script instead of by eye. Python's standard library includes a robots.txt parser:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://yourdomain.com/robots.txt")
rp.read()
for bot in ["OAI-SearchBot", "ChatGPT-User", "GPTBot",
            "Claude-SearchBot", "ClaudeBot", "PerplexityBot", "Googlebot"]:
    print(bot, rp.can_fetch(bot, "https://yourdomain.com/"))

Treat the result as a quick check, not the final word. Different parsers handle edge cases (wildcards, conflicting Allow and Disallow lines) slightly differently. For Google specifically, Search Console's robots.txt report shows which robots.txt files Google found, when it last fetched them and any parsing errors.

Also check subdomains. Each host needs its own file: shop.yourdomain.com doesn't inherit the rules of www.yourdomain.com.

Step 2: Test what your server returns to each bot

This is the step most people skip, and it catches the most surprises. Request a page while identifying as each bot, and compare the answer with what a normal browser gets.

With curl in a terminal on Mac or Linux (Windows 10 and later include curl.exe too; put the command on one line there):

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://yourdomain.com/

Repeat it with each user agent string. The companies publish example strings in their docs, and the version numbers change, so copy them from the source rather than from old blog posts.

How to read the results:

Look at the body too, not only the status code. A 200 that contains "Checking your browser" or "Verify you are human" means the bot didn't see your content.

One honest limit: this test only shows how your site treats a user agent name coming from your IP address. Many firewalls also check where a request comes from. A rule that verifies bots by IP may let the real GPTBot in and block your fake one, or the other way round. Use Step 2 to find blocks based on the name, and Step 4 to confirm what happens to the real bots.

What we found on our own site

We ran this test on www.web2agents.com on 2 November 2026 at 09:02 Lisbon time, using the homepage and 11 user agents. Our robots.txt allows all of them except Bytespider (ByteDance's crawler), which we disallow.

When we first checked our site, our robots.txt still allowed Bytespider while the CDN in front of our host was already refusing it. The two layers disagreed, and robots.txt alone would never have shown it. We now say the same thing in both places.

Step 3: Check your CDN, firewall and host settings

If Step 2 shows a block you didn't plan, look at the layer that returned it. The response headers often tell you: a server or similar header usually names the CDN.

Common places to look:

• AI-crawler toggles in your CDN. Cloudflare introduced a one-click option to block AI crawlers in September 2024 and, from 1 July 2025, asks every new domain whether to allow AI crawlers, with blocking as the default. If your site went on Cloudflare after that date, check the setting rather than assuming.
• Bot protection modes. Generic "bot fight" or challenge modes can catch AI bots. Perplexity notes that Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules and "can still challenge Perplexity bots".
• Managed robots.txt. Some CDNs can add AI rules to your robots.txt for you. Cloudflare's managed robots.txt adds its directives in front of your existing file. Open the live file to see what bots actually get, not only the version in your CMS.
• Your website builder or host. Hosted platforms may apply their own bot rules at the CDN, outside your control panel. If you can't change them, at least make your robots.txt match, so the two layers say the same thing.
• Security plugins and rate limits. Plugins that block "bad bots" by name sometimes include AI bots in their lists.

If you want to allow a bot through a WAF, follow the company's own instructions. Perplexity's docs, for example, recommend matching both the user agent and its published IP ranges in the rule, so impostors don't get the same pass.

Step 4: Confirm real visits in your logs

The final check is whether the real bots are reaching you. You'll need server logs or your CDN's analytics.

1. Search your logs for the bot names, for example OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot.
2. Look at the status codes they received. A run of 403s or challenges means Step 3 isn't fixed yet.
3. Check the IP addresses. Compare them against the published lists: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json and perplexity.ai/perplexitybot.json.

Step 3 matters because anyone can type "GPTBot" into a user agent. Fake AI bots exist, and some of them probe for passwords. A visit only counts as GPTBot if the IP is in OpenAI's list.

Two things to keep in mind. First, OpenAI says it may add a robots.txt marker to the user agent when fetching that file, so you may see two slightly different strings for the same bot. Second, no visits doesn't always mean you're blocked. A small or new site may simply not have been crawled yet. If Steps 1 to 3 are clean, give it time.

A quick checklist

Check your own site

Want the short version? The free Web2Agents AI check shows which of 12 AI bots can access your homepage, along with a readiness score out of 30. The AI-Ready Kit adds a 19-bot access table, and you can build a clean robots.txt with the free robots.txt and llms.txt generator.