FREE SCAN · AI-READY KIT

How Web2Agents works

Start with the free scan to see where your website stands with AI. Get the AI-Ready Kit when you want the full audit, the fixes and the files.

The free scan.

A quick AI-readiness check of your homepage. Free, no sign-up.

1

Enter your website.

Type your root domain (for example yourwebsite.com) and your email in the form on the homepage.

2

We scan your homepage.

We look at its headings, structured data, links, forms and FAQ content, plus your robots.txt and sitemap.xml.

3

Get your report by email.

An online report with your score out of 30 across 6 areas, and whether your robots.txt blocks AI bots.

Where the free scan stops.

  • It checks your homepage only, not the rest of your site.
  • You get one score per area, not the individual checks or the evidence behind it.
  • Bot access is summarised in 3 groups from your robots.txt. Bots aren't tested one by one.
  • No page-by-page issues and no ranked list of what to fix first.
  • No files: nothing is written for your site to upload.

Diagnosis only. Want to build a robots.txt or llms.txt yourself? Try the Free .txt Generator →

The AI-Ready Kit.

The full audit of up to 25 pages, plus the files to fix what we find. €19 one-time, incl. VAT (launch price, normally €29).

1

Enter your website on the kit page.

We check that your site is reachable before you pay.

2

Pay securely with Stripe.

One-time payment, no subscription. You get a receipt by email.

3

We analyse up to 25 pages.

Our crawler follows your sitemap and internal links, respects your robots.txt, runs 36 checks across the same 6 areas and tests 19 search and AI bots.

4

Your kit arrives by email.

Usually within minutes; larger sites can take a little longer. The download link is valid for 14 days, so save the ZIP.

5

Upload the files.

Follow INSTALL.md to add robots.txt, llms.txt and the schema snippets to your site, with steps for Webflow, WordPress, Shopify, Wix and Squarespace.

What's in the kit.

  • 10-page PDF report: your score out of 30 and all 36 checks, with the evidence we found
  • Up to 10 fixes, ranked by impact and effort
  • AI bot access table for 19 bots: what your robots.txt allows and what your server actually returns
  • robots.txt, ready to upload (your existing rules are kept)
  • llms.txt and llms-full.txt
  • Schema (JSON-LD) snippets: Organization, WebSite and per-page blocks
  • INSTALL.md with step-by-step instructions
Get the AI-Ready Kit →

Under the hood.

How the AI-Ready Kit scan works, taken straight from our code.

Our crawler

Identifies itself as Web2AgentsKit, with a link back to web2agents.com in its user-agent, so you can spot it in your logs.

Pages

Up to 25 HTML pages, starting with your homepage. Same domain only; www and non-www count as one site.

How pages are found

Your sitemap first (from the Sitemap line in robots.txt, or /sitemap.xml), shortest URLs first. Then internal links found on the pages we read.

robots.txt respected

Pages your robots.txt disallows are skipped. If it sets a crawl-delay, we honour it (up to 10 seconds).

Pace and limits

One request at a time, 1.5 seconds apart. 15-second timeout per request, and at most 3 MB read per page. Images, PDFs, CSS and script files are skipped.

What we fetch

Your homepage and its www/non-www twin (to check the redirect), robots.txt, your sitemap, the pages themselves, /llms.txt and /llms-full.txt, plus one homepage request per bot user-agent.

Structured data

We read the JSON-LD in each page's HTML and check that it parses and which schema types it declares.

JavaScript

Not executed. We read the raw HTML your server sends, which is what many AI crawlers read too. Text that only appears after JavaScript runs isn't seen.

Scoring

Pass = 1, warning = ½, fail = 0, weighted by impact (high 3, medium 2, low 1). Each area becomes a 0–5 score, and the six add up to your score out of 30.

The 36 checks.

Grouped into the same 6 areas as the free scan. Each one is marked pass, warning or fail, with the evidence we found.

Content Semantics

7 checks
  • One H1 per page
  • Logical heading order (no skipped levels)
  • Semantic landmarks (main, nav, header, footer)
  • Page language declared
  • Unique, descriptive page titles
  • Unique meta descriptions
  • Images have alt attributes

Structured Data

7 checks
  • JSON-LD present on pages
  • JSON-LD parses without errors
  • Organization or business described
  • WebSite entity declared
  • Absolute URLs inside structured data
  • Open Graph tags (title, description, image)
  • Breadcrumbs on inner pages

Navigability

7 checks
  • HTTPS with one consistent host
  • robots.txt with explicit rules
  • XML sitemap found and referenced
  • Absolute canonical URL on every page
  • Crawled pages respond 200
  • Homepage links to key pages with real links
  • No unintended noindex

Actionability

5 checks
  • Form fields have labels
  • Submit buttons have clear text
  • No hard CAPTCHA wall on forms
  • Machine-readable contact (email, phone or ContactPoint)
  • Actions and offers described in schema

AI Promptability

5 checks
  • Content readable without JavaScript*
  • Enough text per page (150+ words)
  • Question-and-answer (FAQ) content
  • Clear, concise sentences (22 words or fewer on average)
  • Descriptive link text (not “click here”)

Agent Collaboration

5 checks
  • llms.txt published
  • llms-full.txt published
  • robots.txt names AI bots explicitly
  • AI search and assistant bots allowed
  • Your server/CDN serves AI bot user-agents

* We don't run JavaScript, so this check currently shows as a warning marked “not tested”.

The 19 bots we test.

From OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Amazon, Meta, Common Crawl, ByteDance, DuckDuckGo and Mistral.

Search & AI answer engines

7 bots
OAI-SearchBot
Claude-SearchBot
PerplexityBot
Googlebot
Bingbot
Applebot
Amazonbot

AI assistants fetching live

5 bots
ChatGPT-User
Claude-User
Perplexity-User
DuckAssistBot
MistralAI-User

AI training crawlers

7 bots
GPTBot
ClaudeBot
Google-Extended
Applebot-Extended
CCBot
Bytespider
Meta-ExternalAgent

What “tested” means.

  • robots.txt: we read your rules for each bot by name, the way Google documents it (the most specific rule wins; * and $ wildcards supported), and note whether you name the bot explicitly.
  • Live request: we request your homepage once with each bot's published user-agent and record what your server returns. A 401, 403, 406, 429, 451 or 503 response, or a challenge page, counts as blocked.
  • Google-Extended and Applebot-Extended have no crawler of their own, so they're checked in robots.txt only. That makes 17 live requests.

Live requests come from our server, not the bots' own networks. Some CDNs verify real bots by IP address, so the real crawler can be treated differently.

Inside the report.

The real 9-page report for web2agents.com: 27/30, 13 pages analysed, 36 checks.

Page 1
Your score out of 30, a chart of the 6 areas, and a 0–5 bar for each area with the checks passed.
Page 2
Are bots being blocked? (search crawlers, AI training bots, AI assistants) and a plain-English summary.
Page 3
Top fixes, ranked: up to 10, failures first, then by impact and effort. Each shows the evidence and whether a kit file solves it.
Pages 3–5
Detailed results: all 36 checks by area, each marked pass, warning or fail with the evidence we found.
Pages 6–7
AI bot access table for the 19 bots, then page-by-page issues: titles, meta descriptions, headings, canonicals, images and structured data.
Pages 8–9
Your paste-ready files, with previews of your robots.txt and llms.txt.
AI-Ready Kit sample report for web2agents.com, page 1: score 27/30 with a 0-5 score for each of the 6 areas
Page 1: your score
AI-Ready Kit sample report, page 4: detailed results with pass, warning or fail and the evidence for each check
Page 4: every check, with evidence
AI-Ready Kit sample report, page 6: AI bot access table with the robots.txt rule and live request result for each of 19 bots
Page 6: AI bot access

Sample files.

Short excerpts from the real kit for web2agents.com. Yours are written for your site.

robots.txt

Groups bots by purpose. Training is allowed by default; switch one line to Disallow to opt out.

# --- Search engines and AI answer engines (keep allowed to stay visible) ---
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
# … 4 more
Allow: /
 
# --- AI model training crawlers - policy: ALLOW ---
User-agent: GPTBot
User-agent: ClaudeBot
# … 5 more
Allow: /
llms.txt

A curated map of your site for AI assistants (llmstxt.org format).

# Web2Agents
 
> Free AI-readiness scan: see how AI search tools, assistants and agents read your homepage, scored across 6 areas. Need the fixes? Get the AI-Ready Kit.
 
- Language: en
- Website: https://www.web2agents.com/
 
## Main pages
 
- [AI-Ready Kit: Fix Your Website for AI Search](https://www.web2agents.com/kit): Your full AI-readiness audit plus ready-to-upload robots.txt, llms.txt and schema files. 36 checks, 19 AI bots. One-time €19, delivered by email.
schema/organization.html

Paste-ready JSON-LD for the site-wide <head>.

{
"@context": "https://schema.org",
"@type": "Organization",
"@id": "https://www.web2agents.com/#organization",
"name": "Web2Agents",
"url": "https://www.web2agents.com/",
"logo": {
"@type": "ImageObject",
"url": "https://cdn.prod.website-files.com/…/webtoagents-logo-icon.png"
},
"description": "Free AI-readiness scan: see how AI search tools, …"
}

Free scan vs. AI-Ready Kit.

Free scan
AI-Ready Kit
Price
Free
€19 one-time, incl. VAT
Pages analysed
Your homepage
Up to 25 pages
Checks
Score per area
36 checks, with evidence
Bots
3 groups, from robots.txt
19 bots, robots.txt + live test
Ranked fixes
–
Up to 10
Files to upload
–
robots.txt, llms.txt, schema, INSTALL.md
Delivered
Online report by email
PDF + files (ZIP) by email

FAQ.

What if my site blocks the scan?

Contact us and we'll help you allow access and re-run the analysis. If we still can't deliver a working kit, we refund you in full.

Do I need a developer?

You don't need to write code. INSTALL.md explains where each file goes on Webflow, WordPress, Shopify, Wix and Squarespace. If someone else runs your site, forward them the ZIP.

How long does the kit take?

Usually a few minutes after payment; larger sites can take a little longer. The download link stays valid for 14 days.

Is it a subscription?

No. It's a one-time payment for one website. Questions? Contact us (subject: Customer support).

Ready to fix it?

Your report, your fixes, your files, in one download. €19 one-time, incl. VAT (launch price, normally €29).

Get the AI-Ready Kit →

Secure checkout by Stripe · Delivered by email within minutes · Download link valid for 14 days

Or start with the free scan →