News
News

Is your site blocking Google's AI agent? Google's new guide shows where to look

Short answer

It might be, without you noticing. Google-Agent is the name Google's AI agents use when they browse websites and take actions for people. On 17 November 2026 Google published a guide to letting it through your CDN, firewall or bot protection. It also lists the places where sites still block it after "allowing" it, from CAPTCHAs to cart pages and rate limits.

News cover: a spider web against a dark background, under the headline asking whether your site is blocking Google's AI agent.
On this page

    Key facts

    • Google says Google-Agent is used by agents hosted on its infrastructure "to navigate the web and perform actions upon user request".
    • Real Google-Agent requests include "Google-Agent" in the User-Agent header and come from IP addresses Google publishes in a file called user-triggered-agents.json.
    • Google is also testing Web Bot Auth, a digital signature that copycat bots can't fake, using the identity agent.bot.goog.
    • Google says verified Google-Agent requests should skip bot challenges, including CAPTCHAs and JavaScript challenges.
    • Because a person asked for the visit, Google says user-triggered fetchers like this one generally ignore robots.txt rules.

    What happened

    On 17 November 2026 Google added a new page to its crawling documentation: Allow Google-Agent traffic. Google-Agent is not a search crawler. It is the agent that visits a site when someone asks Google's AI to do something there, like search a catalogue or add an item to a cart. Google says allowing it in your security tools "helps prevent legitimate requests from being blocked while preserving your existing security controls".

    What this means for your store

    1. Ask your hosting, CDN or bot-protection provider: "Do you verify and allow Google-Agent?" Some services do it for you. If yours doesn't, Google explains how to check the requests yourself.
    2. Test more than your home page: Google warns that the allow rule must also cover on-site search, filters, product pages, add-to-cart, booking and the API calls behind them. An agent that gets stuck at the cart can't finish the job.
    3. Look for CAPTCHAs and rate limits: pop-up challenges on search or forms, and per-IP limits, can stop the agent. Many users' requests leave Google from the same pool of addresses, so they can look like one very busy visitor.
    4. Don't rely on robots.txt: it doesn't decide whether Google-Agent gets in; your firewall does. Our post on firewalls blocking AI bots shows where these blocks hide, and our Meta Muse post covers the same check for another agent.

    The technical details

    Google's suggested rule: confirm the User-Agent contains Google-Agent, confirm the IP is in user-triggered-agents.json and, if the request is signed, verify the Signature and Signature-Input headers for Signature-Agent: g="https://agent.bot.goog" (HTTP Message Signatures, RFC 9421). Then skip bot challenges. Your allow rule must run before broad data-centre blocks, managed bot rules or "block AI agents" categories. Keep IP lists in sync, and make sure load balancers and proxies pass on the original IP and headers.

    Want to know if AI agents can reach, read and buy from your site? Check it with Web2Agents.

    FAQ

    Sources

    Newsletter

    New guides on AI visibility, in your inbox.

    Practical, sourced articles on AI crawlers, llms.txt, structured data and AI search. A new one every day.

    Unsubscribe anytime. See our Privacy Policy.