Is your site blocking Google's AI agent? Google's new guide shows where to look
It might be, without you noticing. Google-Agent is the name Google's AI agents use when they browse websites and take actions for people. On 17 November 2026 Google published a guide to letting it through your CDN, firewall or bot protection. It also lists the places where sites still block it after "allowing" it, from CAPTCHAs to cart pages and rate limits.

- Google says Google-Agent is used by agents hosted on its infrastructure "to navigate the web and perform actions upon user request".
- Real Google-Agent requests include "Google-Agent" in the User-Agent header and come from IP addresses Google publishes in a file called user-triggered-agents.json.
- Google is also testing Web Bot Auth, a digital signature that copycat bots can't fake, using the identity agent.bot.goog.
- Google says verified Google-Agent requests should skip bot challenges, including CAPTCHAs and JavaScript challenges.
- Because a person asked for the visit, Google says user-triggered fetchers like this one generally ignore robots.txt rules.
What happened
On 17 November 2026 Google added a new page to its crawling documentation: Allow Google-Agent traffic. Google-Agent is not a search crawler. It is the agent that visits a site when someone asks Google's AI to do something there, like search a catalogue or add an item to a cart. Google says allowing it in your security tools "helps prevent legitimate requests from being blocked while preserving your existing security controls".
What this means for your store
- Ask your hosting, CDN or bot-protection provider: "Do you verify and allow Google-Agent?" Some services do it for you. If yours doesn't, Google explains how to check the requests yourself.
- Test more than your home page: Google warns that the allow rule must also cover on-site search, filters, product pages, add-to-cart, booking and the API calls behind them. An agent that gets stuck at the cart can't finish the job.
- Look for CAPTCHAs and rate limits: pop-up challenges on search or forms, and per-IP limits, can stop the agent. Many users' requests leave Google from the same pool of addresses, so they can look like one very busy visitor.
- Don't rely on robots.txt: it doesn't decide whether Google-Agent gets in; your firewall does. Our post on firewalls blocking AI bots shows where these blocks hide, and our Meta Muse post covers the same check for another agent.
The technical details
Google's suggested rule: confirm the User-Agent contains Google-Agent, confirm the IP is in user-triggered-agents.json and, if the request is signed, verify the Signature and Signature-Input headers for Signature-Agent: g="https://agent.bot.goog" (HTTP Message Signatures, RFC 9421). Then skip bot challenges. Your allow rule must run before broad data-centre blocks, managed bot rules or "block AI agents" categories. Keep IP lists in sync, and make sure load balancers and proxies pass on the original IP and headers.
Want to know if AI agents can reach, read and buy from your site? Check it with Web2Agents.
FAQ
Sources
- Allow Google-Agent traffic, Google for Developers (17 Nov 2026)
- Google's user-triggered fetchers, Google for Developers
- Crawling infrastructure changelog, Google for Developers (17 Nov 2026 entry)
- New Google Search Help Document On How To Allow Google-Agent Traffic, Search Engine Roundtable (17 Nov 2026)


















