Will AI agents visit your website? Google's CEO says they'll come two ways
Yes, and in two ways. Google CEO Sundar Pichai expects AI agents to reach websites by browsing pages the way a person does, and through protocols built for agents, such as Google's Universal Commerce Protocol (UCP). He expects both to grow: protocols scale better but will take longer to arrive, so browsing agents will keep visiting pages in the meantime.
- Pichai said AI agents will reach websites both by browsing like a user and through agent protocols, and that "you have both ways and they're going to develop."
- He called protocols "a far more scalable approach" but expects them to take longer to develop.
- Google's hosted agents browse with the Google-Agent user agent. Google lists it as a user-triggered fetcher, and these fetchers generally ignore robots.txt rules.
- Google says merchants can adopt UCP to enable agentic actions in AI Mode and Gemini, starting with direct buying.
What happened
In an interview with Alex Heath, reported by Search Engine Journal on 22 October, Pichai described "a complex moment" for agents and the web. He pointed to "a lot of momentum behind MCP or agent protocols or payment protocols" and predicted "a chaotic evolution" before standards settle. He also said Google plans to make Gemini's agent features, mostly limited to paid plans today, more widely available over the next few months.
What this means for your site or store
Think of it as two front doors. Through the first, an agent reads your pages like a shopper. Through the second, it calls your shop directly through a protocol. You need both to work.
- Put prices, stock, delivery times and policies in plain text on your pages, not only in images or pop-ups.
- Keep your basket and forms simple. An agent that browses goes through them step by step, just as a person would.
- If you sell through Google Merchant Center, keep your product feed accurate and follow Google's UCP programme.
For the store side of this shift, see how BigCommerce turned on WebMCP for its stores.
The technical details
Google-Agent uses the IP ranges in Google's user-triggered-agents.json file. Because a person asks for the visit, robots.txt usually won't stop it, and user-agent strings can be faked, so check the IP ranges before you trust or block one. Protocols such as MCP (a standard way for AI agents to use a website's tools) and UCP are the second door. robots.txt and llms.txt still matter for the crawlers that feed AI search answers.
Create free robots.txt and llms.txt files with our free generator, or get robots.txt, llms.txt and schema written for your site with the AI-Ready Kit.
FAQ
Sources
- Google CEO Says Agents Will Hit Sites Two Ways, Pages And APIs, Search Engine Journal
- List of Google user-triggered fetchers, Google for Developers
- Getting started with Universal Commerce Protocol on Google, Google for Developers
