Can you trust an AI agent to pick the right product? Visa's shopping test says often no
Not yet, not consistently. On 5 November 2026 Visa released its Agentic Commerce Benchmark: 22 AI models from 12 labs took on 15 shopping challenges and failed 42.8% of test runs. About 54% of those failed runs still ended with the AI buying something that did not meet the customer's requirements. The best models passed 82.2% of attempts; the worst passed 28.9%.
- Visa tested 22 AI models from 12 labs on 15 shopping challenges, such as buying a TV, a laptop or a personal care appliance.
- Overall pass rate was 57.2%. The best models passed 82.2% of attempts; the worst passed 28.9%.
- In 54% of failed runs, the AI still made a purchase that missed the customer's needs or skipped some requirements.
- All three attempts at the same task passed only 34.2% of the time across models and scenarios.
- Visa's Trust Index (September 2026) found that 72% of U.S. consumers had used an AI assistant, but only 23% trusted AI to handle payment transactions for them.
What happened
In a press release on 5 November, Visa launched the Visa Agentic Commerce Benchmark. Models had to discover the customer's hidden requirements through conversation, then buy only if something fitted — or refuse to buy if nothing did. Lucky guesses did not count. Anyone can explore the research and try the interactive test on Visa Payments Frontier.
What this means for your store or business
- Make the deciding facts impossible to miss: agents fail when they cannot uncover budget, specs and constraints. Clear titles, attributes, compatibility and use-case copy help both humans and agents. Our note on how shoppers use ChatGPT to find products covers the same basics.
- Expect imperfect agent orders for a while: more than half of Visa's failed runs still ended in a purchase. Plan for returns and wrong-spec support when an agent thought it had the right product.
- Trust is still the bottleneck: only 23% of U.S. consumers trusted AI with payments in Visa's September Trust Index. Agent traffic can grow before payment trust does.
The technical details
Visa says this is the first agentic commerce benchmark that requires both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision. Models that uncovered more hidden buying factors averaged a 70.2% pass rate; those that uncovered the least passed 44.4%. Consistency was weak even after a model had already passed a scenario. Visa plans to extend the tests to checkout, payment and order placement.
Want to know whether AI agents can find and buy from your site? Run our free AI check.
