Ezra | EdenFi

AI agents failed 42.8% of Visa's shopping tests In more than half of those failures, the agent bought something anyway 😳 Which is striking when you consider that six in ten UK adults told YouGov they'd stop using an AI shopping agent after one mistake. So we needed a benchmark to see where its failing. Cue Visa... --- Visa launched its Agentic Commerce Benchmark on Thursday: - Testing 22 AI models from 12 labs, 15 shopping tasks, three runs each - 57.2% of runs passed (best model 82.2%, worst 28.9%) - A model passed all three runs of a task only 34.2% of the time - 23.1% of all runs ended in a purchase that failed the test In 23.1% of Visa's runs, the agent made the wrong call and bought. The top two models tied at 82.2% and failed differently. Meta's Muse Wonder bought on 2 of its 8 failed runs, Google's Gemini 4 Argon on 5 of 8, and Claude Fable 5.1 on all 11 --- Their methodology is interesting They set up one AI to play the shopper and hide most of what it wants. The personal agent or model has to ask the right questions, search a simulated store, then buy or walk away. Visa stopped the test at the buy decision and says payment testing comes next. So this is entirely a test of how close they get to completing the payment, not what happens after you press the buy button. That is still, overall, very helpful, I imagine, especially to the labs who will want to train their models against this benchmark (Aka hill climbing) --- Agents fail at buying things for two reasons. 1. The agent can't do the job. It misses what you care about, or gets stuck on a messy checkout form or an edge case. (What Visa tested) 2. The payment stack blocks the agent. The merchant or bank might block an agent from paying at all. Because if it's not you typing your 16 digits, how would the merchant know its you and not a stolen card? A stronger model is more likely to fix the first problem. The second needs the merchant and your bank to know which agent is buying and who pays if it gets it wrong. --- We're very early. In Wednesday's Brainfood I laid out the four ways an agent pays today: 1. Your real card, typed in by a robot the merchant doesn't trust 2. A one-time virtual card, so your bank sees the platform instead of the shop 3. A network token (a stand-in for your card number) issued to the agent 4. A balance you loaded up front, often in stablecoins, so your bank never sees the agent The first two break most often. That leaves network tokens and stablecoin balances, and I think we get both. Banks are holding back on agent tokens until they agree who's liable, and that takes time. You can read the full piece here: fintechbrainfood.com/p/payme…