What AI shopping agents were asked, what they said, and what the evidence shows
The Agent Mistakes Index is a small, repeatable test of consumer AI shopping agents available in the United States. Each agent gets the same written shopping tasks, word for word. We compare the instructions, the agent's own summary, and the evidence: screen recordings, screenshots and order/cancellation emails.
- 6 makers, 7 scored surfaces: Muse (Meta), Amazon Alexa for Shopping, Google (Gemini app and AI Mode in Search, scored separately), Microsoft Copilot, Perplexity, and ChatGPT (confirmation screen only).
- 8 tasks with rules anyone can check, such as size, quantity, a total cap, no substitutes, and no subscription.
- Real orders on 3 tasks; the others stop at the final confirmation screen and are labelled "would have bought".
- Method fixed in advance and public before the first test. Every later change is dated in the changelog.
- A second scorer independently re-scores a random 30% of runs. The sample was locked in before testing.
Status
Round 1 testing is complete apart from the price-watch task, which runs to 11:59 PM ET on October 28, 2026. Scoring comes next. Results are expected in mid-November 2026, published with the run log and redacted evidence so anyone can re-score them.
We don't name a "best" agent. With about eight tasks per agent, small differences don't mean much.