Critiques we expect, and our answers

"Eight tasks per agent is far too few."

Agreed, it's small. We report what happened on these dates, not how often it would happen in general, and we won't name a winner unless a gap is large and consistent. In exchange, every run is fully documented, so you can see each case rather than rely on a percentage.

"One tester means bias."

That's why the prompts, scripted answers, scoring rules and weights were published before the first test. A second person independently re-scores a random 30% of runs, the agreement rate is published, and redacted evidence is public, with hashes of the originals, so anyone can re-score.

"You cherry-picked runs."

Each agent gets each task once. A task is re-run only after a technical failure before a confirmation screen, and that attempt is logged. Trial runs are disclosed and kept out of the data. Correct results are reported as prominently as mistakes.

"The agents shopped at different stores, so it isn't fair."

We test each agent the way a consumer would use it, including where it chooses to shop. That's stated plainly: this is not a same-store comparison.

"ChatGPT only went to the confirmation screen."

Yes, and its results are labelled "would have bought". The method also discloses that the ChatGPT plan used in earlier tasks lapsed on October 22, 2026, so its remaining runs are reported as "not run" rather than repeated on a different setup.

"You changed the method during the test."

Every change has a dated entry in the changelog with its reason and whether any affected test had been run. The commit history shows each edit. Tasks, prompts, scoring and weights have not changed.

"The weights are arbitrary."

They are a choice, and we say so. We publish raw counts of S1–S4 so you can use your own weights.

"Why should anyone trust an anonymous tester?"

Nothing in the results depends on who ran them. The method was fixed in advance, redacted evidence is public with hashes of the originals, the second-scorer sample is locked by a published fingerprint, and companies get 7 days to dispute anything before publication.

"Who pays you?"

Nobody. The test is self-funded. We accept no money, credits, affiliate commissions or review units from agent makers, retailers, merchants or payment companies. Links aren't affiliate links.

Have a critique that isn't here? Email it or open an issue.