Second scorer: questions and answers

What does the second scorer do?

They independently score a random 30% of Round 1 runs, working from the evidence alone: recordings, screenshots, order emails and the written rules.

Who is it?

A second person who is not involved in running the tests.

Who picks which runs they score?

Nobody. Before testing, a fixed random seed and a public script, select_sample.py, decided the sample. The seed's SHA-256 fingerprint was published in advance and the seed is revealed with the results. How to check it.

How is the sample drawn?

Each run ID (for example t10-copilot-p2) is ranked by SHA-256(seed + "|" + run ID). The lowest-ranked 30%, rounded up, are scored by the second scorer. That pool includes runs scored "could not complete".

What happens when the scorers disagree?

We write a rule that settles the disagreement, add it to the method, and apply it to every run. The agreement rate is published.

Were any scoring rules fixed before testing?

Yes. Rules for T01 sock size ranges, T02 wattage, T08 soap and pump bottles, and listing-versus-tag conflicts were in the method before testing started.