Verify it yourself

You shouldn't have to trust us. Here are three checks you can run.

1. The method was public before testing

Every edit to the method is in the public commit history. Check the date of the first version, then read the dated changelog at the bottom of the method.

git clone https://github.com/agentmistakesindex/methodology.git
cd methodology
git log --reverse --format="%h %ad %s" --date=iso -- METHODOLOGY.md

Or open the commit history on GitHub.

2. The second-scorer sample couldn't be picked after the fact

Before testing, we published the SHA-256 fingerprint of a secret seed:

a7a94d19b5eff967112f8ad35cde13fc5a14dde6297a6af19dc463f4c7e8d94c

When results are out, we reveal the seed. Check that it matches:

printf '%s' 'THE-REVEALED-SEED' | sha256sum

Then regenerate the exact 30% sample from the published run IDs, using only Python 3:

python3 select_sample.py --seed 'THE-REVEALED-SEED' run_ids.txt

The script prints whether the seed matches the fingerprint, then lists the sample. Read the script; it's about 50 lines.

3. The redacted evidence matches untouched originals

Every redacted file is published next to the SHA-256 hash of its unredacted original. If an original is ever shown, for example to a company exercising its right of reply, anyone can confirm it is the same file:

sha256sum original-file.pdf

Found something that doesn't add up? Tell us. Confirmed errors are fixed publicly within 48 hours and logged in the changelog.