Rewrite match_play.py: binary-vs-binary comparison, real SPRT, EBF/NPS/first-move-beta stats, fix a completion-order bias in the live progress readout.

match_play.py previously compared one fixed binary with two loaded
evaldna files -- built for the Texel auto-tuning pipeline this project
has since moved away from in favor of hand-tuned constants baked
directly into eval.c. Converted to compare two separate compiled
binaries instead (head_engine/candidate_engine positional args, no
DNA loading at all) -- a real build is now required per side, but
that's the right model: what's being compared is two source trees,
not two parameter files loaded into an otherwise-identical process.

Found and deleted eval_tune/binary_match_play.py, a pre-existing (never
committed) sibling that already did binary-vs-binary comparison but
predates and is now strictly superseded by this rewrite (no SPRT, no
stats capture, and the same completion-order bias fixed below) --
keeping both would have left two overlapping tools to drift out of
sync, the same duplication problem this session spent all night
removing from eval.c itself.

Added a real SPRT (Sequential Probability Ratio Test), the same
formulation fishtest/cutechess-cli use: two Elo hypotheses (H0/H1)
tested via the log-likelihood ratio of a normal approximation to the
per-game trinomial (W/D/L) score, variance re-estimated from the
running W/D/L mix as games accumulate. Verified against a Monte Carlo
simulation before landing: correctly resolves H0 at true_elo=0 (~6-19k
games) and H1 at game counts matching the theoretical fixed-N table
almost exactly (30 Elo: ~1.4-2k, 20 Elo: ~2.3-4.5k, 10 Elo: ~5-9k, 5
Elo: ~10-25k). Required restructuring the game scheduler from
"submit everything upfront, as_completed" to a bounded rolling window
(at most --workers games in flight) so it can actually stop early
once SPRT concludes instead of having thousands of already-launched
futures it can't usefully cancel.

Fixed a real, previously-unnoticed bug that explains a specific
observed symptom (candidate consistently scoring high for the first
~500 games of a 1000-game run, then eroding toward 0.5 -- every run,
same direction, which is what tipped this off as systematic rather
than noise): the live "score so far" readout processed games in
*completion* order (as_completed), not submission order, and decisive
games plausibly finish faster than grindy draws/losses (a winning side
wraps up before --max-plies; a losing/drawing side often runs long).
That means candidate wins arrive disproportionately early and the live
average was a biased mid-run estimator -- high at first, eroding as
slower non-win games trickle in. The *final* score was never actually
wrong (order-independent, sums the same regardless of arrival order),
just the progress narrative watched live. Fixed by buffering
out-of-order completions and only advancing the printed running score
through games in their original submission order.

Also fixed --sd defaulting to 8 even when --st was passed -- the
argparse mutually-exclusive group only stops both flags being given
together, it does nothing about one flag's default silently applying
when only the other was specified. --st alone was being silently
ignored in favor of sd=8 the whole time. Now --sd defaults to None and
only falls back to 8 when neither --sd nor --st is given.

Added EngineStats: per-engine (not per-color, since candidate/baseline
swap sides every other game) speed and tree-shape capture -- NPS, EBF
(nodes**(1/depth) per move, same formula script.c's suite runs use),
and first-move-beta-cutoff-rate, all pulled from the same
PostMoveSearchReport block every move already prints (confirmed this
prints after EVERY move, not just script-run suite summaries, and
confirmed the "move" line prints BEFORE the stats block against
root.c, not after -- the read loop needed restructuring to keep
reading past the move line rather than stopping on it). No separate
benchmark pass needed; these come from the same games already being
played for the strength comparison.

Deliberately not built tonight, flagged as a possible follow-up: an
opening-pool independence concern (sample_openings picks random byte
offsets into the TWIC pool, which could pull duplicate/correlated
lines if TWIC has many games following the same trendy opening in a
season -- would make the standard-error math slightly overconfident,
not wrong in direction).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01M9ZDiJhiUajUxh95mTXCFJ

0 files changed, 0 insertions, 0 deletions