|
and Scott's overnight-SPRT shortcut (test_vs_head.sh); gitignore generated caches/PGN output.
tune_eval_dna.py isn't Texel-tuning-specific tooling anymore -- it's a
load-bearing dependency (match_play.py does `from tune_eval_dna import
Engine`), so it needs to be tracked for match_play.py to run at all on
a fresh checkout, independent of whatever happens to the rest of the
auto-tuning pipeline. filter_pgn.py (builds twic_filtered.pgn, the
pool match_play.py's --pgn points at) is similarly not tuning-specific.
test_vs_head.sh is Scott's shortcut for the overnight SPRT run
discussed this session (head_reference/typhoon vs. current build,
--games 20000 --workers 20 --st 1 --sprt --elo0 0 --elo1 5).
Left untracked, Texel-pipeline-specific and matching this session's
move away from auto-tuning: bake_dna.py, dna_diff.py, dna_trend.py,
cycle.sh, run_ecm.sh, compare_ecm_*.py, tuned.dna.
.gitignore: eval_tune/__pycache__/ and eval_tune/opening_cache/ (pure
regeneratable caches) and src/{match_games,self_play_games}.pgn
(match_play.py's --pgn-out game logs, generated output not source) --
the dna/first.dna / dna/original_baseline.dna accidental-commit from
earlier tonight was exactly this class of mistake, catching the
obvious repeat cases now.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01M9ZDiJhiUajUxh95mTXCFJ
|
|
EBF/NPS/first-move-beta stats, fix a completion-order bias in the live progress readout.
match_play.py previously compared one fixed binary with two loaded
evaldna files -- built for the Texel auto-tuning pipeline this project
has since moved away from in favor of hand-tuned constants baked
directly into eval.c. Converted to compare two separate compiled
binaries instead (head_engine/candidate_engine positional args, no
DNA loading at all) -- a real build is now required per side, but
that's the right model: what's being compared is two source trees,
not two parameter files loaded into an otherwise-identical process.
Found and deleted eval_tune/binary_match_play.py, a pre-existing (never
committed) sibling that already did binary-vs-binary comparison but
predates and is now strictly superseded by this rewrite (no SPRT, no
stats capture, and the same completion-order bias fixed below) --
keeping both would have left two overlapping tools to drift out of
sync, the same duplication problem this session spent all night
removing from eval.c itself.
Added a real SPRT (Sequential Probability Ratio Test), the same
formulation fishtest/cutechess-cli use: two Elo hypotheses (H0/H1)
tested via the log-likelihood ratio of a normal approximation to the
per-game trinomial (W/D/L) score, variance re-estimated from the
running W/D/L mix as games accumulate. Verified against a Monte Carlo
simulation before landing: correctly resolves H0 at true_elo=0 (~6-19k
games) and H1 at game counts matching the theoretical fixed-N table
almost exactly (30 Elo: ~1.4-2k, 20 Elo: ~2.3-4.5k, 10 Elo: ~5-9k, 5
Elo: ~10-25k). Required restructuring the game scheduler from
"submit everything upfront, as_completed" to a bounded rolling window
(at most --workers games in flight) so it can actually stop early
once SPRT concludes instead of having thousands of already-launched
futures it can't usefully cancel.
Fixed a real, previously-unnoticed bug that explains a specific
observed symptom (candidate consistently scoring high for the first
~500 games of a 1000-game run, then eroding toward 0.5 -- every run,
same direction, which is what tipped this off as systematic rather
than noise): the live "score so far" readout processed games in
*completion* order (as_completed), not submission order, and decisive
games plausibly finish faster than grindy draws/losses (a winning side
wraps up before --max-plies; a losing/drawing side often runs long).
That means candidate wins arrive disproportionately early and the live
average was a biased mid-run estimator -- high at first, eroding as
slower non-win games trickle in. The *final* score was never actually
wrong (order-independent, sums the same regardless of arrival order),
just the progress narrative watched live. Fixed by buffering
out-of-order completions and only advancing the printed running score
through games in their original submission order.
Also fixed --sd defaulting to 8 even when --st was passed -- the
argparse mutually-exclusive group only stops both flags being given
together, it does nothing about one flag's default silently applying
when only the other was specified. --st alone was being silently
ignored in favor of sd=8 the whole time. Now --sd defaults to None and
only falls back to 8 when neither --sd nor --st is given.
Added EngineStats: per-engine (not per-color, since candidate/baseline
swap sides every other game) speed and tree-shape capture -- NPS, EBF
(nodes**(1/depth) per move, same formula script.c's suite runs use),
and first-move-beta-cutoff-rate, all pulled from the same
PostMoveSearchReport block every move already prints (confirmed this
prints after EVERY move, not just script-run suite summaries, and
confirmed the "move" line prints BEFORE the stats block against
root.c, not after -- the read loop needed restructuring to keep
reading past the move line rather than stopping on it). No separate
benchmark pass needed; these come from the same games already being
played for the strength comparison.
Deliberately not built tonight, flagged as a possible follow-up: an
opening-pool independence concern (sample_openings picks random byte
offsets into the TWIC pool, which could pull duplicate/correlated
lines if TWIC has many games following the same trendy opening in a
season -- would make the standard-error math slightly overconfident,
not wrong in direction).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01M9ZDiJhiUajUxh95mTXCFJ
|
