diff options
| -rw-r--r-- | src/CLAUDE.md | 47 |
1 files changed, 47 insertions, 0 deletions
diff --git a/src/CLAUDE.md b/src/CLAUDE.md index 881b062..e286cd9 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -78,6 +78,53 @@ lag well behind actual progress while a run is in flight; it catches up fully once the process exits. Don't trust a logfile's line count as a live progress indicator for a running batch job. +### The three curated suites and the `head_reference` protocol + +Day-to-day search-change evaluation uses three small, curated EPD files +under `tests/` rather than the full 881-problem `ecm.ep_` (too slow to +iterate on) or a single random sample (too noisy -- a handful of flips +either way swamps the signal at that size): + +- **`tests/ecm_ringers.ep_`** (11 positions) -- clean discriminators + hand-picked because a baseline engine solves them reliably and a + regressed one reliably doesn't (or vice versa). Fastest signal, but + small enough that +/-1 solve is not necessarily real. +- **`tests/ecm_confident_quick.ep_`** (90 positions) -- a larger, + still-fast sample the engine is expected to do well on. +- **`tests/ecm_hard_quick.ep_`** (90 positions) -- a harder sample, + more sensitive to search-quality changes (pruning/reduction/ordering) + since these positions need real depth/precision to crack. + +Run all three together, not just one -- a change can look great on one +and cost solves on another (see `lmr_testing/RESULTS.md` for several +examples this cut both ways). + +**Always compare against a fixed, known-good reference binary+logs, one +variable at a time -- never against "the working tree as it happened to +be a few hours ago."** `head_reference/` (sibling of `src/`, not +`tests/`) holds exactly this: a binary built from a specific commit, +plus full logs for all three suites at both `sd 10` (fixed depth) and +`sn 5000000` (fixed node budget per position) in `head_reference/logs/`, +and a `README.md` recording the commit hash and confirmed live/dead +state of every search mechanism (LMR, EFP, etc.) at that point -- +confirmed by reading the code, not assumed from memory of what a +diagnostic build's `if (FALSE)` happened to say. Rebuild this directory +(and update its README) whenever committing a change that becomes the +new comparison baseline, so the next investigation always has +something to check itself against before drawing conclusions -- +this exact gap (assuming a disabled-in-the-working-tree mechanism was +also disabled at HEAD, when it wasn't) cost a full afternoon +re-deriving EFP's actual committed behavior mid-investigation. + +Pick `sd` vs `sn` based on what the change is expected to affect: `sd` +(fixed depth) is the right choice when judging whether a pruning/ +ordering change makes the tree smaller or larger for the same search +effort; `sn` (fixed node budget) is the right choice when judging how +many positions solve within a fixed cost. Don't run both for every +single candidate change -- pick whichever matches what's actually being +tested (the reference logs in `head_reference/` keep both, precisely so +a later investigation can choose either without re-running HEAD). + ## Eval tuning pipeline (`eval_tune/`) `tune_eval_dna.py` is a Texel-style tuner: drives `typhoon` as an |