Document the three-curated-suites testing methodology and the

head_reference protocol.

Realized while asking "where is this documented" that it wasn't --
scattered implicitly across lmr_testing/RESULTS.md and
head_reference/README.md but never stated as a methodology anywhere
durable. Also records the sd-vs-sn guidance (pick based on what the
change is expected to affect) and flags the specific gap that cost
real time this session: assuming a working tree's disabled-mechanism
state matched HEAD's actual committed behavior, when it didn't.

1 files changed, 47 insertions, 0 deletions

diff --git a/src/CLAUDE.md b/src/CLAUDE.md
index 881b062..e286cd9 100644
--- a/src/CLAUDE.md
+++ b/src/CLAUDE.md
@@ -78,6 +78,53 @@ lag well behind actual progress while a run is in flight; it catches up fully
 once the process exits. Don't trust a logfile's line count as a live
 progress indicator for a running batch job.
 
+### The three curated suites and the `head_reference` protocol
+
+Day-to-day search-change evaluation uses three small, curated EPD files
+under `tests/` rather than the full 881-problem `ecm.ep_` (too slow to
+iterate on) or a single random sample (too noisy -- a handful of flips
+either way swamps the signal at that size):
+
+- **`tests/ecm_ringers.ep_`** (11 positions) -- clean discriminators
+  hand-picked because a baseline engine solves them reliably and a
+  regressed one reliably doesn't (or vice versa). Fastest signal, but
+  small enough that +/-1 solve is not necessarily real.
+- **`tests/ecm_confident_quick.ep_`** (90 positions) -- a larger,
+  still-fast sample the engine is expected to do well on.
+- **`tests/ecm_hard_quick.ep_`** (90 positions) -- a harder sample,
+  more sensitive to search-quality changes (pruning/reduction/ordering)
+  since these positions need real depth/precision to crack.
+
+Run all three together, not just one -- a change can look great on one
+and cost solves on another (see `lmr_testing/RESULTS.md` for several
+examples this cut both ways).
+
+**Always compare against a fixed, known-good reference binary+logs, one
+variable at a time -- never against "the working tree as it happened to
+be a few hours ago."** `head_reference/` (sibling of `src/`, not
+`tests/`) holds exactly this: a binary built from a specific commit,
+plus full logs for all three suites at both `sd 10` (fixed depth) and
+`sn 5000000` (fixed node budget per position) in `head_reference/logs/`,
+and a `README.md` recording the commit hash and confirmed live/dead
+state of every search mechanism (LMR, EFP, etc.) at that point --
+confirmed by reading the code, not assumed from memory of what a
+diagnostic build's `if (FALSE)` happened to say. Rebuild this directory
+(and update its README) whenever committing a change that becomes the
+new comparison baseline, so the next investigation always has
+something to check itself against before drawing conclusions --
+this exact gap (assuming a disabled-in-the-working-tree mechanism was
+also disabled at HEAD, when it wasn't) cost a full afternoon
+re-deriving EFP's actual committed behavior mid-investigation.
+
+Pick `sd` vs `sn` based on what the change is expected to affect: `sd`
+(fixed depth) is the right choice when judging whether a pruning/
+ordering change makes the tree smaller or larger for the same search
+effort; `sn` (fixed node budget) is the right choice when judging how
+many positions solve within a fixed cost. Don't run both for every
+single candidate change -- pick whichever matches what's actually being
+tested (the reference logs in `head_reference/` keep both, precisely so
+a later investigation can choose either without re-running HEAD).
+
 ## Eval tuning pipeline (`eval_tune/`)
 
 `tune_eval_dna.py` is a Texel-style tuner: drives `typhoon` as an