diff options
Diffstat (limited to 'src/CLAUDE.md')
| -rw-r--r-- | src/CLAUDE.md | 101 |
1 files changed, 101 insertions, 0 deletions
diff --git a/src/CLAUDE.md b/src/CLAUDE.md index c7c7e4a..52a6877 100644 --- a/src/CLAUDE.md +++ b/src/CLAUDE.md @@ -317,6 +317,107 @@ bake-in-if-better, repeatable indefinitely against a growing game pool. large improvement it first looked like. Always diff ECM numbers taken at the same `sd` depth. +## Dynamic move ordering experiments (2026-08-29) + +A single long session that redesigned how quiet ("leftover", i.e. +sub-`GOOD_MOVE`) moves get ordered, replacing several hand-picked-and-never- +revisited heuristics with evidence-driven ones. Landed changes: retired +`NumLeftoverMovesToSelect` (see below), added a continuation-history table, +retired hung-piece-escape's unconditional tier promotion in favor of a +same-tier `FLEE_BONUS` nudge, and made the countermove table's tier +promotion evidence-gated instead of automatic. See `generate.c`'s +`_ScoreAllMoves` routine-description comment for the resulting move-ordering +hierarchy. The methodology below is the more durable takeaway -- reusable +for any future move-ordering question, not just the ones it already +answered. + +**Core technique: measure the *pick*, not the *game*.** Aggregate solve +counts on a 90-position suite are too noisy to tune a single move-ordering +knob against (a handful of positions flipping either way swamps the +signal) -- confirmed independently twice now, once by a prior session (see +`lmr_testing/RESULTS.md`'s "given-up leftover"/"would-prune surprise rate" +methodology) and once by this one. Instead, instrument the *local, per-move +outcome* (did trying this move raise alpha or fail high?) at a much larger +sample size, and read percentages off that instead of solve/pass counts. + +- **The "select nothing" / zero-selection-budget trick**: temporarily force + the leftover-selection budget to 0 (a compile-time constant swap + + rebuild, not a runtime flag -- keep this diagnostic-only, never commit + it) so that only the single "discovered we're in leftover territory" + transition move gets a full `SelectBestWithHistory` scan per node. That + scan still runs over the *entire* remaining pool regardless of budget, so + its result is genuinely "the best-of-remaining leftover, if we could only + afford to pick one" -- a clean, isolated read on whether the move-ordering + heuristic itself is any good, uncontaminated by how many picks the budget + allows. This is what first revealed that the countermove table's exact- + match signal was real (a forced-dominant version of it lifted the + best-leftover fail-high rate by about +1 percentage point, consistently, + across three curated suites) after a *scaled* continuation-history + version of the same signal showed a flat, uninformative response curve + across a 256x range -- the two experiments together showed scale wasn't + the missing variable, key resolution was (exact-move match vs. a coarser + [piece][to]-keyed proxy). +- **The contested-node A/B harness**, for "should class A rank above or + below class B" questions (e.g. countermove match vs. ply-2 killer): + aggregate per-class fail-high rates are confounded by *censoring* -- + whichever class is ranked higher gets tried first, so a lower-ranked + class's measured rate only ever reflects the subset of nodes where + nothing higher-ranked already resolved the position. Fix: at each node, + cheaply detect (piggybacked on the *first* `SelectBestWithHistory` call's + already-full-pool scan, no separate pass needed) whether *both* classes' + candidates are present as genuinely different moves; only log an outcome + on nodes where that's true, and only for the first candidate tried that + belongs to either class. **Caveat discovered by running this twice** (once + for countermove-vs-killer, once for hung-escape-vs-killer): whichever + class is *structurally disadvantaged* in a given test setting only wins + its rare contests via unusually strong self-reinforcing history/ + continuation evidence -- a self-selection effect that inflates the + disadvantaged class's apparent quality and deflates the favored class's, + independent of which is actually the better signal. Don't read a single + A/B setting's numbers at face value; compare each class's *least-filtered* + sample (the setting where it's favored) against the other's, not the + "loser" numbers from either individual run. +- **Evidence calibration**: once a structural pattern-match (countermove + table hit, en-prise escape) is identified, bucket every occurrence by its + own accumulated `g_HistoryCounters`+`g_ContinuationHistory` evidence + (log-ish bands: 0, 1-99, 100-999, ...) and plot fail-high rate per bucket. + This is the test for "is this pattern-match alone trustworthy, or does it + need a track record?" -- countermove matches with zero evidence scored + statistically identically to an ordinary unprivileged leftover (~0.6- + 0.85% FH) while evidence >=10,000 cleared 53-64%; hung-piece-escape's + zero-evidence population was 250-1000x larger than countermove's *and* + scored at the plain-leftover baseline too, revealing that the then- + shipping unconditional promotion was handing free tier-escape treatment + to a huge population that had done nothing to earn it. + +**Load-bearing gotcha, easy to reintroduce by accident**: a bonus added +inside `SelectBestWithHistory` (a selection-time-only nudge to the local +comparison value) never touches a move's persisted `iValue`, so it cannot +change `fIsLeftoverMove`/EFP eligibility no matter how large it is -- +useful for a same-tier nudge like `FLEE_BONUS`, useless if the goal is an +actual tier promotion. A real promotion (like the evidence-gated +countermove bonus) has to be written into `iValue` at *generation* time +(`generate.c`), not selection time. Confirmed by testing: an earlier +"dominant countermove" diagnostic added its huge bonus at selection time +and, despite clearly winning every internal comparison, never once actually +exempted a move from EFP or leftover classification -- the data it produced +was still valid (still genuinely measuring leftover-pool behavior) but the +mechanism didn't do what it looked like it should. + +**`NumLeftoverMovesToSelect` retirement**: the depth-indexed budget that +decided how many leftover moves got a full `SelectBestWithHistory` scan +before falling back to unsorted order was removed entirely once the +evidence above showed the leftover pool has real, findable signal a bailout +was discarding -- `search.c`'s main move loop now always fully selects. +Aggregate node-count deltas from this and related changes bounced around +by double-digit percentages on the smallest curated suite (`ecm_ringers`, +11 positions) with no corresponding solve change -- **on a suite this +small, a..b search's chaotic sensitivity to move order (a few more +depth-N-to-N+1 re-searches, a root fail-high or two) can move node counts +a lot for reasons unrelated to the change being tested. Fail-high +percentages and solve counts are the signal; raw node counts on small +suites mostly aren't.** + ## Environment notes - Shared, multi-user FreeBSD box with genuinely variable load (other Claude |