diff options
| author | Scott Gasch <[email protected]> | 2026-09-13 23:18:07 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-09-13 23:18:07 -0700 |
| commit | 4ff9170fd96e3da049c747bf5215e908a94ff145 (patch) | |
| tree | 6c8e6f846b42518d84357bb3b83014feddc83202 /tests/ECM.md | |
| parent | 78d1889fe38db1fd45b58509c8b3bd535ce691b7 (diff) | |
Add king-safety mate-mining regression pipeline; retune file-storm/
queen-proximity king-safety weights and add a back-flight bonus from it.
Mines real human-game checkmates and matched sound-draw controls
(eval_tune/mine_king_mates.py, shard_mine.py, extract_king_features.py,
match_controls.py, train_king_safety_model.py), then fits a logistic
regression against _EvalKing's raw per-defect breakdown (engine-side
support: chess.h's KING_DEFECT_DUMP_INFO, eval.c's KING_DEFECT_DUMP
instrumentation, command.c's evalkingdump command, all gated behind the
KING_DEFECT_DUMP=1 build flag) to see which defects actually predict
getting mated. See eval_tune/king_safety_data/HANDOFF.md for the full
methodology and findings.
Two rounds of retuning landed from this data so far, both validated via
sd10 curated-suite comparison plus per-position Crafty corroboration
(tests/ECM.md) and gated self-play:
- KingFileDefects/KingStormingPawnDefects (_ComputeFileStormDefects)
bumped ~1.75-2x relative to the rest of the ring-scan defects -- the
regression's most dominant, most stable tier-1 predictors, previously
weighted the same as every other hand-picked defect.
- KING_QUEEN_PROXIMITY_DANGER rescaled from -8/count to -32/count, and
a new KING_BACK_FLIGHT_BONUS (+1/square) term added for empty,
unattacked ring squares on the side of the king away from the
attacker -- a real, non-collinear protective signal the current
EmptyUnattackedSquare table didn't credit at all. Both calibrated by
unstandardizing the regression's coefficients and anchoring against
KING_MISSING_ONE_CASTLE_OPTION's already-live -23 value (same
direct-EVAL_TERM unit family).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P9EDZVcqu6E2bNRm4zDrTf
Diffstat (limited to 'tests/ECM.md')
| -rw-r--r-- | tests/ECM.md | 246 |
1 files changed, 246 insertions, 0 deletions
diff --git a/tests/ECM.md b/tests/ECM.md new file mode 100644 index 0000000..5ff1ed0 --- /dev/null +++ b/tests/ECM.md @@ -0,0 +1,246 @@ +# ECM position notes + +Per-position metadata accumulated while investigating solve-rate losses on +the curated suites (`ecm_ringers.ep_`, `ecm_confident_quick.ep_`, +`ecm_hard_quick.ep_`) during the 2026-09-10/11 extension-taper/qsearch-prune +session. IDs match the `id "ECM.###"` labels `script`/`suite_diff.pl` print +(note: due to how the `.ep_` files pipeline `id`/`go`/`setboard`/`solution` +-- each `id` line is followed by the *previous* block's `setboard`/ +`solution`, not the one textually after it -- so don't assume `grep`ing the +raw `.ep_` file for an id gives you the matching FEN without checking this). + +## Methodology + +A position's HEAD-vs-candidate solve flip was investigated by: + +1. Reading HEAD's own `sd10` log for that position: does the PV settle on + one move with a stable or improving score across iterations, ideally + with a `++`/`+` confirmed-fail-high mark? Or does it flip between + different candidate moves, land on an unconfirmed score at the very + last iteration (especially if it abandons an already-*confirmed* + alternative), or return a score suspiciously close to `0.00`/negative + for the side that's supposedly winning? +2. Where available, an independent second opinion: Crafty's own search + (`sd 15`-`sd 20`) on the same position, and/or typhoon-candidate's own + result at a deeper fixed depth (`sd 15`) than the suite's `sd 10`. +3. A position only counts as **legit** if HEAD's own read was solid *and* + nothing independent contradicts it. A single wobble or disagreement is + enough to call it **flaky**. + +This is a real, if informal, signal -- not a substitute for actually +re-running the suites when a change lands, just a way to avoid re-deriving +"is this specific solve-count delta noise" from scratch every time a +similar-shaped loss reappears (which it does -- most of the positions below +have shown up as a flip across *multiple*, otherwise-unrelated changes this +session). + +## Legit (real tactic, HEAD understood it solidly -- a genuine loss if lost) + +| ID | Suite | FEN | Solution | Why legit | +|---|---|---|---|---| +| ECM.035 | confident_quick | `2r2bk1/4qp2/3n2p1/2R1p1Np/2p1N3/r6P/1Q3PP1/3R2K1 w - -` | Rxc8 | HEAD confirms (`++`, +2.31) before finalizing at +2.79; Crafty's independent deep search converges to ~+4.6; candidate's own `sd 15` also confirms | +| ECM.080 | confident_quick | `5r1k/pb2r1bp/1p2B1p1/n7/4qNQ1/4B3/P4PPP/2RR2K1 w - -` | Rd4 | Single clean line, +3.26, fast solve, no oscillation | +| ECM.128 | hard_quick | `rn2qbr1/2p4k/p2p1nb1/1p1Pp2p/2P4P/2NBBPN1/PP1Q4/2KR2R1 w - -` | Nxh5 | Single clean line, +2.66, no oscillation | +| ECM.071 | hard_quick | `b1n1r1k1/2q2p1p/p4Pp1/1pBP4/4B1P1/2P1Q3/P4K1P/4R3 w - - 0 0` | Qh6 | Confirmed (`++`, +3.34) at depth 9, climbs monotonically to +4.74 by depth 12, same move throughout. Crafty independently finds the position winning even bigger (+7-9) but via a different move (`Kg2`, a quiet consolidation) -- doesn't contradict `Qh6`'s own soundness, just suggests it isn't the single objectively-best try. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.250 | confident_quick | `2r1rbk1/5pp1/bq5p/1pnBPN2/6Q1/N7/5PPP/R2R2K1 w - -` | Bxf7+ | Single clean line, +3.31, fast solve | +| ECM.251 | hard_quick | `1k1r1r2/2b2R2/Bppp2p1/2q4p/Q2pP3/P2P4/2P3PP/1R4K1 w - -` | Bb7 | Instant solve; Crafty's own deep search independently plays Bb7 too (+1.87) | +| ECM.445 | confident_quick | `7r/2p1q1k1/1p3r1p/p1pPpPp1/2P1P1P1/PP6/3Q2KR/7R w - -` | Qxg5+ | HEAD never solves this at `sd10` at all (only flips under `sn5m`); Crafty confirms the move is objectively good and stable (+2.31 to +2.35 across depths 18-20) | +| ECM.519 | confident_quick | `1k1r2r1/p1p3q1/2p2p2/3pn3/1P1b2bB/1QN5/P4PPP/1R2RBK1 b - -` | Nf3+ | Huge, unambiguous score (+7.33), no wavering at all; candidate's own deeper search confirms and strengthens it | +| ECM.147 | confident_quick | `r2b1rk1/5qp1/2p2P2/2p3Pp/p6P/2Q5/PPP5/2KR2R1 w - - 0 0` | g6 | HEAD's own line switches ideas mid-search (Qxc5 -> g6) but the new idea gets its own `++` confirmation (depth 10, +0.75) before the final unconfirmed bump to +0.89; Crafty independently plays the same `g6` and stably converges to +0.49-0.66 across depths 19-22. Thin margin, but move-choice agreement from an independent engine is the deciding evidence -- lost by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.672 | confident_quick | `5rk1/pbrn1ppp/1ppN1q2/2P1p3/3P4/1PRB4/P3QPPP/5RK1 w - - 0 0` | Bxh7+ | HEAD confirms (`++` twice, +4.09/+5.59) with the same first move stable from depth 7 on, score climbing monotonically to +6.22 at depth 10; Crafty independently plays the same move, reaching +9.5 by depth 27. Lost by the candidate LMR-graded-curve build (2026-09-11) -- notably, the *same candidate binary* run to deeper iterative depths (not suite-capped `sd10`) does eventually find `Bxh7+` too, jumping from +2.46 (depth 12, still on a `cxb6`-based line) to +7.59/+7.90 at depths 15-16. Not a permanent blind spot: the graded curve over-reduces this specific sacrifice line early enough to delay discovery by ~7-8 plies past what `sd10` allows, not miss it outright | +| ECM.721 | ringers | `4rrk1/2pn2pb/p1p1qp2/1pb1pN2/P3P1PN/1P1P4/1BP2PK1/R2Q3R w - -` | Nxg7 | Clean, unwavering, +3.19; candidate confirms/strengthens at deeper depth (+4.29) | +| ECM.733 | confident_quick | `2r3k1/pbq1np1p/1p1b1Bp1/8/6QP/2P2N2/B4PP1/4R1K1 w - -` | Bxf7+ | HEAD confirms once (`++`, +1.52); same first move stable across all entries, only the deeper continuation changes -- stronger signal than a plain unconfirmed line. Reconfirmed by Crafty (stable `Bxf7+`, +4.2-4.4, depths 18-22) during the 2026-09-11 LMR graded-curve investigation, which also lost this position | +| ECM.736 | confident_quick | `r2r2k1/1b2qpp1/1p2n3/p1b1p2p/4P1n1/PP1B2PP/1B1NQP2/R3NRK1 b - - 0 0` | Nxf2 | HEAD abandons a confirmed `Nd4` (`++`, +1.51) for `Nxf2`, unconfirmed and briefly dipping (+2.05 -> +1.93) -- looked flaky from HEAD's log alone. Crafty independently and stably plays the same `Nxf2` across 5 straight iterations (depths 24-28, -1.43 to -1.90 from Black's perspective), confirming the move choice despite HEAD's own wobble. Lost by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.734 | hard_quick | `r3r1k1/1bqn1ppp/1pp2p2/8/3P4/1B4N1/PP3PPP/R2QR1K1 w - -` | Bxf7+ | Same shape as ECM.733: confirmed once (`++`, +1.74), same top move stable throughout even as the deeper line's score settles lower | +| ECM.789 | hard_quick | `2r1r1k1/pb1n1pp1/1p1qpn1p/4N1B1/2PP4/3B4/P2Q1PPP/3RR1K1 w - - 0 0` | Bxh6 | Never gets an explicit `++` mark but the move is stable across the last 3 iterations with the score climbing (1.06 -> 1.27 -> 1.69). Crafty independently confirms the same move, also growing (1.89 -> 2.06 -> 2.13). Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.846 | hard_quick | `3qr1k1/rp3pbp/2p3p1/3p1b2/N1nP4/P2NP3/1PQ2PPP/1BRR2K1 b - -` | Rxe3 | **Reclassified from Flaky (2026-09-11)**: this entry previously read "two unconfirmed lines, both under +0.5" from an earlier build. The candidate LMR-graded-curve build, run to real (non-node-capped) depth, stabilizes firmly on `Rxe3` from depth 11 through 15 (+0.31 -> +0.67, no further wobble). Crafty independently and stably confirms the same move across depths 25-30 (-0.82 to -1.12, its own sign convention). Two-engine, deep, stable agreement overturns the old flaky call -- the earlier read was likely just a shallower/different build catching this position mid-wobble before it settles | +| ECM.735 | hard_quick | `r3r1k1/pp3ppp/8/1Nbp3q/3NnBn1/2P5/PPQ2PPP/3R1RK1 b - -` | Ngxf2 | HEAD's own line is clean and solid; Crafty's defense uses the *other* knight (Nexf2) but that's a "which piece" quibble, not a refutation of the idea | +| ECM.803 | hard_quick | `2b2r1k/2p3pp/2Nb4/pP5q/2PP4/P4pP1/5P1P/R1BQ2K1 b - -` | Bxg3 | HEAD's line is clean; Crafty independently finds **mate in 10** starting with Bxg3 -- about as solid as confirmation gets | +| ECM.442 | hard_quick | `4qn1k/1b3rpp/pb2pB2/1pp1P2P/3p1NQ1/P2P2PB/1PP4K/5R2 w - -` | Ng6+ | Not actually a miss -- the king-file-storm-defect candidate (2026-09-14) settles on `Bh4` at `sd10`, but its own deeper iterations (depth 14-15) find and `++`-confirm `Ng6+` (+1.94, climbing to +2.47 by depth 16). A depth-budget delay, not a blind spot; see the 2026-09-14 session note below | + +## Flaky / garbage (suite noise -- don't read a loss here as a real regression) + +| ID | Suite | FEN | "Solution" | Why flaky | +|---|---|---|---|---| +| ECM.213 | ringers | `k3r3/pR5p/PppR1p1p/4nP2/1PP5/8/4B1rP/2K5 w - -` | Re7 | Came off a draw-by-repetition line (`Kd1 ... <=>`) at only +0.37, unconfirmed | +| ECM.270 | ringers | `6k1/1p3p2/4r1p1/1PPrbq1p/2QN4/P2R1P1P/R4KP1/8 b - -` | Bg3+ | Score exactly `0.00`, single unconfirmed line; Crafty's own `sd20` search prefers Rxc5 instead | +| ECM.336 | hard_quick | `r1bqr1k1/pp3pp1/2p4p/3PN1n1/3P1b2/2NB4/PPQ2PPP/3R1RK1 b - -` | Rxe5 | Abandons a *twice*-confirmed cxd5 (swung -0.81 -> -0.16 -> +1.34) for an unconfirmed Rxe5 at the very end | +| ECM.337 | confident_quick | `r3r1k1/pbqn1ppp/1p3n2/1Pb5/2p5/5N1P/PBQ1BPP1/1N1R1RK1 b - -` | Rxe2 | Cycles through 3 different first moves within depth 10, lands unconfirmed | +| ECM.340 | confident_quick | `1k2r2r/1bpQN2p/1p6/P3p2q/4np2/2P2N2/2P2PPP/R3R1K1 b - -` | Rxe7 | Abandons a *thrice*-confirmed Qh6 (+2.49) for a near-identical unconfirmed Rxe7 (+2.76) | +| ECM.370 | hard_quick | `1r2r1k1/1pqb2pp/p2bp1n1/2p3BQ/3pN3/1P1P2P1/P1P3BP/4RRK1 w - -` | Bd8 | Score exactly `0.00` on a repetition line; Crafty prefers a different move at only +0.15 | +| ECM.443 | hard_quick | `2b5/2qrrpk1/5Rp1/2p4p/1pB1PR1P/1P1P2P1/5Q1K/8 w - -` | Qb2 | Two unconfirmed lines, thin margins (+0.92, +1.10) | +| ECM.637 | hard_quick | `r3qr1k/1b4pp/p2RQ3/Bp2p3/4p3/6P1/PPP2P1P/3R2K1 b - -` | Qh5 | Swapped a confirmed e3 (+2.18) for an unconfirmed Qh5 (+2.19) -- a coin-flip; Crafty's own eval also wobbled between -2.33 and -0.80 across nearby depths | +| ECM.715 | confident_quick | `3r1rk1/5pp1/pq1n1n1p/2pPR3/2B2P2/1PBQ2RP/P5P1/6K1 w - -` | Rxg7+ | Abandons a *twice*-confirmed Rge3 (+4.41) for a lower-scoring, unconfirmed Rxg7+ (+3.05) | +| ECM.771 | hard_quick | `3qr1k1/1br1bp1p/p3p1pB/1p1nN3/3PB3/7Q/PP3PPP/3RR1K1 w - -` | Nxf7 | HEAD never breaks +0.56, unconfirmed; Crafty's independent top choice is a different move near dead equal | +| ECM.813 | confident_quick | `r2q1rk1/pp2ppb1/3pn1pQ/3R4/2P3B1/4BR2/PP4PP/6K1 w - -` | Qxg6 | Both of HEAD's candidate lines score *negative* (-0.74, -0.69) -- HEAD's own eval doesn't think its own "solution" is good for the side supposedly winning | +| ECM.422 | hard_quick | `6k1/rnqb3p/5ppQ/2pPp3/p1N1P3/2PB3P/5PP1/1R4K1 w - - 0 0` | f4 | Same move (`f4`) held across 5 iterations but never gets a `++` confirm, thin margin (+0.70 to +0.95). Crafty independently prefers a different move (`Qh4`) at a similarly thin margin (+0.68 to +1.06) -- a genuine disagreement on a razor-thin position, not corroboration. Gained by the candidate LMR-graded-curve build (2026-09-11) but not a confident real solve | +| ECM.830 | confident_quick | `1n2r3/p1pq1kp1/1b1pNpp1/3P4/5RP1/3Q3P/1B3P2/6K1 w - -` | Bxf6 | Oscillates Nxg7/Bxf6 the whole way, no confirmation either side | +| ECM.842 | confident_quick | `r1b1k1nr/3n1p1p/1qpBp1p1/pp2b3/4PN2/PBN2Q2/1PP2PPP/2KR3R w kq - 0 0` | Nxe6 | HEAD abandons a confirmed `Bxe6` (`++`, +1.14) for an unconfirmed `Nxe6`, never reconfirmed. Crafty's own search at `sd 22` doesn't corroborate either -- wobbles between `Qg3` and `Bxe6` across depths 20-22, never playing `Nxe6`. Independent check does not support HEAD's answer, unlike ECM.147/672/736 below -- lost by the candidate LMR-graded-curve build (2026-09-11) but stays classified flaky | +| ECM.875 | hard_quick | `2kr1br1/ppp2p1p/2n1p1q1/2Nn4/3P4/1Q2PB2/PP1B1P1P/2R1K2R b K - 0 0` | Qg1+ | Score exactly `0.00` on a repetition line (`<=>`) across 6 straight iterations (depths 8-13) -- textbook flaky signature. Contradicts the "Gains seen" table's loose note calling this Crafty-confirmed legit in an earlier review; that note appears to have been wrong, or about a different search reaching a different (non-drawing) conclusion. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.879 | hard_quick | `r2r2k1/pp3ppp/2q5/2p5/2P1P1n1/P4P1b/1BQ1BP1P/R3R2K b - - 0 0` | Rd2 | Score exactly `0.00` on a repetition line (`<=>`) -- same signature as ECM.875. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.866 | hard_quick | `7R/1p1r4/2b2p2/2Pp1qk1/3P1bp1/3N4/2Q2PB1/6K1 w - -` | Rg8+ | HEAD's own score is negative (-0.12 to -0.15); Crafty's independent top choice (Qd1) is also near-zero. **Note**: has shown up as a *gain* (candidate solves it, HEAD doesn't) in at least one later run too -- consistent with a genuinely coin-flip position, not a stable read either direction | + +## Gains seen (candidate solves, HEAD doesn't) -- not deeply vetted + +These showed up as improvements across various runs. Only spot-checked, not +put through the full methodology above -- listed for reference, not as a +confirmed-legit set the way the "Legit" table above is. + +| ID | Suite | Notes | +|---|---|---| +| ECM.193 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.476 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.544 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.613 | hard_quick | Seen as a gain across multiple runs | +| ECM.716 | confident_quick | Seen as a gain across multiple runs | +| ECM.779 | hard_quick | Seen as a gain across multiple runs | +| ECM.822 | hard_quick | Seen as a gain across multiple runs | +| ECM.875 | hard_quick | Seen as a gain across multiple runs | +| ECM.489 | hard_quick | Seen once | + +## 2026-09-11 LMR graded-curve session + +A separate investigation replaced typhoon's flat `-ONE_PLY` LMR reduction +with a graded depth/move-number curve (fit to this engine's own measured +tail-fail-high-rate data, not a borrowed formula -- see `dynamic.c`'s +`InitLMRTable`), plus a QSearch-drop guard in `GetLMRReduction` (never +reduce far enough to land below `THREE_QUARTERS_PLY` remaining depth). +Measured against `head_reference` (`ddf932b`) on `confident_quick`: +`sd10` net -4/+2 (net -3 across all three suites combined), `sn5000000` +net -3/+3 depth-adjusted (net +4 combined, `hard_quick` alone +7 with ++0.5 extra ply -- a real depth-payoff signal). + +Re-investigated all 6 `confident_quick` losses (identical set at both +`sd10` and `sn5m` -- a reproducible effect, not scattered noise) and both +gains against `head_reference`'s own log plus a fresh independent Crafty +check (`sd 22`, not run through the full methodology before this +session). Net result: **4 of 6 losses are legit** (733, 672, 147, 736 -- +147 and 736 were previously unclassified and looked flaky from HEAD's +log alone, but Crafty's independent move-choice agreement overturned +that read), only 2 are flaky (715, 842). Both gains (519, 035) are +already-established legit. That's a real, non-noise net-negative +tactical cost on `confident_quick` specifically -- the aggregate +suite-level numbers understated this at first glance (`sd10` -4 raw +looked mostly explainable as noise until the per-position Crafty check +was actually run). + +ECM.672 is worth remembering for future LMR work specifically: the same +candidate binary, given enough iterative depth (not suite-capped +`sd10`), *does* eventually find the correct `Bxh7+` sacrifice (depth +15-16, jumping from +2.46 to +7.59/+7.90) after several iterations +spent on an unrelated `cxb6`-based line. The graded curve isn't blind to +this tactic, it's delaying discovery by roughly 7-8 plies past what a +depth-10 budget allows -- a genuine cost at fixed/shallow search depths, +but not the kind of permanent blind spot a bad exemption rule would +produce. Worth checking whether this delay is concentrated in +sacrifice/attacking lines specifically (where the "safe to reduce" +heuristics this curve is built from may be least trustworthy) before +concluding the curve needs a different shape rather than just deeper +search to pay off. + +**`hard_quick`'s `sn5000000` gains investigated too** (11 gains, 4 losses +-- 193, 476, 779, 866, not re-investigated here since they're losses on +the suite that was already net-positive). Of the 11 gains: **7 are legit** +(071, 251, 734, 735, 789, 803, 846 -- 846 is a reclassification, see its +updated Legit-table entry above: an earlier build's shallower read +tagged it flaky, but this candidate's deeper search plus Crafty both +stably converge on the same move), **3 are flaky** (422 -- genuine +engine-vs-engine disagreement on a thin margin, not corroboration; 875, +879 -- both exact `0.00` on a repetition line), and **1 is genuinely +unresolved** (640): the candidate's own extended search twice-confirms +`Rxb2` at +3.66 (the `sn5m`-capped run's PV/played-move mismatch was +just the node budget cutting off *before* this switch, not a bug), but +Crafty independently searches to depth 30 and never once prefers +`Rxb2`, staying on a different move (`Bd8`) near dead equal. Two +real-depth searches disagreeing after real depth is a different, more +interesting kind of uncertainty than the usual flaky signature (score +exactly 0, unconfirmed thin margins) -- worth a dedicated line-by-line +look at the `Rxb2` continuation before counting it either way. + +**Combined verdict across both suites, real (non-flaky) positions only**: +9 legit gains (035, 519, 071, 251, 734, 735, 789, 803, 846) vs. 4 legit +losses (733, 672, 147, 736), with ECM.640 unresolved. A materially more +favorable picture than the raw suite aggregates suggested on their own +-- this is why the per-position Crafty-corroboration check matters more +than trusting `sd10`/`sn5000000` solve counts at face value, especially +at this suite size. + +## 2026-09-14 king-safety file-storm-defect session + +Retuned `eval.c`'s `KingFileDefects`/`KingStormingPawnDefects` tables +(inside `_ComputeFileStormDefects`) per `eval_tune/king_safety_data/HANDOFF.md`'s +tier-1 finding: `{+2,+1,+0}` -> `{+4,+2,+0}` and +`{+0,+4,+3,+1,+0,+0,+0,+0}` -> `{+0,+7,+5,+2,+0,+0,+0,+0}`, a real-game +mate-mining logistic regression's two most dominant, most stable +predictors, previously weighted the same as every other hand-picked +king-safety defect. Measured against `head_reference` (`ddf932b`) at +`sd10`: `ecm_ringers` -2, `ecm_confident_quick` -1, `ecm_hard_quick` -1. + +Per-position investigation (same methodology as the LMR session above) +found almost none of this is real: +- `ECM.715`, `ECM.866` (losses) and `ECM.813`, `ECM.443`, `ECM.771` + (gains) were already tagged flaky above. +- `ECM.716`, `ECM.193`, `ECM.476` (losses) were already listed in the + "Gains seen" table from other sessions -- a position that has already + flip-flopped direction before isn't new evidence of a regression. +- `ECM.501`, `ECM.801` (ringers losses, previously unclassified): both + show HEAD *itself* wobbling between candidate moves and landing on its + final answer only at the last iteration with no `++` confirmation + (`ECM.501`: Rf7 -> Rc8 -> Rf7 -> Rb8 -> Nxf3; `ECM.801`: Nd4 held + stably for 3 iterations then an unconfirmed jump to Nh6+ only at + depth 10) -- textbook flaky signature, not a real loss either + direction. +- `ECM.442` (hard_quick loss): not a miss at all -- see its updated + entry in the Legit table above. The candidate's own deeper iterations + (depth 14-15) find and `++`-confirm the same `Ng6+` HEAD plays at + `sd10`; the `sd10` "loss" is purely a depth-budget delay (climbing + from +0.91 at depth 10 to a confirmed +1.94 by depth 14), the same + shape as `ECM.672`'s entry above. +- `ECM.298` (hard_quick loss, previously unclassified): HEAD converges + cleanly on `Re6`/`Rf6` (+2.7 to +2.9) and the suite's labeled solution + is `Re6`, but the candidate instead finds and `++`-confirms `h4` + (+5.49 by depth 11, climbing to +8.87 by depth 17) -- and Crafty's own + independent deep search (`sd 27`, ~9.3s to switch onto it after + spending depths 20-23 stuck on `Rf6` at only +2.9-3.2) agrees `h4` is + objectively much stronger (+10.17) than the suite's labeled answer. + Notably the candidate found this line in 2.5s vs. Crafty's 9.3s. + Not a regression -- if anything, evidence the candidate is finding a + better resource than the suite's own labeled solution, faster than an + independent reference engine. Worth relabeling this suite's expected + answer if it's ever revisited, rather than trusting `Re6` as ground + truth. + +**Net read**: of the 4 nominal net solve losses (-2/-1/-1 across the +three suites), 7 of 9 individual position losses were already-known or +freshly-confirmed noise, one (`ECM.442`) is a depth-budget artifact, and +the last (`ECM.298`) is arguably a candidate improvement mislabeled as a +loss by the suite's fixed expected answer. No surviving evidence of a +real tactical regression from this change at the position level, despite +the suite-aggregate solve counts looking mildly negative -- another data +point for not trusting raw `sd10` solve deltas on suites this size +without the per-position check. + +Self-play (`match_play.py`, ongoing) at 110 games: -29 =47 +34 +(score 0.523, llr=+0.11, still far from either SPRT bound) -- too early +to call, but notably `avg depth: candidate=9.96 head=10.25` (candidate +searching ~0.3 ply shallower at the same `st` budget) with +`avg ebf: candidate=4.442 head=4.396` (branching factor slightly +*higher*, i.e. the tree got slightly fatter, not narrower) -- a +non-trivial shape change in the search tree from what was purely an +eval-weight retune around king safety, presumably via move-ordering/ +pruning decisions that read king danger differently now (e.g. anything +gated on `KING_SAFETY_BY_COUNTER`'s bucket or a null-move/LMR +verification threshold sensitive to eval score swings). Worth keeping +in mind for future king-safety retunes: even a "just the numbers" +change to an eval sub-term can visibly shift average search depth, not +just leaf-node scores. + +## Caveat + +This list is a snapshot from one session's worth of investigation, not a +permanent classification -- a position flagged "flaky" here reflects HEAD's +behavior *at the commit this session started from* (`9d1e32d`) plus +whatever candidate builds were tested through it. If the eval or search +changes significantly later, re-derive rather than trust this table blindly +-- especially the "Gains" table, which was never run through the full +methodology in the first place. |
