diff options
| author | Scott Gasch <[email protected]> | 2026-09-13 23:18:07 -0700 |
|---|---|---|
| committer | Scott Gasch <[email protected]> | 2026-09-13 23:18:07 -0700 |
| commit | 4ff9170fd96e3da049c747bf5215e908a94ff145 (patch) | |
| tree | 6c8e6f846b42518d84357bb3b83014feddc83202 | |
| parent | 78d1889fe38db1fd45b58509c8b3bd535ce691b7 (diff) | |
Add king-safety mate-mining regression pipeline; retune file-storm/
queen-proximity king-safety weights and add a back-flight bonus from it.
Mines real human-game checkmates and matched sound-draw controls
(eval_tune/mine_king_mates.py, shard_mine.py, extract_king_features.py,
match_controls.py, train_king_safety_model.py), then fits a logistic
regression against _EvalKing's raw per-defect breakdown (engine-side
support: chess.h's KING_DEFECT_DUMP_INFO, eval.c's KING_DEFECT_DUMP
instrumentation, command.c's evalkingdump command, all gated behind the
KING_DEFECT_DUMP=1 build flag) to see which defects actually predict
getting mated. See eval_tune/king_safety_data/HANDOFF.md for the full
methodology and findings.
Two rounds of retuning landed from this data so far, both validated via
sd10 curated-suite comparison plus per-position Crafty corroboration
(tests/ECM.md) and gated self-play:
- KingFileDefects/KingStormingPawnDefects (_ComputeFileStormDefects)
bumped ~1.75-2x relative to the rest of the ring-scan defects -- the
regression's most dominant, most stable tier-1 predictors, previously
weighted the same as every other hand-picked defect.
- KING_QUEEN_PROXIMITY_DANGER rescaled from -8/count to -32/count, and
a new KING_BACK_FLIGHT_BONUS (+1/square) term added for empty,
unattacked ring squares on the side of the king away from the
attacker -- a real, non-collinear protective signal the current
EmptyUnattackedSquare table didn't credit at all. Both calibrated by
unstandardizing the regression's coefficients and anchoring against
KING_MISSING_ONE_CASTLE_OPTION's already-live -23 value (same
direct-EVAL_TERM unit family).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01P9EDZVcqu6E2bNRm4zDrTf
| -rw-r--r-- | src/GNUmakefile | 3 | ||||
| -rwxr-xr-x | src/chess.h | 67 | ||||
| -rw-r--r-- | src/command.c | 109 | ||||
| -rw-r--r-- | src/eval.c | 227 | ||||
| -rw-r--r-- | src/eval_tune/extract_king_features.py | 186 | ||||
| -rw-r--r-- | src/eval_tune/king_safety_data/HANDOFF.md | 254 | ||||
| -rw-r--r-- | src/eval_tune/match_controls.py | 168 | ||||
| -rw-r--r-- | src/eval_tune/mine_king_mates.py | 315 | ||||
| -rw-r--r-- | src/eval_tune/shard_mine.py | 110 | ||||
| -rwxr-xr-x | src/eval_tune/test_vs_head.sh | 26 | ||||
| -rw-r--r-- | src/eval_tune/train_king_safety_model.py | 189 | ||||
| -rw-r--r-- | tests/ECM.md | 246 |
12 files changed, 1885 insertions, 15 deletions
diff --git a/src/GNUmakefile b/src/GNUmakefile index e3ae855..3613e40 100644 --- a/src/GNUmakefile +++ b/src/GNUmakefile @@ -147,6 +147,9 @@ endif ifdef CALIBRATE_POSITIONAL PROFILE += -DCALIBRATE_POSITIONAL endif +ifdef KING_DEFECT_DUMP +PROFILE += -DKING_DEFECT_DUMP +endif ifdef CALIBRATE_BASE_MARGIN PROFILE += -DCALIBRATE_BASE_MARGIN endif diff --git a/src/chess.h b/src/chess.h index 2681701..df59626 100755 --- a/src/chess.h +++ b/src/chess.h @@ -1367,6 +1367,70 @@ typedef struct _KING_SAFETY_HASH_ENTRY } KING_SAFETY_HASH_ENTRY; +#ifdef KING_DEFECT_DUMP +// Ring-slot indexing matches KingSafetyDeltas' order in eval.c exactly +// (-17,-16,-15,-1,+1,+15,+16,+17): slots 0-7, no bucketing -- kept at +// full per-square resolution rather than collapsed into front/side/back, +// since the direct in-front square (delta +-16) and the two front- +// diagonal squares (+-15/+-17) are genuinely different attack geometries +// (e.g. a same-file rook vs. a diagonal bishop) that a bucket would +// conflate. Sample size (tens of thousands of mined positions) has no +// trouble supporting this many features. +#define KING_RING_NUM_SLOTS 8 + +#define KING_RING_PAWN 0 +#define KING_RING_MINOR 1 +#define KING_RING_ROOK 2 +#define KING_RING_QUEEN 3 +#define KING_RING_NUM_PIECE_TYPES 4 + +// Raw, per-defect, per-square breakdown of _EvalKing's ring-square scan, +// filled in (only under this build flag) instead of/alongside the normal +// collapse into one scalar uCounter -- feeds a per-defect regression +// against real mate outcomes instead of trusting the hand-picked weights +// already baked into KING_SAFETY_BY_COUNTER / KING_COUNTER_BY_ATTACK_PATTERN. +// See CLAUDE.md's king-safety discussion. +typedef struct _KING_DEFECT_DUMP_INFO +{ + // [slot][piece type]: 1 if this ring square is attacked or x-rayed by + // this piece type, same bits _EvalKing's own bvAttack/bvXray already + // compute per square -- just retained per-slot instead of OR'd + // together into one whole-ring bvPattern. + FLAG fAttackedByPieceType[KING_RING_NUM_SLOTS][KING_RING_NUM_PIECE_TYPES]; + // [slot]: off-board squares (near an edge/corner) leave all four FALSE. + FLAG fOnBoard[KING_RING_NUM_SLOTS]; + FLAG fAttackedUndefended[KING_RING_NUM_SLOTS]; + FLAG fAttackedDefended[KING_RING_NUM_SLOTS]; + FLAG fXrayedUnattacked[KING_RING_NUM_SLOTS]; + FLAG fEmptyFlight[KING_RING_NUM_SLOTS]; + ULONG uFileStormDefects; + // Split-out sub-components of uFileStormDefects, from two already- + // separate hand-picked tables (KingFileDefects / KingStormingPawnDefects) + // -- kept apart so a regression can weigh "open/half-open file near + // the king" independently from "how far an enemy pawn has advanced up + // that file", rather than only ever seeing their pre-summed total. + ULONG uFileStormOpenFile; + ULONG uFileStormStormingPawn; + ULONG uQueenNearKing; + ULONG uInitialCounterByLocation; + ULONG uCounterTotal; // final uCounter, pre-clamp + ULONG uSafetyTableValue; // KING_SAFETY_BY_COUNTER[MINU(uCounterTotal,41)] + // Attacker's (xColor's) army strength -- pos->uArmyScaler[xColor], + // the exact index REDUCED_MATERIAL_DOWN_SCALER itself uses to scale + // the final king-safety score. Captured so training can control for + // it directly (a defect pattern under a weak attacker converts to + // mate far less often than the identical pattern under a full + // attacking army, regardless of what the raw defect counts say) -- + // see CLAUDE.md's king-safety discussion. + ULONG uAttackerArmyScaler; + FLAG fCastled; + ULONG uMinorsAtHome; + FLAG fMissingCastleShort; + FLAG fMissingCastleLong; +} +KING_DEFECT_DUMP_INFO; +#endif + #define NUM_SPLIT_PTRS_IN_CONTEXT (8) // @@ -1450,6 +1514,9 @@ typedef struct _SEARCHER_THREAD_CONTEXT // rejected), so there's no risk of runaway nested diagnosis. FLAG fDiagUnprunedSubtree; #endif +#ifdef KING_DEFECT_DUMP + KING_DEFECT_DUMP_INFO sKingDefectDump[2]; // filled by _EvalKing, indexed by color +#endif CHAR szLastPV[SMALL_STRING_LEN_CHAR]; } SEARCHER_THREAD_CONTEXT; diff --git a/src/command.c b/src/command.c index e6bf9fc..1696690 100644 --- a/src/command.c +++ b/src/command.c @@ -547,6 +547,107 @@ Return value: } +#ifdef KING_DEFECT_DUMP +COMMAND(EvalKingDumpCommand) +/** + +Routine description: + + This function implements the 'evalkingdump' engine command, only + built under KING_DEFECT_DUMP. Runs a full static eval (same + persistent-context pattern as EvalCommand) then prints one + space-separated key=value line per side with _EvalKing's raw, + per-defect breakdown (chess.h's KING_DEFECT_DUMP_INFO) instead of + the single collapsed score -- feeds an offline regression against + real mate outcomes. See CLAUDE.md's king-safety discussion. + + Usage: + + evalkingdump + +Parameters: + + The COMMAND macro hides four arguments from the input parser: + + CHAR *szInput : the full line of input + ULONG argc : number of argument chunks + CHAR *argv[] : array of ptrs to each argument chunk + POSITION *pos : a POSITION pointer to operate on + +Return value: + + void + +**/ +{ + static SEARCHER_THREAD_CONTEXT *ctx = NULL; + static ULONG uLastSeenDnaGeneration = (ULONG)-1; + static const CHAR *szPieceName[KING_RING_NUM_PIECE_TYPES] = + { "pawn", "minor", "rook", "queen" }; + ULONG uColor; + ULONG uSlot, uPieceType; + KING_DEFECT_DUMP_INFO *p; + + if (NULL == ctx) + { + ctx = SystemAllocateMemory(sizeof(SEARCHER_THREAD_CONTEXT)); + if (NULL == ctx) + { + Trace("Out of memory.\n"); + return; + } + InitializeSearcherContext(pos, ctx); + uLastSeenDnaGeneration = g_uDnaGeneration; + } + else if (uLastSeenDnaGeneration != g_uDnaGeneration) + { + InitializeSearcherContext(pos, ctx); + uLastSeenDnaGeneration = g_uDnaGeneration; + } + else + { + ReInitializeSearcherContext(pos, ctx); + } + (void)Eval(ctx, -INFINITY, +INFINITY, NULL); + + for (uColor = 0; uColor < 2; uColor++) + { + p = &ctx->sKingDefectDump[uColor]; + Trace("KINGDEFECTDUMP color=%s", COLOR_NAME(uColor)); + for (uSlot = 0; uSlot < KING_RING_NUM_SLOTS; uSlot++) + { + if (!p->fOnBoard[uSlot]) + { + Trace(" sq%u_offboard=1", uSlot); + continue; + } + for (uPieceType = 0; uPieceType < KING_RING_NUM_PIECE_TYPES; uPieceType++) + { + Trace(" sq%u_atk_%s=%u", + uSlot, szPieceName[uPieceType], + p->fAttackedByPieceType[uSlot][uPieceType]); + } + Trace(" sq%u_undef=%u sq%u_def=%u sq%u_xray=%u sq%u_flight=%u", + uSlot, p->fAttackedUndefended[uSlot], + uSlot, p->fAttackedDefended[uSlot], + uSlot, p->fXrayedUnattacked[uSlot], + uSlot, p->fEmptyFlight[uSlot]); + } + Trace(" filestorm=%u filestormopenfile=%u filestormpawn=%u " + "queenproximity=%u initloc=%u countertotal=%u " + "safetyvalue=%u attackerarmyscaler=%u castled=%u minorsathome=%u " + "misscastleshort=%u misscastlelong=%u\n", + p->uFileStormDefects, p->uFileStormOpenFile, p->uFileStormStormingPawn, + p->uQueenNearKing, + p->uInitialCounterByLocation, p->uCounterTotal, + p->uSafetyTableValue, p->uAttackerArmyScaler, + p->fCastled, p->uMinorsAtHome, + p->fMissingCastleShort, p->fMissingCastleLong); + } +} +#endif + + COMMAND(QSearchCommand) /** @@ -2454,6 +2555,14 @@ COMMAND_PARSER_ENTRY g_ParserTable[] = FALSE, FALSE, "Run a static eval" }, +#ifdef KING_DEFECT_DUMP + { "evalkingdump", + EvalKingDumpCommand, + FALSE, + FALSE, + FALSE, + "Dump _EvalKing's raw per-defect breakdown for both sides" }, +#endif { "evaldna", EvalDnaCommand, FALSE, @@ -712,13 +712,44 @@ static SCORE KING_SAFETY_BY_COUNTER[42] = // see the 180-position king-safety-term distribution sampled // 2026-08-30 (median -27, p75 -16, only ~1% worse than -200) for the // scale this needs to slot into. +// 2026-09-14: rescaled from the original -8/count step (see the +// 2026-08-30 comment above -- that step was calibrated against a +// *removed* legacy term's own magnitude, not against real mate data). +// king_safety_data/HANDOFF.md's mate-mining logistic regression found +// `queenproximity` (after folding in the collinear front/back/sq3/sq4 +// atk_queen sub-features it was absorbing -- see +// eval_tune/king_safety_data/HANDOFF.md's 2026-09-14 addendum) has a +// per-unit raw coefficient of +1.00, calibrated against +// `misscastleshort`'s own cleaned coefficient (+0.71) and this table's +// sibling constant KING_MISSING_ONE_CASTLE_OPTION's already-live -23 +// value (same direct-EVAL_TERM unit family, unlike the file-storm +// tables below which feed the separate uCounter/KING_SAFETY_BY_COUNTER +// lookup and need their own, different calibration) -- see +// (misscastleshort_live / misscastleshort_raw_coef) = k, then +// k * queenproximity_raw_coef =~ -32/count. Validate via the usual +// sd10 + self-play gate before trusting this magnitude. static SCORE KING_QUEEN_PROXIMITY_DANGER[7] = -{// 0 1 2 3 4 5 6 - +0, -8, -16, -24, -32, -40, -48 +{// 0 1 2 3 4 5 6 + +0, -32, -64, -96, -128, -160, -192 }; static SCORE KING_MISSING_ONE_CASTLE_OPTION = -23; +// 2026-09-14: new term, not a retune of an existing one. +// king_safety_data/HANDOFF.md's regression found `back_flight` (an +// empty, unattacked ring square on the side of the king AWAY from the +// attacker) is a real, non-collinear PROTECTIVE signal (negative raw +// coefficient -0.035, i.e. more open escape squares behind the king +// predicts LESS mate risk) -- unlike `front_flight`, which is a real +// danger signal already captured by EmptyUnattackedSquare's front +// weight. Calibrated the same way as KING_QUEEN_PROXIMITY_DANGER above +// (misscastleshort-anchored k =~ -32.4, applied to back_flight's raw +// coefficient) =~ +1.1/square, rounded to +1. Small and deliberately +// so -- this is the single most novel, least-collinear finding from +// the regression, worth landing and validating in isolation before +// combining with anything else. +static SCORE KING_BACK_FLIGHT_BONUS = 1; + // Flat "this is bad" flags for _EvalTrappedPieces -- not an attempt to // price the material outcome (search resolves that), just a nudge away // from positions with a piece that looks stuck. ENPRISE_AND_TRAPPED is @@ -788,7 +819,8 @@ static DNA_BASE_SIZE g_EvalDNA[] = { DNA_ARRAY(KING_TO_CENTER), DNA_ARRAY(KING_SAFETY_BY_COUNTER), DNA_ARRAY(KING_QUEEN_PROXIMITY_DANGER), - DNA_VAR(KING_MISSING_ONE_CASTLE_OPTION) + DNA_VAR(KING_MISSING_ONE_CASTLE_OPTION), + DNA_VAR(KING_BACK_FLIGHT_BONUS) }; ULONG @@ -2402,13 +2434,19 @@ Return value: **/ { + // 2026-09-14: scaled up ~1.75-2x relative to the other _EvalKing + // defect tables (king_safety_data/HANDOFF.md, tier 1 -- these two + // came back as the dominant, most stable predictors of a real mate + // by a clear margin over every other defect, but previously shared + // the same +0..+4 magnitude as everything else). Keep this table in + // sync with _ComputeFileStormDefectsSplit's copy below. static ULONG KingStormingPawnDefects[8] = { - +0, +4, +3, +1, +0, +0, +0, +0 + +0, +7, +5, +2, +0, +0, +0, +0 }; static ULONG KingFileDefects[3] = { - +2, +1, +0 + +4, +2, +0 }; ULONG ufColor = FLIP(uColor); COOR c = pos->cNonPawns[uColor][0]; @@ -2449,6 +2487,79 @@ Return value: } +#ifdef KING_DEFECT_DUMP +static void +_ComputeFileStormDefectsSplit(IN POSITION *pos, + IN const PAWN_HASH_ENTRY *pHash, + IN ULONG uColor, + OUT ULONG *puOpenFile, + OUT ULONG *puStormingPawn) +/** + +Routine description: + + Debug-only sibling of _ComputeFileStormDefects that reports its two + independent sub-ideas (open/half-open file bonus vs. storming-pawn + distance penalty) separately instead of pre-summed, so a regression + can weigh them independently -- they already live in two distinct + hand-picked tables (KingFileDefects / KingStormingPawnDefects) with + no structural change needed to retune them individually once this + tells us their relative importance. See CLAUDE.md's king-safety + discussion. Logic intentionally identical to _ComputeFileStormDefects, + just split into two accumulators -- keep both in sync if either + changes. + +**/ +{ + // Keep in sync with _ComputeFileStormDefects's copy above. + static ULONG KingStormingPawnDefects[8] = + { + +0, +7, +5, +2, +0, +0, +0, +0 + }; + static ULONG KingFileDefects[3] = + { + +4, +2, +0 + }; + ULONG ufColor = FLIP(uColor); + COOR c = pos->cNonPawns[uColor][0]; + COOR cFileSq, cSquare; + ULONG u, v; + BITBOARD bb; + int w; + + *puOpenFile = 0; + *puStormingPawn = 0; + for (w = -1; w <= 1; w++) + { + cFileSq = c + w; + if (!IS_ON_BOARD(cFileSq)) + { + continue; + } + u = FILE(cFileSq) + 1; + v = (pHash->uCountPerFile[WHITE][u] > 0) + + (pHash->uCountPerFile[BLACK][u] > 0); + ASSERT((v >= 0) && (v <= 2)); + *puOpenFile += (KingFileDefects[v] + ((v < 2) && ((u == 1) || (u == 8)))); + bb = pos->bbPawns[ufColor] & BBFILE[u - 1]; + if (bb) + { + if (uColor == WHITE) + { + cSquare = CoorFromBitBoardRank1ToRank8(&bb); + } + else + { + cSquare = CoorFromBitBoardRank8ToRank1(&bb); + } + ASSERT(IS_ON_BOARD(cSquare)); + *puStormingPawn += KingStormingPawnDefects[DISTANCE(cSquare, c)]; + } + } +} +#endif + + static ULONG _GetFileStormDefects(IN SEARCHER_THREAD_CONTEXT *ctx, IN POSITION *pos, @@ -4121,6 +4232,7 @@ Return value: BITV bvDefend; BITV bvPattern = 0; ULONG uFlightSquares; + ULONG uBackFlightSquares; ULONG uQueenNearKing = 0; BITBOARD bb; SCORE i; @@ -4147,6 +4259,21 @@ Return value: +15, +16, +17, 0 }; + // 2026-09-14: color-dependent front/back ring-slot classification, + // mirroring EmptyAttackedSquare/EmptyUnattackedSquare's own existing + // asymmetry (WHITE's front is slots 0/1/2, BLACK's is 5/6/7) and + // matching eval_tune/train_king_safety_model.py's BUCKET_SLOTS -- + // used to isolate the back-bucket flight-square bonus below. Slots + // 3/4 (the same-rank side squares) are neither front nor back, same + // as the regression's own treatment. + static const FLAG KING_RING_IS_BACK[2][8] = + { + // BLACK: back = slots 0,1,2 (front = 5,6,7) + { TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE }, + // WHITE: back = slots 5,6,7 (front = 0,1,2) + { FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE }, + }; + static ULONG EmptyAttackedSquare[2][8] = { { @@ -4181,6 +4308,10 @@ Return value: 2 }; +#ifdef KING_DEFECT_DUMP + KING_DEFECT_DUMP_INFO *pDump; +#endif + ASSERT(IS_ON_BOARD(c)); p = pos->rgSquare[c].pPiece; ASSERT(p && IS_KING(p)); @@ -4188,6 +4319,10 @@ Return value: xColor = FLIP(uColor); ASSERT(IS_VALID_COLOR(uColor)); pos->bbKingAttacks[uColor] |= g_KingAttacksBB[c]; +#ifdef KING_DEFECT_DUMP + pDump = &ctx->sKingDefectDump[uColor]; + memset(pDump, 0, sizeof(KING_DEFECT_DUMP_INFO)); +#endif // Don't bother with king safety when the other side has no mating // material. @@ -4201,12 +4336,16 @@ Return value: // Prepare to do king safety uCounter = (u >= KEEP_KING_AT_HOME_THRESHOLD) * KING_INITIAL_COUNTER_BY_LOCATION[uColor][c]; +#ifdef KING_DEFECT_DUMP + pDump->uInitialCounterByLocation = uCounter; +#endif #ifdef EVAL_DUMP Trace("%s Initial KS Counter: %u\n", COLOR_NAME(uColor), uCounter); #endif BITBOARD bbEnemyOcc = _BuildFriendlySideBB(pos, xColor); uFlightSquares = 0; + uBackFlightSquares = 0; u = 0; ASSERT(KingSafetyDeltas[u] != 0); do @@ -4264,6 +4403,54 @@ Return value: // uQueenNearKing += (((bvAttack | bvXray) & QUEEN_BIT) != 0); +#ifdef KING_DEFECT_DUMP + { + BITV bvEither = bvAttack | bvXray; + ASSERT(u < KING_RING_NUM_SLOTS); + pDump->fOnBoard[u] = TRUE; + pDump->fAttackedByPieceType[u][KING_RING_PAWN] = + ((bvEither & PAWN_BIT) != 0); + pDump->fAttackedByPieceType[u][KING_RING_MINOR] = + ((bvEither & MINOR_BIT) != 0); + pDump->fAttackedByPieceType[u][KING_RING_ROOK] = + ((bvEither & ROOK_BIT) != 0); + pDump->fAttackedByPieceType[u][KING_RING_QUEEN] = + ((bvEither & QUEEN_BIT) != 0); + if (bvAttack != 0) + { + // NB: bvDefend here still includes the own-king bit + // that the real code strips below via bvDefend &= + // ~KING_BIT once xrayed/multiply-attacked -- match + // that exact "is this square adequately defended" + // semantics rather than the raw pre-strip value. + BITV bvDefendForDump = bvDefend; + if (bvXray || (bvAttack & (bvAttack - 1))) + { + bvDefendForDump &= ~KING_BIT; + } + if (bvDefendForDump == 0) + { + pDump->fAttackedUndefended[u] = TRUE; + } + else + { + pDump->fAttackedDefended[u] = TRUE; + } + } + else + { + if (bvXray != 0) + { + pDump->fXrayedUnattacked[u] = TRUE; + } + if (!(pos->bbOccupied & sq)) + { + pDump->fEmptyFlight[u] = TRUE; + } + } + } +#endif + // // Tried a branchless rewrite of this attacked/unattacked // split (2026-09-09): compute both sides' contributions @@ -4303,10 +4490,12 @@ Return value: } else { + FLAG fEmpty = ((pos->bbOccupied & sq) == 0); uCounter += (bvXray != 0); ASSERT(u < 8); uCounter += EmptyUnattackedSquare[uColor][u]; - uFlightSquares += ((pos->bbOccupied & sq) == 0); + uFlightSquares += fEmpty; + uBackFlightSquares += (fEmpty && KING_RING_IS_BACK[uColor][u]); } } u++; @@ -4327,6 +4516,12 @@ Return value: // isolated sd10 suite testing before that landing). // uCounter += _GetFileStormDefects(ctx, pos, pHash, uColor); +#ifdef KING_DEFECT_DUMP + pDump->uFileStormDefects = _GetFileStormDefects(ctx, pos, pHash, uColor); + _ComputeFileStormDefectsSplit(pos, pHash, uColor, + &pDump->uFileStormOpenFile, + &pDump->uFileStormStormingPawn); +#endif #ifdef EVAL_DUMP Trace("%s KS Counter post-open file/stormers: %u\n", COLOR_NAME(uColor), uCounter); @@ -4351,6 +4546,14 @@ Return value: // are scaling it based on the _other_ side's material. uCounter = MINU(uCounter, 41); i = KING_SAFETY_BY_COUNTER[uCounter]; +#ifdef KING_DEFECT_DUMP + pDump->uQueenNearKing = uQueenNearKing; + pDump->uCounterTotal = uCounter; + pDump->uSafetyTableValue = (ULONG)(-i); + pDump->fCastled = pos->fCastled[uColor]; + pDump->uMinorsAtHome = pos->uMinorsAtHome[uColor]; + pDump->uAttackerArmyScaler = pos->uArmyScaler[xColor]; +#endif #ifdef CALIBRATE_POSITIONAL // KING_SAFETY_BY_COUNTER is <= 0 throughout (a penalty); the // calibration histogram bins magnitude, matching iKingSwingP90's @@ -4375,6 +4578,14 @@ Return value: KING_QUEEN_PROXIMITY_DANGER[MINU(uQueenNearKing, 6)], "queen proximity danger"); + ASSERT(uBackFlightSquares <= 3); + EVAL_TERM(uColor, + KING, + c, + iKingScore, + (SCORE)(KING_BACK_FLIGHT_BONUS * uBackFlightSquares), + "back rank flight bonus"); + // Bonus for castling / penalty for loss of castle. Also, if side has // not yet castled, be concerned with undeveloped minor pieces. if (FALSE == pos->fCastled[uColor]) @@ -4415,6 +4626,10 @@ Return value: ((CASTLE_LONG[uColor] & pos->bvCastleInfo) == 0) * (SCORE)uDevelopmentScaler) / 8), "can't castle long"); +#ifdef KING_DEFECT_DUMP + pDump->fMissingCastleShort = ((CASTLE_SHORT[uColor] & pos->bvCastleInfo) == 0); + pDump->fMissingCastleLong = ((CASTLE_LONG[uColor] & pos->bvCastleInfo) == 0); +#endif } } diff --git a/src/eval_tune/extract_king_features.py b/src/eval_tune/extract_king_features.py new file mode 100644 index 0000000..509f823 --- /dev/null +++ b/src/eval_tune/extract_king_features.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Feature extraction driver: takes the mined FENs in mate_training_data.csv +and runs each through the engine's `evalkingdump` command (KING_DEFECT_DUMP +build only) to pull _EvalKing's raw per-defect breakdown, joining it back +onto each row's label/metadata to produce the final regression-ready CSV. + +Sharded across N engine subprocesses (one `script` file each, run in +parallel) since 86k FENs through a single serial engine invocation would +be needlessly slow on this 24-core box -- same reasoning as +shard_mine.py's PGN sharding. + +Each engine invocation processes its shard via `script <file>` containing +alternating `setboard <fen>` / `evalkingdump` lines; evalkingdump always +emits exactly two `KINGDEFECTDUMP color=...` lines (black then white, in +that fixed loop order) per position, unconditionally -- so the Nth pair +of KINGDEFECTDUMP lines in the shard's output corresponds to the Nth row +of that shard's input, and we pick out whichever color each row's +"mated_color_or_sampled_color" column names. +""" +import argparse +import csv +import os +import re +import subprocess +import sys +import time + +KV_RE = re.compile(r"(\S+)=(\S+)") + +# Fixed schema, since an off-board ring square (king near an edge/corner) +# emits only "sqN_offboard=1" while an on-board square emits 9 separate +# keys -- different king positions otherwise produce entirely different +# column sets, which would silently misalign rows if we inferred columns +# from whichever row happened to be parsed first (caught by a per-shard +# "feature key mismatch" warning during testing). +PIECE_NAMES = ("pawn", "minor", "rook", "queen") +PER_SQUARE_ON_BOARD_KEYS = [f"atk_{p}" for p in PIECE_NAMES] + \ + ["undef", "def", "xray", "flight"] +GLOBAL_KEYS = ["filestorm", "filestormopenfile", "filestormpawn", + "queenproximity", "initloc", "countertotal", + "safetyvalue", "attackerarmyscaler", "castled", + "minorsathome", "misscastleshort", "misscastlelong"] + +FEATURE_KEYS = [] +for _sq in range(8): + FEATURE_KEYS.append(f"sq{_sq}_offboard") + for _k in PER_SQUARE_ON_BOARD_KEYS: + FEATURE_KEYS.append(f"sq{_sq}_{_k}") +FEATURE_KEYS += GLOBAL_KEYS + + +def parse_kingdefectdump_line(line): + """Returns (color, {key: value_str}) from a 'KINGDEFECTDUMP color=... ...' + line, normalized to FEATURE_KEYS' fixed schema -- off-board squares get + offboard=1 and all per-square on-board keys defaulted to 0 rather than + left absent.""" + parts = line.strip().split(None, 1) + assert parts[0] == "KINGDEFECTDUMP" + raw_kv = dict(KV_RE.findall(parts[1])) + color = raw_kv.pop("color") + + kv = {} + for sq in range(8): + offboard_key = f"sq{sq}_offboard" + if offboard_key in raw_kv: + kv[offboard_key] = raw_kv[offboard_key] + for k in PER_SQUARE_ON_BOARD_KEYS: + kv[f"sq{sq}_{k}"] = "0" + else: + kv[offboard_key] = "0" + for k in PER_SQUARE_ON_BOARD_KEYS: + full_key = f"sq{sq}_{k}" + kv[full_key] = raw_kv[full_key] + for k in GLOBAL_KEYS: + kv[k] = raw_kv[k] + assert set(kv.keys()) == set(FEATURE_KEYS), \ + f"parsed keys don't match fixed schema: {set(kv) ^ set(FEATURE_KEYS)}" + return color, kv + + +def run_shard(engine_path, rows, shard_dir, shard_idx): + script_path = os.path.join(shard_dir, f"extract_{shard_idx}.script") + out_log = os.path.join(shard_dir, f"extract_{shard_idx}.out") + with open(script_path, "w") as f: + f.write("force\n") + f.write("nopost\n") + for row in rows: + f.write(f"setboard {row['fen']}\n") + f.write("evalkingdump\n") + + cmd = [engine_path, "--batch", "--command", + f"script {script_path}"] + with open(out_log, "w") as logf: + rc = subprocess.run(cmd, stdout=logf, stderr=subprocess.STDOUT).returncode + if rc != 0: + raise RuntimeError(f"shard {shard_idx} engine exited {rc}, see {out_log}") + + dumps = [] + with open(out_log) as f: + for line in f: + if line.startswith("KINGDEFECTDUMP"): + dumps.append(parse_kingdefectdump_line(line)) + + if len(dumps) != 2 * len(rows): + raise RuntimeError( + f"shard {shard_idx}: expected {2 * len(rows)} KINGDEFECTDUMP " + f"lines (2 per row), got {len(dumps)} -- a setboard likely " + f"failed silently and evalkingdump re-used a stale position. " + f"See {out_log}.") + + out_rows = [] + for i, row in enumerate(rows): + black_color, black_kv = dumps[2 * i] + white_color, white_kv = dumps[2 * i + 1] + assert black_color == "black" and white_color == "white", \ + f"row {i}: unexpected color order {black_color}/{white_color}" + target = row["mated_color_or_sampled_color"] + kv = black_kv if target == "black" else white_kv + out_row = dict(row) + out_row.update(kv) + out_rows.append(out_row) + + os.remove(script_path) + os.remove(out_log) + return out_rows + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("training_csv") + ap.add_argument("out_csv") + ap.add_argument("--engine", default="../typhoon_kingdump") + ap.add_argument("--num-shards", type=int, default=8) + ap.add_argument("--work-dir", default=None) + args = ap.parse_args() + + engine_path = os.path.abspath(args.engine) + if not os.path.isfile(engine_path): + print(f"Engine not found: {engine_path}", file=sys.stderr) + sys.exit(1) + + with open(args.training_csv, newline="") as f: + rows = list(csv.DictReader(f)) + print(f"Loaded {len(rows)} rows from {args.training_csv}", file=sys.stderr) + + work_dir = args.work_dir or os.path.dirname(os.path.abspath(args.out_csv)) + os.makedirs(work_dir, exist_ok=True) + + n = args.num_shards + shard_size = (len(rows) + n - 1) // n + shards = [rows[i:i + shard_size] for i in range(0, len(rows), shard_size)] + + t0 = time.time() + all_out_rows = [[] for _ in shards] + + import concurrent.futures + with concurrent.futures.ThreadPoolExecutor(max_workers=n) as pool: + futures = {} + for idx, shard_rows in enumerate(shards): + fut = pool.submit(run_shard, engine_path, shard_rows, work_dir, idx) + futures[fut] = idx + for fut in concurrent.futures.as_completed(futures): + idx = futures[fut] + out_rows = fut.result() + all_out_rows[idx] = out_rows + print(f"Shard {idx} done ({len(out_rows)} rows)", file=sys.stderr) + + print(f"All shards done in {time.time() - t0:.1f}s", file=sys.stderr) + + base_cols = ["label", "mated_color_or_sampled_color", "n_back", + "game_index", "control_source"] + fieldnames = base_cols + FEATURE_KEYS + + with open(args.out_csv, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=fieldnames) + w.writeheader() + for out_rows in all_out_rows: + for row in out_rows: + w.writerow({k: row.get(k) for k in fieldnames}) + + total = sum(len(r) for r in all_out_rows) + print(f"Wrote {total} feature rows to {args.out_csv}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/src/eval_tune/king_safety_data/HANDOFF.md b/src/eval_tune/king_safety_data/HANDOFF.md new file mode 100644 index 0000000..b05e3d2 --- /dev/null +++ b/src/eval_tune/king_safety_data/HANDOFF.md @@ -0,0 +1,254 @@ +# King-safety defect regression: handoff + +## Goal + +`_EvalKing` (eval.c) sums a pile of hand-picked "defect" constants into one +scalar (`uCounter`), then runs that through a single lookup table +(`KING_SAFETY_BY_COUNTER`). The hand-picked weights assume every defect +counts equally, which is suspect -- e.g. a queen touching an undefended +ring square and a knight touching one add the exact same `+1`. This +project mines real games to see which defects actually predict getting +mated, so `eval.c`'s weights can be informed by data instead of guesswork. + +**Method**: find real middlegame checkmates in human games, look at the +losing king's position N plies earlier, and compare its raw defect +pattern against a matched "sound" control position (drawn games) with the +same attacker material and development level. Fit a logistic regression; +features with a real, stable coefficient are candidates for reweighting +in `eval.c`. See `CLAUDE.md`'s "King-safety eval improvement project" +section (search for it) for the full narrative this file summarizes. + +## Where everything is + +- **Final training data**: `remine_v4/king_features_matched_v4.csv` -- + 86,362 rows (43,181 matched positive/control pairs). **Use this one.** +- `remine_v4/merged.csv` -- raw mined FENs before feature extraction + (172,735 rows: 43,181 positives + 129,554 control candidates). +- `remine_v4/king_features_v4.csv` -- all 172,735 rows with full + `_EvalKing` defect breakdown, before matching trims it down to the + 86,362 confound-controlled rows. +- Pipeline scripts live in `eval_tune/` (sibling of this dir), not here: + `mine_king_mates.py`, `shard_mine.py`, `extract_king_features.py`, + `match_controls.py`, `train_king_safety_model.py`. +- Engine side: `chess.h`'s `KING_DEFECT_DUMP_INFO` struct, + `_EvalKing`'s `#ifdef KING_DEFECT_DUMP` instrumentation in eval.c, and + command.c's `evalkingdump` command -- all gated behind the + `KING_DEFECT_DUMP=1` gmake build flag (NOT part of a normal build). + `../typhoon_kingdump` is a saved binary built with that flag; rebuild + it (`gmake clean` first -- flag changes don't trigger recompilation on + their own, see "Gotchas" below) if `eval.c`/`chess.h`/`command.c` + change. + +## Final result (from `king_features_matched_v4.csv`) + +Logistic regression, 80/20 train/test split, standardized coefficients. +Test AUC 0.91 (plausible for "engineered danger features vs. proven-sound +draws", not suspiciously perfect). Ranked by |coefficient|, most-to-least +trustworthy tier first (stability = full-population vs. top-tercile-army- +strength fits agreeing): + +**Tier 1 -- dominant, very stable**: `filestormopenfile` (+0.57 to +0.80), +`filestormpawn` (+0.51 to +0.54). These are eval.c's `KingFileDefects[3]` +and `KingStormingPawnDefects[8]` tables (open/half-open file near the +king; how far an enemy pawn has advanced up that file), already +independent tables -- lowest-risk place to start a real retune. + +**Tier 2 -- strong, stable**: `front_offboard`, `front_def`, `sq4_offboard`, +`front_undef`, `front_atk_minor`, `front_atk_rook`, `misscastleshort`, +`front_flight` (+0.33 to +0.45). "front" = the diagonal/vertical ring +squares one rank away, in the direction each side attacks toward (see +"Feature schema" below for the exact slot mapping -- do NOT assume this +means "in front of the king" the way a human would first guess; see the +correction note there). `sq3`/`sq4` are the two same-rank side squares, +kept individually resolved. + +**Tier 3 -- moderate, less stable**: `back_offboard` (drifts the most of +any surviving feature, treat with suspicion), `sq3_undef`, +`attackerarmyscaler`, `queenproximity`, `back_atk_minor`, `front_atk_queen`. + +**Confirmed noise -- do not retune based on this data**: `minorsathome` +(coefficient ~0.001, was previously the #1 "signal" before two sampling +bugs got fixed -- see "Bugs found" below), `initloc`, `castled` (exactly +0.0000, fully collinear with `misscastleshort`/`misscastlelong`). + +## Translating this into eval.c -- confidence and plan + +This is NOT a "throw away eval.c and use the model" situation. The plan +is targeted retuning of existing constants/tables, informed by relative +coefficient *ratios*, not literal log-odds-to-centipawn conversion: + +1. **`KingFileDefects`/`KingStormingPawnDefects` deserve a real weight + increase relative to everything else** -- they're the strongest + signal by a clear margin and already live in their own small, + isolated tables (`_ComputeFileStormDefects` in eval.c). Lowest risk, + highest expected payoff, start here. +2. **`EmptyAttackedSquare`/`EmptyUnattackedSquare`'s front/back asymmetry + is directionally right but probably too weak a spread** -- front- + bucket features consistently outrank back-bucket ones here by a good + margin. +3. **`misscastleshort`'s constant likely deserves a bump.** +4. **Don't touch `KING_INITIAL_COUNTER_BY_LOCATION` or the undeveloped- + minors term** based on this data -- both came back as noise. +5. Neither `KingFileDefects`/`KingStormingPawnDefects` nor the + ring-scan tables are currently `DNA_ARRAY`-registered (checked: only + `KING_SAFETY_BY_COUNTER` and `KING_QUEEN_PROXIMITY_DANGER` are under + king-safety). Retuning them means a manual edit (optionally promoting + them to file scope + registering as DNA arrays for future tuning), + not "rerun the tuner." +6. **Absolute magnitude has to be calibrated the normal way** -- run any + change through `sd10` ECM suite comparison against `head_reference/` + and `match_play.py`'s gated head-to-head match, same as any other + hand-tuned eval constant. The regression tells you *shape* + (which defects matter more relative to each other), not the right + centipawn magnitude for a chess engine's scoring scale. +7. `CountKingSafetyDefects` (the cheap lazy-eval proxy) stays untouched + until a real change lands and is validated -- agreed early in this + project, still the right call. + +## Feature schema (read this before extending the pipeline) + +`KingSafetyDeltas` in eval.c (`-17,-16,-15,-1,+1,+15,+16,+17`) defines 8 +ring slots, indices 0-7 in that order. **Correction made mid-project**: +slots 3 and 4 (deltas -1/+1) are the two SAME-RANK side squares, not +"front of the king" -- an earlier description in conversation got this +wrong. The diagonal/vertical slots (0,1,2 and 5,6,7) are the ones with a +real front/back asymmetry, matching `EmptyAttackedSquare`'s own existing +color-dependent weighting (WHITE's front is slots 0/1/2, BLACK's is +5/6/7 -- see `_EvalKing`'s comments and `KING_RING_FRONT`-style logic). + +`extract_king_features.py` dumps full per-slot resolution (all 8 slots x +4 piece types x several state flags) since the engine-side cost of doing +so is free once you're calling `evalkingdump` anyway. `train_king_safety_model.py` +is where slots 0/1/2/5/6/7 get bucketed into "front"/"back" sums +(color-dependent per row's `mated_color_or_sampled_color`), while slots 3/4 +stay individually resolved -- this bucketing was added in this session +after the diagonal per-slot coefficients turned out to be the noisiest, +most sign-flipping entries in every full-vs-subset drift comparison. +`filestorm` is deliberately NOT used as a training feature -- its two +independent sub-components (`filestormopenfile`, `filestormpawn`) are +used instead, since they map onto two already-separate eval.c tables and +there's no reason to make the regression re-derive their relative +importance from one pre-summed number. + +## Bugs found and fixed this session (read before trusting old data) + +If you find any `.csv` file in this project OTHER than +`remine_v4/*_v4.csv` and its merged.csv, IT PREDATES ONE OR MORE OF THESE +FIXES AND SHOULD NOT BE USED. (Older files were deleted during cleanup, +but if this pipeline gets rerun and produces intermediate files again, +this is the checklist for whether they're trustworthy.) + +1. **Bare-king endgame mates leaking into "middlegame" positives.** Only + checking the attacker's material (`>= 800cp`, roughly rook+bishop) let + trivial K+Q-vs-lone-K technique mates through. Fixed by also requiring + `total_non_pawn_piece_count(board) >= 8` (mirrors `_EvalKing`'s own + `pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0] < 8` + endgame boundary exactly) -- see `is_middlegame()` in + `mine_king_mates.py`. +2. **`min_ply` too low (10, i.e. move 5) for control sampling.** A large + fraction of the pool's draws are short draws-by-agreement close to + `filter_pgn.py`'s own 40-ply floor. Combined with a large end-guard + (staying clear of the game's end), the only valid sampling window on a + short draw was the early opening -- where minors are legitimately + still undeveloped for completely ordinary reasons. This alone caused + `minorsathome` to look like a massive, spurious "signal" (controls + showed `minorsathome>=2` in ~85% of samples). Verified the engine's + own computation was correct on a real developed-middlegame FEN -- + 100% a sampling-window bug, not an eval bug. Fixed: `min_ply` raised + to 20 (move 10), and a game too short to satisfy both `min_ply` and + the end-guard is now skipped entirely rather than force-sampled early. +3. **Control-sampling ply selection secretly always picked the earliest + plies in a game's valid window.** `rng.shuffle(candidates)` followed + by `for idx in sorted(candidates)` completely undid the shuffle. + Fixed in two steps (the first fix wasn't sufficient on its own -- + see the code comment in `mine_controls()` for why): randomly + subsample which plies to try, replay ascending through all of them + ONCE (required -- boards can only be walked forward), collect every + result that passes filters into a list, THEN randomly choose which + results to actually keep via `rng.sample` on that list. Verified: + sampled-ply-as-fraction-of-game-length now has a median around 0.46 + with real tail into 60%+, instead of clustering at 20-30%. +4. **Matching-radius bug treated two very differently-scaled confound + columns with one shared distance metric.** `attackerarmyscaler` + spans 0-31 (`ASSERT`-confirmed hard clamp ceiling), `minorsathome` + spans 0-4. A single Chebyshev radius meant a "tight" radius-2 match + still let `minorsathome` drift by up to 2 -- literally half its + entire range. This is why `minorsathome` was STILL the #1 "signal" + even after an initial matching attempt. Fixed: `match_controls.py` + now takes `--exact-cols` (must match exactly, no radius search -- + `minorsathome` defaults here) and `--radius-cols` (allowed to expand + outward, `attackerarmyscaler` defaults here) as separate concepts. +5. **KDTree-based nearest-neighbor matching matched almost nothing** (308 + of 43,181 positives) because the match columns are low-cardinality + integers -- huge exact-value clusters mean a KDTree's "k nearest" are + massively tied, so the first few positives processed in a cluster + exhaust the shared candidate set and everyone else in that cluster + fails to match. Replaced with exact-bucket + expanding-integer-radius + matching (see `match_controls.py`), which has no such collision + problem. +6. **`_EvalKing`'s per-square ring scan implicitly assumed all 8 slots + were geometrically equivalent for feature purposes**; slots 3/4 + (side squares) vs. the diagonal slots turned out to behave + differently enough that per-slot resolution on the diagonals was + pure noise at the sample sizes available -- see "Feature schema" + above. +7. **Board-diagram spam.** `setboard` prints a full ASCII board dump on + every call by default (`g_Options.fShouldPost`, see `DumpPosition` in + gamelist.c), which dominated per-row I/O cost during feature + extraction. Fixed by prepending `nopost` to every extraction script. +8. **Two separate concurrent-process races during interactive mining**, + both self-inflicted mid-session: (a) `$!` after a `cd ... && nohup + ...` pipeline captured the wrong PID, leaving a stale duplicate + process racing on the same output file as a supposedly-fresh rerun; + (b) killing that process didn't fully stop it before a new one + started. No lasting damage (caught via `ps aux` before trusting any + output), but: **always verify a background job is actually singular + via `ps aux | grep <script>` before trusting its output**, especially + after a kill+relaunch sequence. +9. **`sizeof(KING_DEFECT_DUMP)` name collision with the + `-DKING_DEFECT_DUMP` build flag.** The build flag is defined as + literal `1` by the compiler; naming the struct type the same thing + caused the preprocessor to mangle the struct declaration wherever the + type name appeared. Renamed the struct to `KING_DEFECT_DUMP_INFO`. + If you ever add a new `#ifdef SOME_NAME` build flag AND a same-named + type/variable in the same session, expect this failure mode. +10. **`gmake` doesn't recompile on a flag-only change.** After the struct + rename (#9) and other `chess.h` edits, a `gmake KING_DEFECT_DUMP=1` + without a preceding `gmake clean` silently reused stale `.o` files + from an earlier flag-less build, producing a binary that didn't + actually have `evalkingdump` registered, with no compile error at + all. Always `gmake clean` first when a build-flag combination + changes, not just when headers change (this is already called out + in `CLAUDE.md` for header changes, but the flag-only case is easy to + forget too). + +## Performance notes for rerunning + +- `pgn/twic_filtered.pgn` (642,268 games) mines in a few minutes when + sharded 12-24 ways via `shard_mine.py` (byte-range sharding on real + game boundaries, found via a fast raw-text `[Event ` scan). Don't run + it unsharded -- a single-process full scan took ~40 minutes. +- Feature extraction (`extract_king_features.py`) is the slow stage: + ~54 minutes for 172,735 rows sharded 18 ways on a contended 24-core + box. `nopost` (fix #7 above) is essential -- without it, board-diagram + printing dominates I/O cost. +- `match_controls.py` is pure in-memory bucket lookups, seconds even for + the full dataset -- never worth sharding. +- If `twic_filtered.pgn` (642k games, >=2400 Elo / >40 ply filtered) ever + proves insufficient for match density, `pgn/twic.pgn` is 7-8x larger + (4,099,094 games, unfiltered) and available as a fallback source -- + not needed this run (100% match rate achieved on `twic_filtered.pgn` + once the above bugs were fixed). + +## Suggested next steps + +1. Pick 2-3 concrete constant changes from Tier 1/2 above (start with + `KingFileDefects`/`KingStormingPawnDefects` -- lowest risk). +2. Land them as a normal `eval.c` diff, run `precommit_check.sh`. +3. `sd10` comparison across the three curated suites against + `head_reference/`. +4. `match_play.py` gated head-to-head match against the pre-change DNA. +5. Only keep the change if `LOWER95 >= 0.5`. +6. Once a change lands, consider whether `CountKingSafetyDefects` (lazy + eval) needs corresponding rework -- deferred by design until a real + change is validated (see project goal section above). diff --git a/src/eval_tune/match_controls.py b/src/eval_tune/match_controls.py new file mode 100644 index 0000000..10bb821 --- /dev/null +++ b/src/eval_tune/match_controls.py @@ -0,0 +1,168 @@ +#!/usr/bin/env python3 +"""Case-control matching to remove the attacker-material/development +confound found in the naive full-population regression: attackerarmyscaler +alone got AUC 0.99 distinguishing mate-precursor positions from +draw-derived controls, causing quasi-complete separation that destabilizes +every other coefficient in the joint fit. Fix: match each positive to the +nearest available control by (attackerarmyscaler, minorsathome) instead of +using controls sampled freely from the draw pool. See CLAUDE.md's +king-safety discussion for the full methodology. + +Uses exact-bucket + expanding-radius matching, not a KDTree: the match +columns are low-cardinality integers, so exact-value clusters are huge and +a KDTree's k nearest neighbors are highly tied -- every positive in a big +cluster sees almost the same "nearest k" candidates, so the first few +positions processed exhaust that shared candidate set and everyone else in +the cluster fails to match (confirmed: a k=50 KDTree approach matched only +308 of 43181 positives). Bucketing by exact integer key, popping +candidates on use, and expanding search radius only when a bucket is +empty has no such collision problem and is O(1) amortized per positive. + +--exact-cols vs --radius-cols, NOT a single shared radius across all match +columns: a first version used one Chebyshev radius across both +attackerarmyscaler (range 0-31) and minorsathome (range 0-4) -- "radius <= +2" is tight for army-scaler (~6% of its range) but is HALF of +minorsathome's entire range, so positives kept matching against controls +with a wildly different minorsathome just to satisfy the harder +army-scaler dimension. Confirmed by inspecting matched pairs: mean +minorsathome gap of 1.3 survived a nominally "tight" radius-2 match, +leaving minorsathome as the regression's #1 feature for reasons that were +substantially still the sampling artifact, not real signal. Fix: columns +in --exact-cols must match exactly (no radius search at all -- if a +positive's exact value has no candidates left, it's unmatched, full +stop); only --radius-cols expand outward. minorsathome defaults to exact +since its range is small enough that "close" isn't actually close. +""" +import argparse +import csv +import itertools +import random +import sys +from collections import defaultdict + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("features_csv") + ap.add_argument("out_csv") + ap.add_argument("--exact-cols", nargs="+", default=["minorsathome"], + help="Match columns that must match EXACTLY -- no " + "radius search. Use for low-cardinality columns " + "where 'close' isn't actually close.") + ap.add_argument("--radius-cols", nargs="+", default=["attackerarmyscaler"], + help="Match columns allowed to expand outward by " + "--max-radius if no exact match is available.") + ap.add_argument("--max-radius", type=int, default=30, + help="Max Chebyshev distance across --radius-cols to " + "search outward before giving up on a positive") + ap.add_argument("--seed", type=int, default=42) + args = ap.parse_args() + + rng = random.Random(args.seed) + + with open(args.features_csv, newline="") as f: + reader = csv.DictReader(f) + assert reader.fieldnames is not None + fieldnames = list(reader.fieldnames) + rows = list(reader) + + positives = [r for r in rows if r["label"] == "1"] + candidates = [r for r in rows if r["label"] == "0"] + print(f"Loaded {len(positives)} positives, {len(candidates)} control " + f"candidates", file=sys.stderr) + print(f"Exact-match columns: {args.exact_cols}", file=sys.stderr) + print(f"Radius-search columns (max radius {args.max_radius}): " + f"{args.radius_cols}", file=sys.stderr) + + def exact_key_of(r): + return tuple(int(round(float(r[c]))) for c in args.exact_cols) + + def radius_key_of(r): + return tuple(int(round(float(r[c]))) for c in args.radius_cols) + + # Two-level index: exact key -> radius key -> list of candidate rows. + # A positive only ever searches within its own exact-key bucket, and + # within that, expands outward over the radius-cols only. + buckets = defaultdict(lambda: defaultdict(list)) + for r in candidates: + buckets[exact_key_of(r)][radius_key_of(r)].append(r) + for ekey in buckets: + for rkey in buckets[ekey]: + rng.shuffle(buckets[ekey][rkey]) + + ndim = len(args.radius_cols) + + def offsets_at_radius(radius): + if radius == 0: + return [(0,) * ndim] + rng_vals = range(-radius, radius + 1) + return [combo for combo in itertools.product(rng_vals, repeat=ndim) + if max(abs(v) for v in combo) == radius] + + shells = [offsets_at_radius(r) for r in range(args.max_radius + 1)] + + matched_pairs = [] + unmatched = 0 + unmatched_no_exact_bucket = 0 + radius_hist = defaultdict(int) + + for pos_row in positives: + ekey = exact_key_of(pos_row) + exact_bucket = buckets.get(ekey) + if exact_bucket is None: + unmatched += 1 + unmatched_no_exact_bucket += 1 + continue + pkey = radius_key_of(pos_row) + found = None + found_radius = None + for radius, shell in enumerate(shells): + for off in shell: + cand_key = tuple(pkey[i] + off[i] for i in range(ndim)) + cand_list = exact_bucket.get(cand_key) + if cand_list: + found = cand_list.pop() + found_radius = radius + break + if found is not None: + break + if found is None: + unmatched += 1 + continue + radius_hist[found_radius] += 1 + matched_pairs.append((pos_row, found)) + + print(f"Matched {len(matched_pairs)} pairs, {unmatched} positives " + f"left unmatched ({unmatched_no_exact_bucket} had no candidates " + f"at all with that exact-column value, {unmatched - unmatched_no_exact_bucket} " + f"exhausted radius {args.max_radius} on the radius columns)", + file=sys.stderr) + print("Match radius histogram (0 = exact match on everything):", + file=sys.stderr) + for r in sorted(radius_hist): + print(f" radius {r}: {radius_hist[r]}", file=sys.stderr) + + with open(args.out_csv, "w", newline="") as f: + w = csv.DictWriter(f, fieldnames=fieldnames) + w.writeheader() + for pos_row, ctl_row in matched_pairs: + w.writerow(pos_row) + w.writerow(ctl_row) + + print(f"Wrote {2 * len(matched_pairs)} matched rows to {args.out_csv}", + file=sys.stderr) + + import numpy as np + all_cols = args.exact_cols + args.radius_cols + def full_key_of(r): + return tuple(int(round(float(r[c]))) for c in all_cols) + matched_pos_vals = np.array([full_key_of(p) for p, _ in matched_pairs]) + matched_ctl_vals = np.array([full_key_of(c) for _, c in matched_pairs]) + for i, col in enumerate(all_cols): + diff = np.abs(matched_pos_vals[:, i] - matched_ctl_vals[:, i]) + print(f" {col}: mean|diff|={diff.mean():.2f} max|diff|={diff.max():.2f}", + file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/src/eval_tune/mine_king_mates.py b/src/eval_tune/mine_king_mates.py new file mode 100644 index 0000000..7b5866b --- /dev/null +++ b/src/eval_tune/mine_king_mates.py @@ -0,0 +1,315 @@ +#!/usr/bin/env python3 +"""Mine middlegame checkmate positions + matched controls for king-safety +defect regression. See CLAUDE.md / conversation context: _EvalKing's +counted "defects" are summed into one scalar and run through one lookup +table with no per-defect weighting; this pulls training data to fit +relative weights instead of hand-picking them. + +Output: a CSV of (fen, label, mated_color, n_back, game_id) rows. FEN +extraction only -- no engine call here. The actual per-defect feature +values get pulled from these FENs by the engine-side dump command +(separate step). +""" +import argparse +import chess +import chess.pgn +import csv +import random +import sys + +# Matches eval.c's DO_KING_SAFETY_THRESHOLD minus the VALUE_KING baseline +# _EvalKing adds internally: real threshold is "enemy has at least a +# rook+bishop of non-pawn, non-king material" before king safety is even +# scored at all. No point mining data _EvalKing would ignore. +MATING_MATERIAL_THRESHOLD_CP = 800 + +PIECE_VALUES_CP = { + chess.PAWN: 0, # excluded: "non-pawn material" by definition + chess.KNIGHT: 300, + chess.BISHOP: 300, + chess.ROOK: 500, + chess.QUEEN: 975, + chess.KING: 0, # excluded: baseline, not a mating-material contributor +} + + +def non_pawn_material_cp(board, color): + total = 0 + for piece_type in (chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN): + total += len(board.pieces(piece_type, color)) * PIECE_VALUES_CP[piece_type] + return total + + +# Mirrors _EvalKing's own middlegame/endgame boundary exactly (eval.c: +# "u = pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0]; if (u < +# 8)") -- total non-pawn pieces, including both kings, across both sides. +# Below this, _EvalKing itself stops caring about ring-square/file-storm +# defects and switches to endgame-only terms (centralize king, escort +# passer). Checking the attacker's material alone (non_pawn_material_cp) +# is NOT sufficient on its own: a lone queen mating a bare king trivially +# clears that floor but is a basic mating technique, not a middlegame +# king-safety collapse -- caught by manually sampling FENs from the mined +# set (several K+Q-vs-lone-K endings slipped through material-only +# filtering). +def total_non_pawn_piece_count(board): + count = 2 # both kings + for color in (chess.WHITE, chess.BLACK): + for piece_type in (chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN): + count += len(board.pieces(piece_type, color)) + return count + + +MIDDLEGAME_MIN_TOTAL_NON_PAWN_PIECES = 8 + + +def is_middlegame(board): + return total_non_pawn_piece_count(board) >= MIDDLEGAME_MIN_TOTAL_NON_PAWN_PIECES + + +def mine_game(game, n_backs, min_ply_from_start=20): + """Yields (fen, mated_color, n_back) for a game ending in checkmate.""" + board = game.board() + moves = list(game.mainline_moves()) + if not moves: + return + # Replay to the end to confirm checkmate and find the mated side. + for mv in moves: + board.push(mv) + if not board.is_checkmate(): + return + mated_color = board.turn # side to move at mate = side that got mated + total_plies = len(moves) + + for n_back in n_backs: + idx = total_plies - n_back + if idx < min_ply_from_start: + continue + replay = game.board() + for mv in moves[:idx]: + replay.push(mv) + # Only meaningful if the mated side's king safety was even "in + # scope" for _EvalKing at this point -- attacker must have + # mating material. + attacker = not mated_color + if non_pawn_material_cp(replay, attacker) < MATING_MATERIAL_THRESHOLD_CP: + continue + if not is_middlegame(replay): + continue + yield replay.fen(), mated_color, n_back + + +def is_draw(game): + return game.headers.get("Result", "") == "1/2-1/2" + + +def mine_controls(game, n_needed, rng, min_ply=20, max_ply_from_end=15): + """Yields (fen, color) control samples from a non-mate game (or a mate + game's earlier-game portion), gated on the same material threshold, + kept well clear of the game's ending so a real collapse (if any) isn't + accidentally captured as a "stable" control. + + Draw games should be called with a larger max_ply_from_end than + decisive/resignation games (see is_draw callers in main()) -- a draw + only vouches for a king position if the position stayed quiet for a + real stretch *after* the sample point too, not just "the game + eventually drew" (e.g. a middlegame attack that later got defused into + a simplified drawn endgame doesn't vouch for the earlier position). + + min_ply must be large enough that development is realistically done + (move ~10, ply 20) -- NOT just "not literally move 1". A large chunk + of the pool's draws are short draws-by-agreement close to + filter_pgn.py's own 40-ply floor; combined with a big max_ply_from_end + and a too-small min_ply, the only plies satisfying BOTH constraints on + a short game land in the early opening, where minorsathome/development + features are legitimately still nonzero for completely ordinary + reasons -- not a real king-safety defect. Confirmed as the actual bug + behind an earlier version's controls showing minorsathome>=2 in ~85% + of samples: the engine's own computation was verified correct on a + real developed middlegame FEN, so the contamination was 100% a + sampling-window problem, not an eval bug. + + A game too short to satisfy both min_ply and max_ply_from_end is + skipped entirely rather than forced into an early sample -- fewer, + honest control samples beat many contaminated ones.""" + moves = list(game.mainline_moves()) + total_plies = len(moves) + end_guard = max_ply_from_end + if total_plies < min_ply + end_guard: + return + candidates = list(range(min_ply, max(min_ply, total_plies - end_guard))) + if not candidates: + return + # BUG #1 (found while investigating why controls skewed early-game): + # shuffling `candidates` then iterating `sorted(candidates)` completely + # undid the shuffle, so every game always visited its window in + # ascending ply order and grabbed the *earliest* plies satisfying + # per_game_budget -- systematically starving late-middlegame coverage + # in any game with a wide candidate window, regardless of min_ply. + # + # BUG #2 (found immediately after "fixing" #1 with a random subsample + # + sort): stopping the ascending-order loop as soon as `produced >= + # n_needed` still biases toward whichever subsample members happen to + # be smallest, since ascending iteration always finds the low end of + # the subsample first. Measured: even after subsampling, sampled plies + # still clustered at the 20th-40th percentile of game length instead + # of spreading across the whole game. + # + # Fix for both: replay ascending through an oversampled random subset + # ONCE (required -- python-chess boards are only walkable forward) to + # collect every candidate result that passes the material/middlegame + # filters, then pick which ones to actually keep via `rng.sample` on + # that result LIST -- decoupling "must replay in order" from "which + # results count" is what actually gives uniform ply coverage. + num_to_try = min(len(candidates), max(n_needed * 4, 20)) + sample_indices = sorted(rng.sample(candidates, num_to_try)) + replay = game.board() + ply = 0 + valid_results = [] + for idx in sample_indices: + while ply < idx: + replay.push(moves[ply]) + ply += 1 + if not is_middlegame(replay): + continue + for color in (chess.WHITE, chess.BLACK): + attacker = not color + if non_pawn_material_cp(replay, attacker) >= MATING_MATERIAL_THRESHOLD_CP: + valid_results.append((replay.fen(), color)) + + if len(valid_results) > n_needed: + valid_results = rng.sample(valid_results, n_needed) + for fen, color in valid_results: + yield fen, color + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("pgn_file") + ap.add_argument("out_csv") + ap.add_argument("--n-backs", type=int, nargs="+", default=[4, 6, 8, 10]) + ap.add_argument("--max-games", type=int, default=None, + help="Stop after scanning this many games (0/None = all)") + ap.add_argument("--control-ratio", type=float, default=1.0, + help="Controls per positive, roughly") + ap.add_argument("--draw-only-controls", action="store_true", + help="Don't fall back to decisive/resignation games " + "for controls even if draws don't fill the quota") + ap.add_argument("--draw-end-guard", type=int, default=24, + help="Plies of quiet play required *after* a draw " + "control sample, stricter than decisive games " + "since a draw only vouches for a position that " + "stayed sound past the sample point too") + ap.add_argument("--decisive-end-guard", type=int, default=15) + ap.add_argument("--min-ply", type=int, default=20, + help="Earliest ply eligible for any sample (positive " + "or control) -- must be well past normal opening " + "development (ply 20 = move 10), not just >0. " + "See mine_controls' docstring for why this " + "matters a lot for short draws.") + ap.add_argument("--seed", type=int, default=42) + ap.add_argument("--byte-start", type=int, default=0, + help="Seek here before reading (shard support -- " + "must be a real game-start offset, e.g. from " + "shard_mine.py, not an arbitrary byte)") + ap.add_argument("--byte-end", type=int, default=None, + help="Stop once past this offset (shard support)") + args = ap.parse_args() + + rng = random.Random(args.seed) + positives = [] + draw_pool_games = [] + decisive_pool_games = [] + + with open(args.pgn_file, encoding="utf-8", errors="replace") as f: + if args.byte_start: + f.seek(args.byte_start) + n_scanned = 0 + while True: + if args.max_games and n_scanned >= args.max_games: + break + if args.byte_end is not None and f.tell() >= args.byte_end: + break + game = chess.pgn.read_game(f) + if game is None: + break + n_scanned += 1 + if n_scanned % 20000 == 0: + print(f"...scanned {n_scanned} games, " + f"{len(positives)} positives so far", file=sys.stderr) + try: + found_any = False + for fen, mated_color, n_back in mine_game( + game, args.n_backs, min_ply_from_start=args.min_ply): + positives.append((fen, 1, chess.COLOR_NAMES[mated_color], + n_back, n_scanned)) + found_any = True + if not found_any: + if is_draw(game): + draw_pool_games.append(game) + else: + decisive_pool_games.append(game) + except Exception as e: + print(f"skip game {n_scanned}: {e}", file=sys.stderr) + continue + + print(f"Scanned {n_scanned} games -> {len(positives)} positive samples " + f"from checkmate games. Control pools: {len(draw_pool_games)} " + f"draws, {len(decisive_pool_games)} decisive/other.", + file=sys.stderr) + + n_controls_needed = int(len(positives) * args.control_ratio) + controls = [] + per_game_budget = 2 + + # Draws first -- a position two comparable players couldn't crack is + # stronger "this was sound" evidence than a resignation, where the + # king position may simply never have been tested by that opponent. + rng.shuffle(draw_pool_games) + for game in draw_pool_games: + if len(controls) >= n_controls_needed: + break + try: + for fen, color in mine_controls(game, per_game_budget, rng, + min_ply=args.min_ply, + max_ply_from_end=args.draw_end_guard): + controls.append((fen, 0, chess.COLOR_NAMES[color], -1, None, "draw")) + except Exception: + continue + + n_from_draws = len(controls) + print(f"Collected {n_from_draws} control samples from draws.", + file=sys.stderr) + + if not args.draw_only_controls and len(controls) < n_controls_needed: + rng.shuffle(decisive_pool_games) + for game in decisive_pool_games: + if len(controls) >= n_controls_needed: + break + try: + for fen, color in mine_controls(game, per_game_budget, rng, + min_ply=args.min_ply, + max_ply_from_end=args.decisive_end_guard): + controls.append((fen, 0, chess.COLOR_NAMES[color], -1, None, "decisive")) + except Exception: + continue + print(f"Collected {len(controls) - n_from_draws} additional control " + f"samples from decisive/resignation games (quota not met by " + f"draws alone).", file=sys.stderr) + + print(f"Collected {len(controls)} control samples total.", file=sys.stderr) + + with open(args.out_csv, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["fen", "label", "mated_color_or_sampled_color", "n_back", + "game_index", "control_source"]) + for row in positives: + w.writerow(row + (None,)) + for row in controls: + w.writerow(row) + + print(f"Wrote {len(positives) + len(controls)} rows to {args.out_csv}", + file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/src/eval_tune/shard_mine.py b/src/eval_tune/shard_mine.py new file mode 100644 index 0000000..6c82ed3 --- /dev/null +++ b/src/eval_tune/shard_mine.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Parallelize mine_king_mates.py across N shards by game-boundary byte +offset, since a full 642k-game scan is CPU-bound (python-chess replay + +SAN parsing per game) and this box has 24 cores. Finds "[Event " line +starts via a fast raw byte scan (not a full PGN parse) to get exact game +boundaries, splits into N even-sized shards by game count, launches one +mine_king_mates.py subprocess per shard with --byte-start/--byte-end, then +concatenates the resulting CSVs (one header, rows from all shards). +""" +import argparse +import csv +import os +import subprocess +import sys +import time + + +def find_game_offsets(pgn_file): + offsets = [] + pos = 0 + with open(pgn_file, "rb") as f: + for line in f: + if line.startswith(b"[Event "): + offsets.append(pos) + pos += len(line) + return offsets, pos # pos ends up as total file size + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("pgn_file") + ap.add_argument("out_dir") + ap.add_argument("--num-shards", type=int, default=8) + ap.add_argument("--extra-args", default="", + help="Extra args passed through to mine_king_mates.py, " + "e.g. '--control-ratio 1.0'") + args = ap.parse_args() + + os.makedirs(args.out_dir, exist_ok=True) + script_dir = os.path.dirname(os.path.abspath(__file__)) + mine_script = os.path.join(script_dir, "mine_king_mates.py") + + print("Scanning for game boundaries...", file=sys.stderr) + t0 = time.time() + offsets, file_size = find_game_offsets(args.pgn_file) + print(f"Found {len(offsets)} games, {file_size} bytes, " + f"in {time.time() - t0:.1f}s", file=sys.stderr) + + n = args.num_shards + games_per_shard = len(offsets) // n + shard_bounds = [] + for i in range(n): + start_idx = i * games_per_shard + end_idx = (i + 1) * games_per_shard if i < n - 1 else len(offsets) + byte_start = offsets[start_idx] + byte_end = offsets[end_idx] if end_idx < len(offsets) else file_size + shard_bounds.append((byte_start, byte_end)) + + procs = [] + for i, (byte_start, byte_end) in enumerate(shard_bounds): + out_csv = os.path.join(args.out_dir, f"shard_{i}.csv") + log_file = os.path.join(args.out_dir, f"shard_{i}.log") + cmd = [ + sys.executable, mine_script, args.pgn_file, out_csv, + "--byte-start", str(byte_start), + "--byte-end", str(byte_end), + ] + args.extra_args.split() + print(f"Launching shard {i}: bytes [{byte_start}, {byte_end}) -> {out_csv}", + file=sys.stderr) + with open(log_file, "w") as logf: + p = subprocess.Popen(cmd, stdout=logf, stderr=subprocess.STDOUT) + procs.append((i, p, out_csv)) + + failed = [] + for i, p, out_csv in procs: + rc = p.wait() + print(f"Shard {i} exited with code {rc}", file=sys.stderr) + if rc != 0: + failed.append(i) + + if failed: + print(f"FAILED shards: {failed} -- not merging, check logs in {args.out_dir}", + file=sys.stderr) + sys.exit(1) + + merged_path = os.path.join(args.out_dir, "merged.csv") + header = None + total_rows = 0 + with open(merged_path, "w", newline="") as out_f: + writer = csv.writer(out_f) + for i, _, out_csv in procs: + with open(out_csv, newline="") as in_f: + reader = csv.reader(in_f) + this_header = next(reader) + if header is None: + header = this_header + writer.writerow(header) + elif this_header != header: + print(f"WARNING: shard {i} header mismatch: {this_header}", + file=sys.stderr) + for row in reader: + writer.writerow(row) + total_rows += 1 + + print(f"Merged {total_rows} rows from {n} shards into {merged_path}", + file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/src/eval_tune/test_vs_head.sh b/src/eval_tune/test_vs_head.sh index d5d8006..871b801 100755 --- a/src/eval_tune/test_vs_head.sh +++ b/src/eval_tune/test_vs_head.sh @@ -1,11 +1,19 @@ #!/usr/bin/env bash -python3 ./match_play.py ../../head_reference/typhoon ../typhoon \ - --pgn ../../pgn/twic_filtered.pgn \ - --games 20000 \ - --workers 12 \ - --st 1 \ - --sprt --elo0 0 --elo1 5 \ - --scratch /usr/local/tmp/typhoon_match_overnight \ - --log overnight_match.log \ - --pgn-out ../self_play_games.pgn +set -e + +CANDIDATE=/tmp/typhoon/typhoon_match_play_candidate +HEAD=../../head_reference/typhoon + +cp -f ../typhoon ${CANDIDATE} +python3 ./match_play.py \ + ${HEAD} \ + ${CANDIDATE} \ + --pgn ../../pgn/twic_filtered.pgn \ + --games 20000 \ + --workers 12 \ + --st 1 \ + --sprt --elo0 0 --elo1 5 \ + --scratch /usr/local/tmp/typhoon_match_overnight \ + --log overnight_match.log \ + --pgn-out ../self_play_games.pgn diff --git a/src/eval_tune/train_king_safety_model.py b/src/eval_tune/train_king_safety_model.py new file mode 100644 index 0000000..012e636 --- /dev/null +++ b/src/eval_tune/train_king_safety_model.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Fit a logistic regression on king_features.csv's raw per-defect +breakdown against the mate/control label, to see whether _EvalKing's +defects should be weighted differently than the hand-picked constants +currently baked into KING_SAFETY_BY_COUNTER / KING_COUNTER_BY_ATTACK_PATTERN +/ EmptyAttackedSquare etc. See CLAUDE.md's king-safety discussion for the +full methodology this is built on. + +Deliberately excludes `countertotal` and `safetyvalue` from the feature +set -- those are the CURRENT engine's own derived output (the thing we're +trying to second-guess), not a raw defect. Using them as inputs would be +circular. `initloc` is kept as a feature (a compact king-square exposure +signal) since, unlike the other two, it doesn't already fold in this +sample's own defect counts. + +Attacker material confound (conversation context: _EvalKing scales its +final score by REDUCED_MATERIAL_DOWN_SCALER[armyscaler], but the raw +defect counts here are unscaled, and a real mate is naturally rarer under +a weak attacker regardless of what the defects say) is handled two ways: +included as a plain covariate (attackerarmyscaler), and separately +validated by refitting on the top-tercile-army-strength subset and +diffing coefficients against the full-population fit. + +Ring-square bucketing: slots 3/4 (the KingSafetyDeltas -1/+1 deltas -- the +two SAME-RANK side squares, not "front" despite an earlier mislabeling in +conversation) are kept individually resolved since they carried real, +stable signal in initial runs. The other six slots (the diagonal + +vertical squares one rank away) are collapsed into "front"/"back" bucket +sums, color-dependent per EmptyAttackedSquare's own existing asymmetry +(WHITE's front is slots 0/1/2, BLACK's is 5/6/7 -- see +KingRingBucket-equivalent logic below) -- these six raw per-slot +coefficients were the noisiest, most sign-flipping entries in every +full-vs-top-tercile drift table, and bucketing trades a little resolution +for a lot more stability at the sample sizes actually available here. +`filestorm` itself is dropped as a feature in favor of its two already- +independent sub-components (filestormopenfile, filestormpawn) which map +directly onto separate hand-picked tables in eval.c (KingFileDefects / +KingStormingPawnDefects) -- no reason to force a regression to re-derive +their relative importance from one pre-summed number. +""" +import argparse +import csv +import sys + +import numpy as np +from sklearn.linear_model import LogisticRegression +from sklearn.model_selection import train_test_split +from sklearn.preprocessing import StandardScaler +from sklearn.metrics import roc_auc_score, accuracy_score + +PIECE_NAMES = ("pawn", "minor", "rook", "queen") +PER_SQUARE_KEYS = ["offboard"] + [f"atk_{p}" for p in PIECE_NAMES] + \ + ["undef", "def", "xray", "flight"] + +SIDE_SLOTS = (3, 4) # kept individually resolved +# Color-dependent diagonal/vertical bucket membership, mirroring +# EmptyAttackedSquare's own asymmetry (chess.h: WHITE=1, BLACK=0). +BUCKET_SLOTS = { + "white": {"front": (0, 1, 2), "back": (5, 6, 7)}, + "black": {"front": (5, 6, 7), "back": (0, 1, 2)}, +} + +FEATURE_COLUMNS = [] +for _sq in SIDE_SLOTS: + for _k in PER_SQUARE_KEYS: + FEATURE_COLUMNS.append(f"sq{_sq}_{_k}") +for _bucket in ("front", "back"): + for _k in PER_SQUARE_KEYS: + FEATURE_COLUMNS.append(f"{_bucket}_{_k}") +FEATURE_COLUMNS += ["filestormopenfile", "filestormpawn", "queenproximity", + "initloc", "castled", "minorsathome", + "misscastleshort", "misscastlelong", "attackerarmyscaler"] + +# Reference-only columns: the engine's OWN current derived output for this +# position, kept for comparison but never fed into the model as inputs. +REFERENCE_COLUMNS = ["countertotal", "safetyvalue", "filestorm"] + + +def _row_to_features(r): + color = r["mated_color_or_sampled_color"] + bucket_slots = BUCKET_SLOTS[color] + out = [] + for sq in SIDE_SLOTS: + for k in PER_SQUARE_KEYS: + out.append(float(r[f"sq{sq}_{k}"])) + for bucket_name in ("front", "back"): + slots = bucket_slots[bucket_name] + for k in PER_SQUARE_KEYS: + # Sum across the 3 slots in this bucket -- offboard included, + # since a king near the edge legitimately has fewer on-board + # ring squares and that's itself signal, not noise to hide. + out.append(sum(float(r[f"sq{s}_{k}"]) for s in slots)) + for c in ["filestormopenfile", "filestormpawn", "queenproximity", + "initloc", "castled", "minorsathome", + "misscastleshort", "misscastlelong", "attackerarmyscaler"]: + out.append(float(r[c])) + return out + + +def load_data(csv_path): + with open(csv_path, newline="") as f: + reader = csv.DictReader(f) + rows = list(reader) + y = np.array([int(r["label"]) for r in rows], dtype=int) + X = np.array([_row_to_features(r) for r in rows], dtype=float) + ref = {c: np.array([float(r[c]) for r in rows]) for c in REFERENCE_COLUMNS} + return X, y, ref + + +def fit_and_report(X, y, label, top_n=25): + scaler = StandardScaler() + X_scaled = scaler.fit_transform(X) + X_train, X_test, y_train, y_test = train_test_split( + X_scaled, y, test_size=0.2, random_state=42, stratify=y) + + clf = LogisticRegression(max_iter=2000, C=1.0) + clf.fit(X_train, y_train) + + y_pred = clf.predict(X_test) + y_prob = clf.predict_proba(X_test)[:, 1] + acc = accuracy_score(y_test, y_pred) + auc = roc_auc_score(y_test, y_prob) + + print(f"\n=== {label} (n={len(y)}, train={len(y_train)}, test={len(y_test)}) ===") + print(f"Test accuracy: {acc:.4f} Test AUC: {auc:.4f}") + + # Standardized coefficients are directly comparable to each other + # (same units: change in log-odds per one std-dev of that feature) + # regardless of the feature's raw scale (a 0/1 flag vs. armyscaler's + # 0-62 range) -- that comparability is the entire point of this + # exercise, so report these, not raw unscaled coefficients. + coefs = clf.coef_[0] + order = np.argsort(-np.abs(coefs)) + print(f"\nTop {top_n} features by |standardized coefficient| " + f"(positive = more mate-predictive):") + for i in order[:top_n]: + print(f" {FEATURE_COLUMNS[i]:28s} {coefs[i]:+.4f}") + + return dict(zip(FEATURE_COLUMNS, coefs)), scaler, clf + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("features_csv") + ap.add_argument("--top-n", type=int, default=25) + args = ap.parse_args() + + X, y, ref = load_data(args.features_csv) + print(f"Loaded {len(y)} rows, {X.shape[1]} features, " + f"{y.sum()} positives / {(y == 0).sum()} controls", file=sys.stderr) + + print("\n--- Reference: current engine's own derived score ---") + for c in REFERENCE_COLUMNS: + pos_mean = ref[c][y == 1].mean() + ctl_mean = ref[c][y == 0].mean() + print(f" {c}: mean(mate)={pos_mean:.1f} mean(control)={ctl_mean:.1f} " + f"ratio={pos_mean / max(ctl_mean, 1e-9):.2f}x") + + full_coefs, _, _ = fit_and_report(X, y, "Full population", top_n=args.top_n) + + # Attacker-material confound check: refit on the top tercile of + # attacker army strength only, where REDUCED_MATERIAL_DOWN_SCALER is + # near its ceiling and roughly constant across the subset -- if + # coefficients move a lot vs. the full-population fit, the raw defect + # weights aren't material-invariant and the interaction should be + # modeled explicitly rather than left as a plain covariate. + army_idx = FEATURE_COLUMNS.index("attackerarmyscaler") + army_vals = X[:, army_idx] + top_tercile_cutoff = np.percentile(army_vals, 66.7) + mask = army_vals >= top_tercile_cutoff + print(f"\nTop-tercile army strength cutoff: attackerarmyscaler >= " + f"{top_tercile_cutoff:.0f} ({mask.sum()} rows)", file=sys.stderr) + strong_coefs, _, _ = fit_and_report( + X[mask], y[mask], "Top-tercile attacker army strength", top_n=args.top_n) + + print(f"\n=== Coefficient drift: full population vs. top-tercile army strength ===") + print(f"(large drift on a feature means its true weight is NOT " + f"material-invariant -- REDUCED_MATERIAL_DOWN_SCALER alone may " + f"not be enough to capture that interaction)") + drift = [(c, full_coefs[c], strong_coefs.get(c, 0.0), + abs(full_coefs[c] - strong_coefs.get(c, 0.0))) + for c in FEATURE_COLUMNS if c != "attackerarmyscaler"] + drift.sort(key=lambda t: -t[3]) + for c, fv, sv, d in drift[:args.top_n]: + print(f" {c:28s} full={fv:+.4f} top_tercile={sv:+.4f} drift={d:.4f}") + + +if __name__ == "__main__": + main() diff --git a/tests/ECM.md b/tests/ECM.md new file mode 100644 index 0000000..5ff1ed0 --- /dev/null +++ b/tests/ECM.md @@ -0,0 +1,246 @@ +# ECM position notes + +Per-position metadata accumulated while investigating solve-rate losses on +the curated suites (`ecm_ringers.ep_`, `ecm_confident_quick.ep_`, +`ecm_hard_quick.ep_`) during the 2026-09-10/11 extension-taper/qsearch-prune +session. IDs match the `id "ECM.###"` labels `script`/`suite_diff.pl` print +(note: due to how the `.ep_` files pipeline `id`/`go`/`setboard`/`solution` +-- each `id` line is followed by the *previous* block's `setboard`/ +`solution`, not the one textually after it -- so don't assume `grep`ing the +raw `.ep_` file for an id gives you the matching FEN without checking this). + +## Methodology + +A position's HEAD-vs-candidate solve flip was investigated by: + +1. Reading HEAD's own `sd10` log for that position: does the PV settle on + one move with a stable or improving score across iterations, ideally + with a `++`/`+` confirmed-fail-high mark? Or does it flip between + different candidate moves, land on an unconfirmed score at the very + last iteration (especially if it abandons an already-*confirmed* + alternative), or return a score suspiciously close to `0.00`/negative + for the side that's supposedly winning? +2. Where available, an independent second opinion: Crafty's own search + (`sd 15`-`sd 20`) on the same position, and/or typhoon-candidate's own + result at a deeper fixed depth (`sd 15`) than the suite's `sd 10`. +3. A position only counts as **legit** if HEAD's own read was solid *and* + nothing independent contradicts it. A single wobble or disagreement is + enough to call it **flaky**. + +This is a real, if informal, signal -- not a substitute for actually +re-running the suites when a change lands, just a way to avoid re-deriving +"is this specific solve-count delta noise" from scratch every time a +similar-shaped loss reappears (which it does -- most of the positions below +have shown up as a flip across *multiple*, otherwise-unrelated changes this +session). + +## Legit (real tactic, HEAD understood it solidly -- a genuine loss if lost) + +| ID | Suite | FEN | Solution | Why legit | +|---|---|---|---|---| +| ECM.035 | confident_quick | `2r2bk1/4qp2/3n2p1/2R1p1Np/2p1N3/r6P/1Q3PP1/3R2K1 w - -` | Rxc8 | HEAD confirms (`++`, +2.31) before finalizing at +2.79; Crafty's independent deep search converges to ~+4.6; candidate's own `sd 15` also confirms | +| ECM.080 | confident_quick | `5r1k/pb2r1bp/1p2B1p1/n7/4qNQ1/4B3/P4PPP/2RR2K1 w - -` | Rd4 | Single clean line, +3.26, fast solve, no oscillation | +| ECM.128 | hard_quick | `rn2qbr1/2p4k/p2p1nb1/1p1Pp2p/2P4P/2NBBPN1/PP1Q4/2KR2R1 w - -` | Nxh5 | Single clean line, +2.66, no oscillation | +| ECM.071 | hard_quick | `b1n1r1k1/2q2p1p/p4Pp1/1pBP4/4B1P1/2P1Q3/P4K1P/4R3 w - - 0 0` | Qh6 | Confirmed (`++`, +3.34) at depth 9, climbs monotonically to +4.74 by depth 12, same move throughout. Crafty independently finds the position winning even bigger (+7-9) but via a different move (`Kg2`, a quiet consolidation) -- doesn't contradict `Qh6`'s own soundness, just suggests it isn't the single objectively-best try. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.250 | confident_quick | `2r1rbk1/5pp1/bq5p/1pnBPN2/6Q1/N7/5PPP/R2R2K1 w - -` | Bxf7+ | Single clean line, +3.31, fast solve | +| ECM.251 | hard_quick | `1k1r1r2/2b2R2/Bppp2p1/2q4p/Q2pP3/P2P4/2P3PP/1R4K1 w - -` | Bb7 | Instant solve; Crafty's own deep search independently plays Bb7 too (+1.87) | +| ECM.445 | confident_quick | `7r/2p1q1k1/1p3r1p/p1pPpPp1/2P1P1P1/PP6/3Q2KR/7R w - -` | Qxg5+ | HEAD never solves this at `sd10` at all (only flips under `sn5m`); Crafty confirms the move is objectively good and stable (+2.31 to +2.35 across depths 18-20) | +| ECM.519 | confident_quick | `1k1r2r1/p1p3q1/2p2p2/3pn3/1P1b2bB/1QN5/P4PPP/1R2RBK1 b - -` | Nf3+ | Huge, unambiguous score (+7.33), no wavering at all; candidate's own deeper search confirms and strengthens it | +| ECM.147 | confident_quick | `r2b1rk1/5qp1/2p2P2/2p3Pp/p6P/2Q5/PPP5/2KR2R1 w - - 0 0` | g6 | HEAD's own line switches ideas mid-search (Qxc5 -> g6) but the new idea gets its own `++` confirmation (depth 10, +0.75) before the final unconfirmed bump to +0.89; Crafty independently plays the same `g6` and stably converges to +0.49-0.66 across depths 19-22. Thin margin, but move-choice agreement from an independent engine is the deciding evidence -- lost by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.672 | confident_quick | `5rk1/pbrn1ppp/1ppN1q2/2P1p3/3P4/1PRB4/P3QPPP/5RK1 w - - 0 0` | Bxh7+ | HEAD confirms (`++` twice, +4.09/+5.59) with the same first move stable from depth 7 on, score climbing monotonically to +6.22 at depth 10; Crafty independently plays the same move, reaching +9.5 by depth 27. Lost by the candidate LMR-graded-curve build (2026-09-11) -- notably, the *same candidate binary* run to deeper iterative depths (not suite-capped `sd10`) does eventually find `Bxh7+` too, jumping from +2.46 (depth 12, still on a `cxb6`-based line) to +7.59/+7.90 at depths 15-16. Not a permanent blind spot: the graded curve over-reduces this specific sacrifice line early enough to delay discovery by ~7-8 plies past what `sd10` allows, not miss it outright | +| ECM.721 | ringers | `4rrk1/2pn2pb/p1p1qp2/1pb1pN2/P3P1PN/1P1P4/1BP2PK1/R2Q3R w - -` | Nxg7 | Clean, unwavering, +3.19; candidate confirms/strengthens at deeper depth (+4.29) | +| ECM.733 | confident_quick | `2r3k1/pbq1np1p/1p1b1Bp1/8/6QP/2P2N2/B4PP1/4R1K1 w - -` | Bxf7+ | HEAD confirms once (`++`, +1.52); same first move stable across all entries, only the deeper continuation changes -- stronger signal than a plain unconfirmed line. Reconfirmed by Crafty (stable `Bxf7+`, +4.2-4.4, depths 18-22) during the 2026-09-11 LMR graded-curve investigation, which also lost this position | +| ECM.736 | confident_quick | `r2r2k1/1b2qpp1/1p2n3/p1b1p2p/4P1n1/PP1B2PP/1B1NQP2/R3NRK1 b - - 0 0` | Nxf2 | HEAD abandons a confirmed `Nd4` (`++`, +1.51) for `Nxf2`, unconfirmed and briefly dipping (+2.05 -> +1.93) -- looked flaky from HEAD's log alone. Crafty independently and stably plays the same `Nxf2` across 5 straight iterations (depths 24-28, -1.43 to -1.90 from Black's perspective), confirming the move choice despite HEAD's own wobble. Lost by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.734 | hard_quick | `r3r1k1/1bqn1ppp/1pp2p2/8/3P4/1B4N1/PP3PPP/R2QR1K1 w - -` | Bxf7+ | Same shape as ECM.733: confirmed once (`++`, +1.74), same top move stable throughout even as the deeper line's score settles lower | +| ECM.789 | hard_quick | `2r1r1k1/pb1n1pp1/1p1qpn1p/4N1B1/2PP4/3B4/P2Q1PPP/3RR1K1 w - - 0 0` | Bxh6 | Never gets an explicit `++` mark but the move is stable across the last 3 iterations with the score climbing (1.06 -> 1.27 -> 1.69). Crafty independently confirms the same move, also growing (1.89 -> 2.06 -> 2.13). Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.846 | hard_quick | `3qr1k1/rp3pbp/2p3p1/3p1b2/N1nP4/P2NP3/1PQ2PPP/1BRR2K1 b - -` | Rxe3 | **Reclassified from Flaky (2026-09-11)**: this entry previously read "two unconfirmed lines, both under +0.5" from an earlier build. The candidate LMR-graded-curve build, run to real (non-node-capped) depth, stabilizes firmly on `Rxe3` from depth 11 through 15 (+0.31 -> +0.67, no further wobble). Crafty independently and stably confirms the same move across depths 25-30 (-0.82 to -1.12, its own sign convention). Two-engine, deep, stable agreement overturns the old flaky call -- the earlier read was likely just a shallower/different build catching this position mid-wobble before it settles | +| ECM.735 | hard_quick | `r3r1k1/pp3ppp/8/1Nbp3q/3NnBn1/2P5/PPQ2PPP/3R1RK1 b - -` | Ngxf2 | HEAD's own line is clean and solid; Crafty's defense uses the *other* knight (Nexf2) but that's a "which piece" quibble, not a refutation of the idea | +| ECM.803 | hard_quick | `2b2r1k/2p3pp/2Nb4/pP5q/2PP4/P4pP1/5P1P/R1BQ2K1 b - -` | Bxg3 | HEAD's line is clean; Crafty independently finds **mate in 10** starting with Bxg3 -- about as solid as confirmation gets | +| ECM.442 | hard_quick | `4qn1k/1b3rpp/pb2pB2/1pp1P2P/3p1NQ1/P2P2PB/1PP4K/5R2 w - -` | Ng6+ | Not actually a miss -- the king-file-storm-defect candidate (2026-09-14) settles on `Bh4` at `sd10`, but its own deeper iterations (depth 14-15) find and `++`-confirm `Ng6+` (+1.94, climbing to +2.47 by depth 16). A depth-budget delay, not a blind spot; see the 2026-09-14 session note below | + +## Flaky / garbage (suite noise -- don't read a loss here as a real regression) + +| ID | Suite | FEN | "Solution" | Why flaky | +|---|---|---|---|---| +| ECM.213 | ringers | `k3r3/pR5p/PppR1p1p/4nP2/1PP5/8/4B1rP/2K5 w - -` | Re7 | Came off a draw-by-repetition line (`Kd1 ... <=>`) at only +0.37, unconfirmed | +| ECM.270 | ringers | `6k1/1p3p2/4r1p1/1PPrbq1p/2QN4/P2R1P1P/R4KP1/8 b - -` | Bg3+ | Score exactly `0.00`, single unconfirmed line; Crafty's own `sd20` search prefers Rxc5 instead | +| ECM.336 | hard_quick | `r1bqr1k1/pp3pp1/2p4p/3PN1n1/3P1b2/2NB4/PPQ2PPP/3R1RK1 b - -` | Rxe5 | Abandons a *twice*-confirmed cxd5 (swung -0.81 -> -0.16 -> +1.34) for an unconfirmed Rxe5 at the very end | +| ECM.337 | confident_quick | `r3r1k1/pbqn1ppp/1p3n2/1Pb5/2p5/5N1P/PBQ1BPP1/1N1R1RK1 b - -` | Rxe2 | Cycles through 3 different first moves within depth 10, lands unconfirmed | +| ECM.340 | confident_quick | `1k2r2r/1bpQN2p/1p6/P3p2q/4np2/2P2N2/2P2PPP/R3R1K1 b - -` | Rxe7 | Abandons a *thrice*-confirmed Qh6 (+2.49) for a near-identical unconfirmed Rxe7 (+2.76) | +| ECM.370 | hard_quick | `1r2r1k1/1pqb2pp/p2bp1n1/2p3BQ/3pN3/1P1P2P1/P1P3BP/4RRK1 w - -` | Bd8 | Score exactly `0.00` on a repetition line; Crafty prefers a different move at only +0.15 | +| ECM.443 | hard_quick | `2b5/2qrrpk1/5Rp1/2p4p/1pB1PR1P/1P1P2P1/5Q1K/8 w - -` | Qb2 | Two unconfirmed lines, thin margins (+0.92, +1.10) | +| ECM.637 | hard_quick | `r3qr1k/1b4pp/p2RQ3/Bp2p3/4p3/6P1/PPP2P1P/3R2K1 b - -` | Qh5 | Swapped a confirmed e3 (+2.18) for an unconfirmed Qh5 (+2.19) -- a coin-flip; Crafty's own eval also wobbled between -2.33 and -0.80 across nearby depths | +| ECM.715 | confident_quick | `3r1rk1/5pp1/pq1n1n1p/2pPR3/2B2P2/1PBQ2RP/P5P1/6K1 w - -` | Rxg7+ | Abandons a *twice*-confirmed Rge3 (+4.41) for a lower-scoring, unconfirmed Rxg7+ (+3.05) | +| ECM.771 | hard_quick | `3qr1k1/1br1bp1p/p3p1pB/1p1nN3/3PB3/7Q/PP3PPP/3RR1K1 w - -` | Nxf7 | HEAD never breaks +0.56, unconfirmed; Crafty's independent top choice is a different move near dead equal | +| ECM.813 | confident_quick | `r2q1rk1/pp2ppb1/3pn1pQ/3R4/2P3B1/4BR2/PP4PP/6K1 w - -` | Qxg6 | Both of HEAD's candidate lines score *negative* (-0.74, -0.69) -- HEAD's own eval doesn't think its own "solution" is good for the side supposedly winning | +| ECM.422 | hard_quick | `6k1/rnqb3p/5ppQ/2pPp3/p1N1P3/2PB3P/5PP1/1R4K1 w - - 0 0` | f4 | Same move (`f4`) held across 5 iterations but never gets a `++` confirm, thin margin (+0.70 to +0.95). Crafty independently prefers a different move (`Qh4`) at a similarly thin margin (+0.68 to +1.06) -- a genuine disagreement on a razor-thin position, not corroboration. Gained by the candidate LMR-graded-curve build (2026-09-11) but not a confident real solve | +| ECM.830 | confident_quick | `1n2r3/p1pq1kp1/1b1pNpp1/3P4/5RP1/3Q3P/1B3P2/6K1 w - -` | Bxf6 | Oscillates Nxg7/Bxf6 the whole way, no confirmation either side | +| ECM.842 | confident_quick | `r1b1k1nr/3n1p1p/1qpBp1p1/pp2b3/4PN2/PBN2Q2/1PP2PPP/2KR3R w kq - 0 0` | Nxe6 | HEAD abandons a confirmed `Bxe6` (`++`, +1.14) for an unconfirmed `Nxe6`, never reconfirmed. Crafty's own search at `sd 22` doesn't corroborate either -- wobbles between `Qg3` and `Bxe6` across depths 20-22, never playing `Nxe6`. Independent check does not support HEAD's answer, unlike ECM.147/672/736 below -- lost by the candidate LMR-graded-curve build (2026-09-11) but stays classified flaky | +| ECM.875 | hard_quick | `2kr1br1/ppp2p1p/2n1p1q1/2Nn4/3P4/1Q2PB2/PP1B1P1P/2R1K2R b K - 0 0` | Qg1+ | Score exactly `0.00` on a repetition line (`<=>`) across 6 straight iterations (depths 8-13) -- textbook flaky signature. Contradicts the "Gains seen" table's loose note calling this Crafty-confirmed legit in an earlier review; that note appears to have been wrong, or about a different search reaching a different (non-drawing) conclusion. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.879 | hard_quick | `r2r2k1/pp3ppp/2q5/2p5/2P1P1n1/P4P1b/1BQ1BP1P/R3R2K b - - 0 0` | Rd2 | Score exactly `0.00` on a repetition line (`<=>`) -- same signature as ECM.875. Gained by the candidate LMR-graded-curve build (2026-09-11) | +| ECM.866 | hard_quick | `7R/1p1r4/2b2p2/2Pp1qk1/3P1bp1/3N4/2Q2PB1/6K1 w - -` | Rg8+ | HEAD's own score is negative (-0.12 to -0.15); Crafty's independent top choice (Qd1) is also near-zero. **Note**: has shown up as a *gain* (candidate solves it, HEAD doesn't) in at least one later run too -- consistent with a genuinely coin-flip position, not a stable read either direction | + +## Gains seen (candidate solves, HEAD doesn't) -- not deeply vetted + +These showed up as improvements across various runs. Only spot-checked, not +put through the full methodology above -- listed for reference, not as a +confirmed-legit set the way the "Legit" table above is. + +| ID | Suite | Notes | +|---|---|---| +| ECM.193 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.476 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.544 | hard_quick | Crafty-confirmed legit in earlier review | +| ECM.613 | hard_quick | Seen as a gain across multiple runs | +| ECM.716 | confident_quick | Seen as a gain across multiple runs | +| ECM.779 | hard_quick | Seen as a gain across multiple runs | +| ECM.822 | hard_quick | Seen as a gain across multiple runs | +| ECM.875 | hard_quick | Seen as a gain across multiple runs | +| ECM.489 | hard_quick | Seen once | + +## 2026-09-11 LMR graded-curve session + +A separate investigation replaced typhoon's flat `-ONE_PLY` LMR reduction +with a graded depth/move-number curve (fit to this engine's own measured +tail-fail-high-rate data, not a borrowed formula -- see `dynamic.c`'s +`InitLMRTable`), plus a QSearch-drop guard in `GetLMRReduction` (never +reduce far enough to land below `THREE_QUARTERS_PLY` remaining depth). +Measured against `head_reference` (`ddf932b`) on `confident_quick`: +`sd10` net -4/+2 (net -3 across all three suites combined), `sn5000000` +net -3/+3 depth-adjusted (net +4 combined, `hard_quick` alone +7 with ++0.5 extra ply -- a real depth-payoff signal). + +Re-investigated all 6 `confident_quick` losses (identical set at both +`sd10` and `sn5m` -- a reproducible effect, not scattered noise) and both +gains against `head_reference`'s own log plus a fresh independent Crafty +check (`sd 22`, not run through the full methodology before this +session). Net result: **4 of 6 losses are legit** (733, 672, 147, 736 -- +147 and 736 were previously unclassified and looked flaky from HEAD's +log alone, but Crafty's independent move-choice agreement overturned +that read), only 2 are flaky (715, 842). Both gains (519, 035) are +already-established legit. That's a real, non-noise net-negative +tactical cost on `confident_quick` specifically -- the aggregate +suite-level numbers understated this at first glance (`sd10` -4 raw +looked mostly explainable as noise until the per-position Crafty check +was actually run). + +ECM.672 is worth remembering for future LMR work specifically: the same +candidate binary, given enough iterative depth (not suite-capped +`sd10`), *does* eventually find the correct `Bxh7+` sacrifice (depth +15-16, jumping from +2.46 to +7.59/+7.90) after several iterations +spent on an unrelated `cxb6`-based line. The graded curve isn't blind to +this tactic, it's delaying discovery by roughly 7-8 plies past what a +depth-10 budget allows -- a genuine cost at fixed/shallow search depths, +but not the kind of permanent blind spot a bad exemption rule would +produce. Worth checking whether this delay is concentrated in +sacrifice/attacking lines specifically (where the "safe to reduce" +heuristics this curve is built from may be least trustworthy) before +concluding the curve needs a different shape rather than just deeper +search to pay off. + +**`hard_quick`'s `sn5000000` gains investigated too** (11 gains, 4 losses +-- 193, 476, 779, 866, not re-investigated here since they're losses on +the suite that was already net-positive). Of the 11 gains: **7 are legit** +(071, 251, 734, 735, 789, 803, 846 -- 846 is a reclassification, see its +updated Legit-table entry above: an earlier build's shallower read +tagged it flaky, but this candidate's deeper search plus Crafty both +stably converge on the same move), **3 are flaky** (422 -- genuine +engine-vs-engine disagreement on a thin margin, not corroboration; 875, +879 -- both exact `0.00` on a repetition line), and **1 is genuinely +unresolved** (640): the candidate's own extended search twice-confirms +`Rxb2` at +3.66 (the `sn5m`-capped run's PV/played-move mismatch was +just the node budget cutting off *before* this switch, not a bug), but +Crafty independently searches to depth 30 and never once prefers +`Rxb2`, staying on a different move (`Bd8`) near dead equal. Two +real-depth searches disagreeing after real depth is a different, more +interesting kind of uncertainty than the usual flaky signature (score +exactly 0, unconfirmed thin margins) -- worth a dedicated line-by-line +look at the `Rxb2` continuation before counting it either way. + +**Combined verdict across both suites, real (non-flaky) positions only**: +9 legit gains (035, 519, 071, 251, 734, 735, 789, 803, 846) vs. 4 legit +losses (733, 672, 147, 736), with ECM.640 unresolved. A materially more +favorable picture than the raw suite aggregates suggested on their own +-- this is why the per-position Crafty-corroboration check matters more +than trusting `sd10`/`sn5000000` solve counts at face value, especially +at this suite size. + +## 2026-09-14 king-safety file-storm-defect session + +Retuned `eval.c`'s `KingFileDefects`/`KingStormingPawnDefects` tables +(inside `_ComputeFileStormDefects`) per `eval_tune/king_safety_data/HANDOFF.md`'s +tier-1 finding: `{+2,+1,+0}` -> `{+4,+2,+0}` and +`{+0,+4,+3,+1,+0,+0,+0,+0}` -> `{+0,+7,+5,+2,+0,+0,+0,+0}`, a real-game +mate-mining logistic regression's two most dominant, most stable +predictors, previously weighted the same as every other hand-picked +king-safety defect. Measured against `head_reference` (`ddf932b`) at +`sd10`: `ecm_ringers` -2, `ecm_confident_quick` -1, `ecm_hard_quick` -1. + +Per-position investigation (same methodology as the LMR session above) +found almost none of this is real: +- `ECM.715`, `ECM.866` (losses) and `ECM.813`, `ECM.443`, `ECM.771` + (gains) were already tagged flaky above. +- `ECM.716`, `ECM.193`, `ECM.476` (losses) were already listed in the + "Gains seen" table from other sessions -- a position that has already + flip-flopped direction before isn't new evidence of a regression. +- `ECM.501`, `ECM.801` (ringers losses, previously unclassified): both + show HEAD *itself* wobbling between candidate moves and landing on its + final answer only at the last iteration with no `++` confirmation + (`ECM.501`: Rf7 -> Rc8 -> Rf7 -> Rb8 -> Nxf3; `ECM.801`: Nd4 held + stably for 3 iterations then an unconfirmed jump to Nh6+ only at + depth 10) -- textbook flaky signature, not a real loss either + direction. +- `ECM.442` (hard_quick loss): not a miss at all -- see its updated + entry in the Legit table above. The candidate's own deeper iterations + (depth 14-15) find and `++`-confirm the same `Ng6+` HEAD plays at + `sd10`; the `sd10` "loss" is purely a depth-budget delay (climbing + from +0.91 at depth 10 to a confirmed +1.94 by depth 14), the same + shape as `ECM.672`'s entry above. +- `ECM.298` (hard_quick loss, previously unclassified): HEAD converges + cleanly on `Re6`/`Rf6` (+2.7 to +2.9) and the suite's labeled solution + is `Re6`, but the candidate instead finds and `++`-confirms `h4` + (+5.49 by depth 11, climbing to +8.87 by depth 17) -- and Crafty's own + independent deep search (`sd 27`, ~9.3s to switch onto it after + spending depths 20-23 stuck on `Rf6` at only +2.9-3.2) agrees `h4` is + objectively much stronger (+10.17) than the suite's labeled answer. + Notably the candidate found this line in 2.5s vs. Crafty's 9.3s. + Not a regression -- if anything, evidence the candidate is finding a + better resource than the suite's own labeled solution, faster than an + independent reference engine. Worth relabeling this suite's expected + answer if it's ever revisited, rather than trusting `Re6` as ground + truth. + +**Net read**: of the 4 nominal net solve losses (-2/-1/-1 across the +three suites), 7 of 9 individual position losses were already-known or +freshly-confirmed noise, one (`ECM.442`) is a depth-budget artifact, and +the last (`ECM.298`) is arguably a candidate improvement mislabeled as a +loss by the suite's fixed expected answer. No surviving evidence of a +real tactical regression from this change at the position level, despite +the suite-aggregate solve counts looking mildly negative -- another data +point for not trusting raw `sd10` solve deltas on suites this size +without the per-position check. + +Self-play (`match_play.py`, ongoing) at 110 games: -29 =47 +34 +(score 0.523, llr=+0.11, still far from either SPRT bound) -- too early +to call, but notably `avg depth: candidate=9.96 head=10.25` (candidate +searching ~0.3 ply shallower at the same `st` budget) with +`avg ebf: candidate=4.442 head=4.396` (branching factor slightly +*higher*, i.e. the tree got slightly fatter, not narrower) -- a +non-trivial shape change in the search tree from what was purely an +eval-weight retune around king safety, presumably via move-ordering/ +pruning decisions that read king danger differently now (e.g. anything +gated on `KING_SAFETY_BY_COUNTER`'s bucket or a null-move/LMR +verification threshold sensitive to eval score swings). Worth keeping +in mind for future king-safety retunes: even a "just the numbers" +change to an eval sub-term can visibly shift average search depth, not +just leaf-node scores. + +## Caveat + +This list is a snapshot from one session's worth of investigation, not a +permanent classification -- a position flagged "flaky" here reflects HEAD's +behavior *at the commit this session started from* (`9d1e32d`) plus +whatever candidate builds were tested through it. If the eval or search +changes significantly later, re-derive rather than trust this table blindly +-- especially the "Gains" table, which was never run through the full +methodology in the first place. |
