summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorScott Gasch <[email protected]>2026-09-13 23:18:07 -0700
committerScott Gasch <[email protected]>2026-09-13 23:18:07 -0700
commit4ff9170fd96e3da049c747bf5215e908a94ff145 (patch)
tree6c8e6f846b42518d84357bb3b83014feddc83202
parent78d1889fe38db1fd45b58509c8b3bd535ce691b7 (diff)
Add king-safety mate-mining regression pipeline; retune file-storm/
queen-proximity king-safety weights and add a back-flight bonus from it. Mines real human-game checkmates and matched sound-draw controls (eval_tune/mine_king_mates.py, shard_mine.py, extract_king_features.py, match_controls.py, train_king_safety_model.py), then fits a logistic regression against _EvalKing's raw per-defect breakdown (engine-side support: chess.h's KING_DEFECT_DUMP_INFO, eval.c's KING_DEFECT_DUMP instrumentation, command.c's evalkingdump command, all gated behind the KING_DEFECT_DUMP=1 build flag) to see which defects actually predict getting mated. See eval_tune/king_safety_data/HANDOFF.md for the full methodology and findings. Two rounds of retuning landed from this data so far, both validated via sd10 curated-suite comparison plus per-position Crafty corroboration (tests/ECM.md) and gated self-play: - KingFileDefects/KingStormingPawnDefects (_ComputeFileStormDefects) bumped ~1.75-2x relative to the rest of the ring-scan defects -- the regression's most dominant, most stable tier-1 predictors, previously weighted the same as every other hand-picked defect. - KING_QUEEN_PROXIMITY_DANGER rescaled from -8/count to -32/count, and a new KING_BACK_FLIGHT_BONUS (+1/square) term added for empty, unattacked ring squares on the side of the king away from the attacker -- a real, non-collinear protective signal the current EmptyUnattackedSquare table didn't credit at all. Both calibrated by unstandardizing the regression's coefficients and anchoring against KING_MISSING_ONE_CASTLE_OPTION's already-live -23 value (same direct-EVAL_TERM unit family). Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01P9EDZVcqu6E2bNRm4zDrTf
-rw-r--r--src/GNUmakefile3
-rwxr-xr-xsrc/chess.h67
-rw-r--r--src/command.c109
-rw-r--r--src/eval.c227
-rw-r--r--src/eval_tune/extract_king_features.py186
-rw-r--r--src/eval_tune/king_safety_data/HANDOFF.md254
-rw-r--r--src/eval_tune/match_controls.py168
-rw-r--r--src/eval_tune/mine_king_mates.py315
-rw-r--r--src/eval_tune/shard_mine.py110
-rwxr-xr-xsrc/eval_tune/test_vs_head.sh26
-rw-r--r--src/eval_tune/train_king_safety_model.py189
-rw-r--r--tests/ECM.md246
12 files changed, 1885 insertions, 15 deletions
diff --git a/src/GNUmakefile b/src/GNUmakefile
index e3ae855..3613e40 100644
--- a/src/GNUmakefile
+++ b/src/GNUmakefile
@@ -147,6 +147,9 @@ endif
ifdef CALIBRATE_POSITIONAL
PROFILE += -DCALIBRATE_POSITIONAL
endif
+ifdef KING_DEFECT_DUMP
+PROFILE += -DKING_DEFECT_DUMP
+endif
ifdef CALIBRATE_BASE_MARGIN
PROFILE += -DCALIBRATE_BASE_MARGIN
endif
diff --git a/src/chess.h b/src/chess.h
index 2681701..df59626 100755
--- a/src/chess.h
+++ b/src/chess.h
@@ -1367,6 +1367,70 @@ typedef struct _KING_SAFETY_HASH_ENTRY
}
KING_SAFETY_HASH_ENTRY;
+#ifdef KING_DEFECT_DUMP
+// Ring-slot indexing matches KingSafetyDeltas' order in eval.c exactly
+// (-17,-16,-15,-1,+1,+15,+16,+17): slots 0-7, no bucketing -- kept at
+// full per-square resolution rather than collapsed into front/side/back,
+// since the direct in-front square (delta +-16) and the two front-
+// diagonal squares (+-15/+-17) are genuinely different attack geometries
+// (e.g. a same-file rook vs. a diagonal bishop) that a bucket would
+// conflate. Sample size (tens of thousands of mined positions) has no
+// trouble supporting this many features.
+#define KING_RING_NUM_SLOTS 8
+
+#define KING_RING_PAWN 0
+#define KING_RING_MINOR 1
+#define KING_RING_ROOK 2
+#define KING_RING_QUEEN 3
+#define KING_RING_NUM_PIECE_TYPES 4
+
+// Raw, per-defect, per-square breakdown of _EvalKing's ring-square scan,
+// filled in (only under this build flag) instead of/alongside the normal
+// collapse into one scalar uCounter -- feeds a per-defect regression
+// against real mate outcomes instead of trusting the hand-picked weights
+// already baked into KING_SAFETY_BY_COUNTER / KING_COUNTER_BY_ATTACK_PATTERN.
+// See CLAUDE.md's king-safety discussion.
+typedef struct _KING_DEFECT_DUMP_INFO
+{
+ // [slot][piece type]: 1 if this ring square is attacked or x-rayed by
+ // this piece type, same bits _EvalKing's own bvAttack/bvXray already
+ // compute per square -- just retained per-slot instead of OR'd
+ // together into one whole-ring bvPattern.
+ FLAG fAttackedByPieceType[KING_RING_NUM_SLOTS][KING_RING_NUM_PIECE_TYPES];
+ // [slot]: off-board squares (near an edge/corner) leave all four FALSE.
+ FLAG fOnBoard[KING_RING_NUM_SLOTS];
+ FLAG fAttackedUndefended[KING_RING_NUM_SLOTS];
+ FLAG fAttackedDefended[KING_RING_NUM_SLOTS];
+ FLAG fXrayedUnattacked[KING_RING_NUM_SLOTS];
+ FLAG fEmptyFlight[KING_RING_NUM_SLOTS];
+ ULONG uFileStormDefects;
+ // Split-out sub-components of uFileStormDefects, from two already-
+ // separate hand-picked tables (KingFileDefects / KingStormingPawnDefects)
+ // -- kept apart so a regression can weigh "open/half-open file near
+ // the king" independently from "how far an enemy pawn has advanced up
+ // that file", rather than only ever seeing their pre-summed total.
+ ULONG uFileStormOpenFile;
+ ULONG uFileStormStormingPawn;
+ ULONG uQueenNearKing;
+ ULONG uInitialCounterByLocation;
+ ULONG uCounterTotal; // final uCounter, pre-clamp
+ ULONG uSafetyTableValue; // KING_SAFETY_BY_COUNTER[MINU(uCounterTotal,41)]
+ // Attacker's (xColor's) army strength -- pos->uArmyScaler[xColor],
+ // the exact index REDUCED_MATERIAL_DOWN_SCALER itself uses to scale
+ // the final king-safety score. Captured so training can control for
+ // it directly (a defect pattern under a weak attacker converts to
+ // mate far less often than the identical pattern under a full
+ // attacking army, regardless of what the raw defect counts say) --
+ // see CLAUDE.md's king-safety discussion.
+ ULONG uAttackerArmyScaler;
+ FLAG fCastled;
+ ULONG uMinorsAtHome;
+ FLAG fMissingCastleShort;
+ FLAG fMissingCastleLong;
+}
+KING_DEFECT_DUMP_INFO;
+#endif
+
#define NUM_SPLIT_PTRS_IN_CONTEXT (8)
//
@@ -1450,6 +1514,9 @@ typedef struct _SEARCHER_THREAD_CONTEXT
// rejected), so there's no risk of runaway nested diagnosis.
FLAG fDiagUnprunedSubtree;
#endif
+#ifdef KING_DEFECT_DUMP
+ KING_DEFECT_DUMP_INFO sKingDefectDump[2]; // filled by _EvalKing, indexed by color
+#endif
CHAR szLastPV[SMALL_STRING_LEN_CHAR];
}
SEARCHER_THREAD_CONTEXT;
diff --git a/src/command.c b/src/command.c
index e6bf9fc..1696690 100644
--- a/src/command.c
+++ b/src/command.c
@@ -547,6 +547,107 @@ Return value:
}
+#ifdef KING_DEFECT_DUMP
+COMMAND(EvalKingDumpCommand)
+/**
+
+Routine description:
+
+ This function implements the 'evalkingdump' engine command, only
+ built under KING_DEFECT_DUMP. Runs a full static eval (same
+ persistent-context pattern as EvalCommand) then prints one
+ space-separated key=value line per side with _EvalKing's raw,
+ per-defect breakdown (chess.h's KING_DEFECT_DUMP_INFO) instead of
+ the single collapsed score -- feeds an offline regression against
+ real mate outcomes. See CLAUDE.md's king-safety discussion.
+
+ Usage:
+
+ evalkingdump
+
+Parameters:
+
+ The COMMAND macro hides four arguments from the input parser:
+
+ CHAR *szInput : the full line of input
+ ULONG argc : number of argument chunks
+ CHAR *argv[] : array of ptrs to each argument chunk
+ POSITION *pos : a POSITION pointer to operate on
+
+Return value:
+
+ void
+
+**/
+{
+ static SEARCHER_THREAD_CONTEXT *ctx = NULL;
+ static ULONG uLastSeenDnaGeneration = (ULONG)-1;
+ static const CHAR *szPieceName[KING_RING_NUM_PIECE_TYPES] =
+ { "pawn", "minor", "rook", "queen" };
+ ULONG uColor;
+ ULONG uSlot, uPieceType;
+ KING_DEFECT_DUMP_INFO *p;
+
+ if (NULL == ctx)
+ {
+ ctx = SystemAllocateMemory(sizeof(SEARCHER_THREAD_CONTEXT));
+ if (NULL == ctx)
+ {
+ Trace("Out of memory.\n");
+ return;
+ }
+ InitializeSearcherContext(pos, ctx);
+ uLastSeenDnaGeneration = g_uDnaGeneration;
+ }
+ else if (uLastSeenDnaGeneration != g_uDnaGeneration)
+ {
+ InitializeSearcherContext(pos, ctx);
+ uLastSeenDnaGeneration = g_uDnaGeneration;
+ }
+ else
+ {
+ ReInitializeSearcherContext(pos, ctx);
+ }
+ (void)Eval(ctx, -INFINITY, +INFINITY, NULL);
+
+ for (uColor = 0; uColor < 2; uColor++)
+ {
+ p = &ctx->sKingDefectDump[uColor];
+ Trace("KINGDEFECTDUMP color=%s", COLOR_NAME(uColor));
+ for (uSlot = 0; uSlot < KING_RING_NUM_SLOTS; uSlot++)
+ {
+ if (!p->fOnBoard[uSlot])
+ {
+ Trace(" sq%u_offboard=1", uSlot);
+ continue;
+ }
+ for (uPieceType = 0; uPieceType < KING_RING_NUM_PIECE_TYPES; uPieceType++)
+ {
+ Trace(" sq%u_atk_%s=%u",
+ uSlot, szPieceName[uPieceType],
+ p->fAttackedByPieceType[uSlot][uPieceType]);
+ }
+ Trace(" sq%u_undef=%u sq%u_def=%u sq%u_xray=%u sq%u_flight=%u",
+ uSlot, p->fAttackedUndefended[uSlot],
+ uSlot, p->fAttackedDefended[uSlot],
+ uSlot, p->fXrayedUnattacked[uSlot],
+ uSlot, p->fEmptyFlight[uSlot]);
+ }
+ Trace(" filestorm=%u filestormopenfile=%u filestormpawn=%u "
+ "queenproximity=%u initloc=%u countertotal=%u "
+ "safetyvalue=%u attackerarmyscaler=%u castled=%u minorsathome=%u "
+ "misscastleshort=%u misscastlelong=%u\n",
+ p->uFileStormDefects, p->uFileStormOpenFile, p->uFileStormStormingPawn,
+ p->uQueenNearKing,
+ p->uInitialCounterByLocation, p->uCounterTotal,
+ p->uSafetyTableValue, p->uAttackerArmyScaler,
+ p->fCastled, p->uMinorsAtHome,
+ p->fMissingCastleShort, p->fMissingCastleLong);
+ }
+}
+#endif
+
+
COMMAND(QSearchCommand)
/**
@@ -2454,6 +2555,14 @@ COMMAND_PARSER_ENTRY g_ParserTable[] =
FALSE,
FALSE,
"Run a static eval" },
+#ifdef KING_DEFECT_DUMP
+ { "evalkingdump",
+ EvalKingDumpCommand,
+ FALSE,
+ FALSE,
+ FALSE,
+ "Dump _EvalKing's raw per-defect breakdown for both sides" },
+#endif
{ "evaldna",
EvalDnaCommand,
FALSE,
diff --git a/src/eval.c b/src/eval.c
index 0a8f71e..709968a 100644
--- a/src/eval.c
+++ b/src/eval.c
@@ -712,13 +712,44 @@ static SCORE KING_SAFETY_BY_COUNTER[42] =
// see the 180-position king-safety-term distribution sampled
// 2026-08-30 (median -27, p75 -16, only ~1% worse than -200) for the
// scale this needs to slot into.
+// 2026-09-14: rescaled from the original -8/count step (see the
+// 2026-08-30 comment above -- that step was calibrated against a
+// *removed* legacy term's own magnitude, not against real mate data).
+// king_safety_data/HANDOFF.md's mate-mining logistic regression found
+// `queenproximity` (after folding in the collinear front/back/sq3/sq4
+// atk_queen sub-features it was absorbing -- see
+// eval_tune/king_safety_data/HANDOFF.md's 2026-09-14 addendum) has a
+// per-unit raw coefficient of +1.00, calibrated against
+// `misscastleshort`'s own cleaned coefficient (+0.71) and this table's
+// sibling constant KING_MISSING_ONE_CASTLE_OPTION's already-live -23
+// value (same direct-EVAL_TERM unit family, unlike the file-storm
+// tables below which feed the separate uCounter/KING_SAFETY_BY_COUNTER
+// lookup and need their own, different calibration) -- see
+// (misscastleshort_live / misscastleshort_raw_coef) = k, then
+// k * queenproximity_raw_coef =~ -32/count. Validate via the usual
+// sd10 + self-play gate before trusting this magnitude.
static SCORE KING_QUEEN_PROXIMITY_DANGER[7] =
-{// 0 1 2 3 4 5 6
- +0, -8, -16, -24, -32, -40, -48
+{// 0 1 2 3 4 5 6
+ +0, -32, -64, -96, -128, -160, -192
};
static SCORE KING_MISSING_ONE_CASTLE_OPTION = -23;
+// 2026-09-14: new term, not a retune of an existing one.
+// king_safety_data/HANDOFF.md's regression found `back_flight` (an
+// empty, unattacked ring square on the side of the king AWAY from the
+// attacker) is a real, non-collinear PROTECTIVE signal (negative raw
+// coefficient -0.035, i.e. more open escape squares behind the king
+// predicts LESS mate risk) -- unlike `front_flight`, which is a real
+// danger signal already captured by EmptyUnattackedSquare's front
+// weight. Calibrated the same way as KING_QUEEN_PROXIMITY_DANGER above
+// (misscastleshort-anchored k =~ -32.4, applied to back_flight's raw
+// coefficient) =~ +1.1/square, rounded to +1. Small and deliberately
+// so -- this is the single most novel, least-collinear finding from
+// the regression, worth landing and validating in isolation before
+// combining with anything else.
+static SCORE KING_BACK_FLIGHT_BONUS = 1;
+
// Flat "this is bad" flags for _EvalTrappedPieces -- not an attempt to
// price the material outcome (search resolves that), just a nudge away
// from positions with a piece that looks stuck. ENPRISE_AND_TRAPPED is
@@ -788,7 +819,8 @@ static DNA_BASE_SIZE g_EvalDNA[] = {
DNA_ARRAY(KING_TO_CENTER),
DNA_ARRAY(KING_SAFETY_BY_COUNTER),
DNA_ARRAY(KING_QUEEN_PROXIMITY_DANGER),
- DNA_VAR(KING_MISSING_ONE_CASTLE_OPTION)
+ DNA_VAR(KING_MISSING_ONE_CASTLE_OPTION),
+ DNA_VAR(KING_BACK_FLIGHT_BONUS)
};
ULONG
@@ -2402,13 +2434,19 @@ Return value:
**/
{
+ // 2026-09-14: scaled up ~1.75-2x relative to the other _EvalKing
+ // defect tables (king_safety_data/HANDOFF.md, tier 1 -- these two
+ // came back as the dominant, most stable predictors of a real mate
+ // by a clear margin over every other defect, but previously shared
+ // the same +0..+4 magnitude as everything else). Keep this table in
+ // sync with _ComputeFileStormDefectsSplit's copy below.
static ULONG KingStormingPawnDefects[8] =
{
- +0, +4, +3, +1, +0, +0, +0, +0
+ +0, +7, +5, +2, +0, +0, +0, +0
};
static ULONG KingFileDefects[3] =
{
- +2, +1, +0
+ +4, +2, +0
};
ULONG ufColor = FLIP(uColor);
COOR c = pos->cNonPawns[uColor][0];
@@ -2449,6 +2487,79 @@ Return value:
}
+#ifdef KING_DEFECT_DUMP
+static void
+_ComputeFileStormDefectsSplit(IN POSITION *pos,
+ IN const PAWN_HASH_ENTRY *pHash,
+ IN ULONG uColor,
+ OUT ULONG *puOpenFile,
+ OUT ULONG *puStormingPawn)
+/**
+
+Routine description:
+
+ Debug-only sibling of _ComputeFileStormDefects that reports its two
+ independent sub-ideas (open/half-open file bonus vs. storming-pawn
+ distance penalty) separately instead of pre-summed, so a regression
+ can weigh them independently -- they already live in two distinct
+ hand-picked tables (KingFileDefects / KingStormingPawnDefects) with
+ no structural change needed to retune them individually once this
+ tells us their relative importance. See CLAUDE.md's king-safety
+ discussion. Logic intentionally identical to _ComputeFileStormDefects,
+ just split into two accumulators -- keep both in sync if either
+ changes.
+
+**/
+{
+ // Keep in sync with _ComputeFileStormDefects's copy above.
+ static ULONG KingStormingPawnDefects[8] =
+ {
+ +0, +7, +5, +2, +0, +0, +0, +0
+ };
+ static ULONG KingFileDefects[3] =
+ {
+ +4, +2, +0
+ };
+ ULONG ufColor = FLIP(uColor);
+ COOR c = pos->cNonPawns[uColor][0];
+ COOR cFileSq, cSquare;
+ ULONG u, v;
+ BITBOARD bb;
+ int w;
+
+ *puOpenFile = 0;
+ *puStormingPawn = 0;
+ for (w = -1; w <= 1; w++)
+ {
+ cFileSq = c + w;
+ if (!IS_ON_BOARD(cFileSq))
+ {
+ continue;
+ }
+ u = FILE(cFileSq) + 1;
+ v = (pHash->uCountPerFile[WHITE][u] > 0) +
+ (pHash->uCountPerFile[BLACK][u] > 0);
+ ASSERT((v >= 0) && (v <= 2));
+ *puOpenFile += (KingFileDefects[v] + ((v < 2) && ((u == 1) || (u == 8))));
+ bb = pos->bbPawns[ufColor] & BBFILE[u - 1];
+ if (bb)
+ {
+ if (uColor == WHITE)
+ {
+ cSquare = CoorFromBitBoardRank1ToRank8(&bb);
+ }
+ else
+ {
+ cSquare = CoorFromBitBoardRank8ToRank1(&bb);
+ }
+ ASSERT(IS_ON_BOARD(cSquare));
+ *puStormingPawn += KingStormingPawnDefects[DISTANCE(cSquare, c)];
+ }
+ }
+}
+#endif
+
+
static ULONG
_GetFileStormDefects(IN SEARCHER_THREAD_CONTEXT *ctx,
IN POSITION *pos,
@@ -4121,6 +4232,7 @@ Return value:
BITV bvDefend;
BITV bvPattern = 0;
ULONG uFlightSquares;
+ ULONG uBackFlightSquares;
ULONG uQueenNearKing = 0;
BITBOARD bb;
SCORE i;
@@ -4147,6 +4259,21 @@ Return value:
+15, +16, +17, 0
};
+ // 2026-09-14: color-dependent front/back ring-slot classification,
+ // mirroring EmptyAttackedSquare/EmptyUnattackedSquare's own existing
+ // asymmetry (WHITE's front is slots 0/1/2, BLACK's is 5/6/7) and
+ // matching eval_tune/train_king_safety_model.py's BUCKET_SLOTS --
+ // used to isolate the back-bucket flight-square bonus below. Slots
+ // 3/4 (the same-rank side squares) are neither front nor back, same
+ // as the regression's own treatment.
+ static const FLAG KING_RING_IS_BACK[2][8] =
+ {
+ // BLACK: back = slots 0,1,2 (front = 5,6,7)
+ { TRUE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, FALSE },
+ // WHITE: back = slots 5,6,7 (front = 0,1,2)
+ { FALSE, FALSE, FALSE, FALSE, FALSE, TRUE, TRUE, TRUE },
+ };
+
static ULONG EmptyAttackedSquare[2][8] =
{
{
@@ -4181,6 +4308,10 @@ Return value:
2
};
+#ifdef KING_DEFECT_DUMP
+ KING_DEFECT_DUMP_INFO *pDump;
+#endif
+
ASSERT(IS_ON_BOARD(c));
p = pos->rgSquare[c].pPiece;
ASSERT(p && IS_KING(p));
@@ -4188,6 +4319,10 @@ Return value:
xColor = FLIP(uColor);
ASSERT(IS_VALID_COLOR(uColor));
pos->bbKingAttacks[uColor] |= g_KingAttacksBB[c];
+#ifdef KING_DEFECT_DUMP
+ pDump = &ctx->sKingDefectDump[uColor];
+ memset(pDump, 0, sizeof(KING_DEFECT_DUMP_INFO));
+#endif
// Don't bother with king safety when the other side has no mating
// material.
@@ -4201,12 +4336,16 @@ Return value:
// Prepare to do king safety
uCounter = (u >= KEEP_KING_AT_HOME_THRESHOLD) *
KING_INITIAL_COUNTER_BY_LOCATION[uColor][c];
+#ifdef KING_DEFECT_DUMP
+ pDump->uInitialCounterByLocation = uCounter;
+#endif
#ifdef EVAL_DUMP
Trace("%s Initial KS Counter: %u\n", COLOR_NAME(uColor), uCounter);
#endif
BITBOARD bbEnemyOcc = _BuildFriendlySideBB(pos, xColor);
uFlightSquares = 0;
+ uBackFlightSquares = 0;
u = 0;
ASSERT(KingSafetyDeltas[u] != 0);
do
@@ -4264,6 +4403,54 @@ Return value:
//
uQueenNearKing += (((bvAttack | bvXray) & QUEEN_BIT) != 0);
+#ifdef KING_DEFECT_DUMP
+ {
+ BITV bvEither = bvAttack | bvXray;
+ ASSERT(u < KING_RING_NUM_SLOTS);
+ pDump->fOnBoard[u] = TRUE;
+ pDump->fAttackedByPieceType[u][KING_RING_PAWN] =
+ ((bvEither & PAWN_BIT) != 0);
+ pDump->fAttackedByPieceType[u][KING_RING_MINOR] =
+ ((bvEither & MINOR_BIT) != 0);
+ pDump->fAttackedByPieceType[u][KING_RING_ROOK] =
+ ((bvEither & ROOK_BIT) != 0);
+ pDump->fAttackedByPieceType[u][KING_RING_QUEEN] =
+ ((bvEither & QUEEN_BIT) != 0);
+ if (bvAttack != 0)
+ {
+ // NB: bvDefend here still includes the own-king bit
+ // that the real code strips below via bvDefend &=
+ // ~KING_BIT once xrayed/multiply-attacked -- match
+ // that exact "is this square adequately defended"
+ // semantics rather than the raw pre-strip value.
+ BITV bvDefendForDump = bvDefend;
+ if (bvXray || (bvAttack & (bvAttack - 1)))
+ {
+ bvDefendForDump &= ~KING_BIT;
+ }
+ if (bvDefendForDump == 0)
+ {
+ pDump->fAttackedUndefended[u] = TRUE;
+ }
+ else
+ {
+ pDump->fAttackedDefended[u] = TRUE;
+ }
+ }
+ else
+ {
+ if (bvXray != 0)
+ {
+ pDump->fXrayedUnattacked[u] = TRUE;
+ }
+ if (!(pos->bbOccupied & sq))
+ {
+ pDump->fEmptyFlight[u] = TRUE;
+ }
+ }
+ }
+#endif
+
//
// Tried a branchless rewrite of this attacked/unattacked
// split (2026-09-09): compute both sides' contributions
@@ -4303,10 +4490,12 @@ Return value:
}
else
{
+ FLAG fEmpty = ((pos->bbOccupied & sq) == 0);
uCounter += (bvXray != 0);
ASSERT(u < 8);
uCounter += EmptyUnattackedSquare[uColor][u];
- uFlightSquares += ((pos->bbOccupied & sq) == 0);
+ uFlightSquares += fEmpty;
+ uBackFlightSquares += (fEmpty && KING_RING_IS_BACK[uColor][u]);
}
}
u++;
@@ -4327,6 +4516,12 @@ Return value:
// isolated sd10 suite testing before that landing).
//
uCounter += _GetFileStormDefects(ctx, pos, pHash, uColor);
+#ifdef KING_DEFECT_DUMP
+ pDump->uFileStormDefects = _GetFileStormDefects(ctx, pos, pHash, uColor);
+ _ComputeFileStormDefectsSplit(pos, pHash, uColor,
+ &pDump->uFileStormOpenFile,
+ &pDump->uFileStormStormingPawn);
+#endif
#ifdef EVAL_DUMP
Trace("%s KS Counter post-open file/stormers: %u\n", COLOR_NAME(uColor),
uCounter);
@@ -4351,6 +4546,14 @@ Return value:
// are scaling it based on the _other_ side's material.
uCounter = MINU(uCounter, 41);
i = KING_SAFETY_BY_COUNTER[uCounter];
+#ifdef KING_DEFECT_DUMP
+ pDump->uQueenNearKing = uQueenNearKing;
+ pDump->uCounterTotal = uCounter;
+ pDump->uSafetyTableValue = (ULONG)(-i);
+ pDump->fCastled = pos->fCastled[uColor];
+ pDump->uMinorsAtHome = pos->uMinorsAtHome[uColor];
+ pDump->uAttackerArmyScaler = pos->uArmyScaler[xColor];
+#endif
#ifdef CALIBRATE_POSITIONAL
// KING_SAFETY_BY_COUNTER is <= 0 throughout (a penalty); the
// calibration histogram bins magnitude, matching iKingSwingP90's
@@ -4375,6 +4578,14 @@ Return value:
KING_QUEEN_PROXIMITY_DANGER[MINU(uQueenNearKing, 6)],
"queen proximity danger");
+ ASSERT(uBackFlightSquares <= 3);
+ EVAL_TERM(uColor,
+ KING,
+ c,
+ iKingScore,
+ (SCORE)(KING_BACK_FLIGHT_BONUS * uBackFlightSquares),
+ "back rank flight bonus");
+
// Bonus for castling / penalty for loss of castle. Also, if side has
// not yet castled, be concerned with undeveloped minor pieces.
if (FALSE == pos->fCastled[uColor])
@@ -4415,6 +4626,10 @@ Return value:
((CASTLE_LONG[uColor] & pos->bvCastleInfo) == 0) *
(SCORE)uDevelopmentScaler) / 8),
"can't castle long");
+#ifdef KING_DEFECT_DUMP
+ pDump->fMissingCastleShort = ((CASTLE_SHORT[uColor] & pos->bvCastleInfo) == 0);
+ pDump->fMissingCastleLong = ((CASTLE_LONG[uColor] & pos->bvCastleInfo) == 0);
+#endif
}
}
diff --git a/src/eval_tune/extract_king_features.py b/src/eval_tune/extract_king_features.py
new file mode 100644
index 0000000..509f823
--- /dev/null
+++ b/src/eval_tune/extract_king_features.py
@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+"""Feature extraction driver: takes the mined FENs in mate_training_data.csv
+and runs each through the engine's `evalkingdump` command (KING_DEFECT_DUMP
+build only) to pull _EvalKing's raw per-defect breakdown, joining it back
+onto each row's label/metadata to produce the final regression-ready CSV.
+
+Sharded across N engine subprocesses (one `script` file each, run in
+parallel) since 86k FENs through a single serial engine invocation would
+be needlessly slow on this 24-core box -- same reasoning as
+shard_mine.py's PGN sharding.
+
+Each engine invocation processes its shard via `script <file>` containing
+alternating `setboard <fen>` / `evalkingdump` lines; evalkingdump always
+emits exactly two `KINGDEFECTDUMP color=...` lines (black then white, in
+that fixed loop order) per position, unconditionally -- so the Nth pair
+of KINGDEFECTDUMP lines in the shard's output corresponds to the Nth row
+of that shard's input, and we pick out whichever color each row's
+"mated_color_or_sampled_color" column names.
+"""
+import argparse
+import csv
+import os
+import re
+import subprocess
+import sys
+import time
+
+KV_RE = re.compile(r"(\S+)=(\S+)")
+
+# Fixed schema, since an off-board ring square (king near an edge/corner)
+# emits only "sqN_offboard=1" while an on-board square emits 9 separate
+# keys -- different king positions otherwise produce entirely different
+# column sets, which would silently misalign rows if we inferred columns
+# from whichever row happened to be parsed first (caught by a per-shard
+# "feature key mismatch" warning during testing).
+PIECE_NAMES = ("pawn", "minor", "rook", "queen")
+PER_SQUARE_ON_BOARD_KEYS = [f"atk_{p}" for p in PIECE_NAMES] + \
+ ["undef", "def", "xray", "flight"]
+GLOBAL_KEYS = ["filestorm", "filestormopenfile", "filestormpawn",
+ "queenproximity", "initloc", "countertotal",
+ "safetyvalue", "attackerarmyscaler", "castled",
+ "minorsathome", "misscastleshort", "misscastlelong"]
+
+FEATURE_KEYS = []
+for _sq in range(8):
+ FEATURE_KEYS.append(f"sq{_sq}_offboard")
+ for _k in PER_SQUARE_ON_BOARD_KEYS:
+ FEATURE_KEYS.append(f"sq{_sq}_{_k}")
+FEATURE_KEYS += GLOBAL_KEYS
+
+
+def parse_kingdefectdump_line(line):
+ """Returns (color, {key: value_str}) from a 'KINGDEFECTDUMP color=... ...'
+ line, normalized to FEATURE_KEYS' fixed schema -- off-board squares get
+ offboard=1 and all per-square on-board keys defaulted to 0 rather than
+ left absent."""
+ parts = line.strip().split(None, 1)
+ assert parts[0] == "KINGDEFECTDUMP"
+ raw_kv = dict(KV_RE.findall(parts[1]))
+ color = raw_kv.pop("color")
+
+ kv = {}
+ for sq in range(8):
+ offboard_key = f"sq{sq}_offboard"
+ if offboard_key in raw_kv:
+ kv[offboard_key] = raw_kv[offboard_key]
+ for k in PER_SQUARE_ON_BOARD_KEYS:
+ kv[f"sq{sq}_{k}"] = "0"
+ else:
+ kv[offboard_key] = "0"
+ for k in PER_SQUARE_ON_BOARD_KEYS:
+ full_key = f"sq{sq}_{k}"
+ kv[full_key] = raw_kv[full_key]
+ for k in GLOBAL_KEYS:
+ kv[k] = raw_kv[k]
+ assert set(kv.keys()) == set(FEATURE_KEYS), \
+ f"parsed keys don't match fixed schema: {set(kv) ^ set(FEATURE_KEYS)}"
+ return color, kv
+
+
+def run_shard(engine_path, rows, shard_dir, shard_idx):
+ script_path = os.path.join(shard_dir, f"extract_{shard_idx}.script")
+ out_log = os.path.join(shard_dir, f"extract_{shard_idx}.out")
+ with open(script_path, "w") as f:
+ f.write("force\n")
+ f.write("nopost\n")
+ for row in rows:
+ f.write(f"setboard {row['fen']}\n")
+ f.write("evalkingdump\n")
+
+ cmd = [engine_path, "--batch", "--command",
+ f"script {script_path}"]
+ with open(out_log, "w") as logf:
+ rc = subprocess.run(cmd, stdout=logf, stderr=subprocess.STDOUT).returncode
+ if rc != 0:
+ raise RuntimeError(f"shard {shard_idx} engine exited {rc}, see {out_log}")
+
+ dumps = []
+ with open(out_log) as f:
+ for line in f:
+ if line.startswith("KINGDEFECTDUMP"):
+ dumps.append(parse_kingdefectdump_line(line))
+
+ if len(dumps) != 2 * len(rows):
+ raise RuntimeError(
+ f"shard {shard_idx}: expected {2 * len(rows)} KINGDEFECTDUMP "
+ f"lines (2 per row), got {len(dumps)} -- a setboard likely "
+ f"failed silently and evalkingdump re-used a stale position. "
+ f"See {out_log}.")
+
+ out_rows = []
+ for i, row in enumerate(rows):
+ black_color, black_kv = dumps[2 * i]
+ white_color, white_kv = dumps[2 * i + 1]
+ assert black_color == "black" and white_color == "white", \
+ f"row {i}: unexpected color order {black_color}/{white_color}"
+ target = row["mated_color_or_sampled_color"]
+ kv = black_kv if target == "black" else white_kv
+ out_row = dict(row)
+ out_row.update(kv)
+ out_rows.append(out_row)
+
+ os.remove(script_path)
+ os.remove(out_log)
+ return out_rows
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("training_csv")
+ ap.add_argument("out_csv")
+ ap.add_argument("--engine", default="../typhoon_kingdump")
+ ap.add_argument("--num-shards", type=int, default=8)
+ ap.add_argument("--work-dir", default=None)
+ args = ap.parse_args()
+
+ engine_path = os.path.abspath(args.engine)
+ if not os.path.isfile(engine_path):
+ print(f"Engine not found: {engine_path}", file=sys.stderr)
+ sys.exit(1)
+
+ with open(args.training_csv, newline="") as f:
+ rows = list(csv.DictReader(f))
+ print(f"Loaded {len(rows)} rows from {args.training_csv}", file=sys.stderr)
+
+ work_dir = args.work_dir or os.path.dirname(os.path.abspath(args.out_csv))
+ os.makedirs(work_dir, exist_ok=True)
+
+ n = args.num_shards
+ shard_size = (len(rows) + n - 1) // n
+ shards = [rows[i:i + shard_size] for i in range(0, len(rows), shard_size)]
+
+ t0 = time.time()
+ all_out_rows = [[] for _ in shards]
+
+ import concurrent.futures
+ with concurrent.futures.ThreadPoolExecutor(max_workers=n) as pool:
+ futures = {}
+ for idx, shard_rows in enumerate(shards):
+ fut = pool.submit(run_shard, engine_path, shard_rows, work_dir, idx)
+ futures[fut] = idx
+ for fut in concurrent.futures.as_completed(futures):
+ idx = futures[fut]
+ out_rows = fut.result()
+ all_out_rows[idx] = out_rows
+ print(f"Shard {idx} done ({len(out_rows)} rows)", file=sys.stderr)
+
+ print(f"All shards done in {time.time() - t0:.1f}s", file=sys.stderr)
+
+ base_cols = ["label", "mated_color_or_sampled_color", "n_back",
+ "game_index", "control_source"]
+ fieldnames = base_cols + FEATURE_KEYS
+
+ with open(args.out_csv, "w", newline="") as f:
+ w = csv.DictWriter(f, fieldnames=fieldnames)
+ w.writeheader()
+ for out_rows in all_out_rows:
+ for row in out_rows:
+ w.writerow({k: row.get(k) for k in fieldnames})
+
+ total = sum(len(r) for r in all_out_rows)
+ print(f"Wrote {total} feature rows to {args.out_csv}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/eval_tune/king_safety_data/HANDOFF.md b/src/eval_tune/king_safety_data/HANDOFF.md
new file mode 100644
index 0000000..b05e3d2
--- /dev/null
+++ b/src/eval_tune/king_safety_data/HANDOFF.md
@@ -0,0 +1,254 @@
+# King-safety defect regression: handoff
+
+## Goal
+
+`_EvalKing` (eval.c) sums a pile of hand-picked "defect" constants into one
+scalar (`uCounter`), then runs that through a single lookup table
+(`KING_SAFETY_BY_COUNTER`). The hand-picked weights assume every defect
+counts equally, which is suspect -- e.g. a queen touching an undefended
+ring square and a knight touching one add the exact same `+1`. This
+project mines real games to see which defects actually predict getting
+mated, so `eval.c`'s weights can be informed by data instead of guesswork.
+
+**Method**: find real middlegame checkmates in human games, look at the
+losing king's position N plies earlier, and compare its raw defect
+pattern against a matched "sound" control position (drawn games) with the
+same attacker material and development level. Fit a logistic regression;
+features with a real, stable coefficient are candidates for reweighting
+in `eval.c`. See `CLAUDE.md`'s "King-safety eval improvement project"
+section (search for it) for the full narrative this file summarizes.
+
+## Where everything is
+
+- **Final training data**: `remine_v4/king_features_matched_v4.csv` --
+ 86,362 rows (43,181 matched positive/control pairs). **Use this one.**
+- `remine_v4/merged.csv` -- raw mined FENs before feature extraction
+ (172,735 rows: 43,181 positives + 129,554 control candidates).
+- `remine_v4/king_features_v4.csv` -- all 172,735 rows with full
+ `_EvalKing` defect breakdown, before matching trims it down to the
+ 86,362 confound-controlled rows.
+- Pipeline scripts live in `eval_tune/` (sibling of this dir), not here:
+ `mine_king_mates.py`, `shard_mine.py`, `extract_king_features.py`,
+ `match_controls.py`, `train_king_safety_model.py`.
+- Engine side: `chess.h`'s `KING_DEFECT_DUMP_INFO` struct,
+ `_EvalKing`'s `#ifdef KING_DEFECT_DUMP` instrumentation in eval.c, and
+ command.c's `evalkingdump` command -- all gated behind the
+ `KING_DEFECT_DUMP=1` gmake build flag (NOT part of a normal build).
+ `../typhoon_kingdump` is a saved binary built with that flag; rebuild
+ it (`gmake clean` first -- flag changes don't trigger recompilation on
+ their own, see "Gotchas" below) if `eval.c`/`chess.h`/`command.c`
+ change.
+
+## Final result (from `king_features_matched_v4.csv`)
+
+Logistic regression, 80/20 train/test split, standardized coefficients.
+Test AUC 0.91 (plausible for "engineered danger features vs. proven-sound
+draws", not suspiciously perfect). Ranked by |coefficient|, most-to-least
+trustworthy tier first (stability = full-population vs. top-tercile-army-
+strength fits agreeing):
+
+**Tier 1 -- dominant, very stable**: `filestormopenfile` (+0.57 to +0.80),
+`filestormpawn` (+0.51 to +0.54). These are eval.c's `KingFileDefects[3]`
+and `KingStormingPawnDefects[8]` tables (open/half-open file near the
+king; how far an enemy pawn has advanced up that file), already
+independent tables -- lowest-risk place to start a real retune.
+
+**Tier 2 -- strong, stable**: `front_offboard`, `front_def`, `sq4_offboard`,
+`front_undef`, `front_atk_minor`, `front_atk_rook`, `misscastleshort`,
+`front_flight` (+0.33 to +0.45). "front" = the diagonal/vertical ring
+squares one rank away, in the direction each side attacks toward (see
+"Feature schema" below for the exact slot mapping -- do NOT assume this
+means "in front of the king" the way a human would first guess; see the
+correction note there). `sq3`/`sq4` are the two same-rank side squares,
+kept individually resolved.
+
+**Tier 3 -- moderate, less stable**: `back_offboard` (drifts the most of
+any surviving feature, treat with suspicion), `sq3_undef`,
+`attackerarmyscaler`, `queenproximity`, `back_atk_minor`, `front_atk_queen`.
+
+**Confirmed noise -- do not retune based on this data**: `minorsathome`
+(coefficient ~0.001, was previously the #1 "signal" before two sampling
+bugs got fixed -- see "Bugs found" below), `initloc`, `castled` (exactly
+0.0000, fully collinear with `misscastleshort`/`misscastlelong`).
+
+## Translating this into eval.c -- confidence and plan
+
+This is NOT a "throw away eval.c and use the model" situation. The plan
+is targeted retuning of existing constants/tables, informed by relative
+coefficient *ratios*, not literal log-odds-to-centipawn conversion:
+
+1. **`KingFileDefects`/`KingStormingPawnDefects` deserve a real weight
+ increase relative to everything else** -- they're the strongest
+ signal by a clear margin and already live in their own small,
+ isolated tables (`_ComputeFileStormDefects` in eval.c). Lowest risk,
+ highest expected payoff, start here.
+2. **`EmptyAttackedSquare`/`EmptyUnattackedSquare`'s front/back asymmetry
+ is directionally right but probably too weak a spread** -- front-
+ bucket features consistently outrank back-bucket ones here by a good
+ margin.
+3. **`misscastleshort`'s constant likely deserves a bump.**
+4. **Don't touch `KING_INITIAL_COUNTER_BY_LOCATION` or the undeveloped-
+ minors term** based on this data -- both came back as noise.
+5. Neither `KingFileDefects`/`KingStormingPawnDefects` nor the
+ ring-scan tables are currently `DNA_ARRAY`-registered (checked: only
+ `KING_SAFETY_BY_COUNTER` and `KING_QUEEN_PROXIMITY_DANGER` are under
+ king-safety). Retuning them means a manual edit (optionally promoting
+ them to file scope + registering as DNA arrays for future tuning),
+ not "rerun the tuner."
+6. **Absolute magnitude has to be calibrated the normal way** -- run any
+ change through `sd10` ECM suite comparison against `head_reference/`
+ and `match_play.py`'s gated head-to-head match, same as any other
+ hand-tuned eval constant. The regression tells you *shape*
+ (which defects matter more relative to each other), not the right
+ centipawn magnitude for a chess engine's scoring scale.
+7. `CountKingSafetyDefects` (the cheap lazy-eval proxy) stays untouched
+ until a real change lands and is validated -- agreed early in this
+ project, still the right call.
+
+## Feature schema (read this before extending the pipeline)
+
+`KingSafetyDeltas` in eval.c (`-17,-16,-15,-1,+1,+15,+16,+17`) defines 8
+ring slots, indices 0-7 in that order. **Correction made mid-project**:
+slots 3 and 4 (deltas -1/+1) are the two SAME-RANK side squares, not
+"front of the king" -- an earlier description in conversation got this
+wrong. The diagonal/vertical slots (0,1,2 and 5,6,7) are the ones with a
+real front/back asymmetry, matching `EmptyAttackedSquare`'s own existing
+color-dependent weighting (WHITE's front is slots 0/1/2, BLACK's is
+5/6/7 -- see `_EvalKing`'s comments and `KING_RING_FRONT`-style logic).
+
+`extract_king_features.py` dumps full per-slot resolution (all 8 slots x
+4 piece types x several state flags) since the engine-side cost of doing
+so is free once you're calling `evalkingdump` anyway. `train_king_safety_model.py`
+is where slots 0/1/2/5/6/7 get bucketed into "front"/"back" sums
+(color-dependent per row's `mated_color_or_sampled_color`), while slots 3/4
+stay individually resolved -- this bucketing was added in this session
+after the diagonal per-slot coefficients turned out to be the noisiest,
+most sign-flipping entries in every full-vs-subset drift comparison.
+`filestorm` is deliberately NOT used as a training feature -- its two
+independent sub-components (`filestormopenfile`, `filestormpawn`) are
+used instead, since they map onto two already-separate eval.c tables and
+there's no reason to make the regression re-derive their relative
+importance from one pre-summed number.
+
+## Bugs found and fixed this session (read before trusting old data)
+
+If you find any `.csv` file in this project OTHER than
+`remine_v4/*_v4.csv` and its merged.csv, IT PREDATES ONE OR MORE OF THESE
+FIXES AND SHOULD NOT BE USED. (Older files were deleted during cleanup,
+but if this pipeline gets rerun and produces intermediate files again,
+this is the checklist for whether they're trustworthy.)
+
+1. **Bare-king endgame mates leaking into "middlegame" positives.** Only
+ checking the attacker's material (`>= 800cp`, roughly rook+bishop) let
+ trivial K+Q-vs-lone-K technique mates through. Fixed by also requiring
+ `total_non_pawn_piece_count(board) >= 8` (mirrors `_EvalKing`'s own
+ `pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0] < 8`
+ endgame boundary exactly) -- see `is_middlegame()` in
+ `mine_king_mates.py`.
+2. **`min_ply` too low (10, i.e. move 5) for control sampling.** A large
+ fraction of the pool's draws are short draws-by-agreement close to
+ `filter_pgn.py`'s own 40-ply floor. Combined with a large end-guard
+ (staying clear of the game's end), the only valid sampling window on a
+ short draw was the early opening -- where minors are legitimately
+ still undeveloped for completely ordinary reasons. This alone caused
+ `minorsathome` to look like a massive, spurious "signal" (controls
+ showed `minorsathome>=2` in ~85% of samples). Verified the engine's
+ own computation was correct on a real developed-middlegame FEN --
+ 100% a sampling-window bug, not an eval bug. Fixed: `min_ply` raised
+ to 20 (move 10), and a game too short to satisfy both `min_ply` and
+ the end-guard is now skipped entirely rather than force-sampled early.
+3. **Control-sampling ply selection secretly always picked the earliest
+ plies in a game's valid window.** `rng.shuffle(candidates)` followed
+ by `for idx in sorted(candidates)` completely undid the shuffle.
+ Fixed in two steps (the first fix wasn't sufficient on its own --
+ see the code comment in `mine_controls()` for why): randomly
+ subsample which plies to try, replay ascending through all of them
+ ONCE (required -- boards can only be walked forward), collect every
+ result that passes filters into a list, THEN randomly choose which
+ results to actually keep via `rng.sample` on that list. Verified:
+ sampled-ply-as-fraction-of-game-length now has a median around 0.46
+ with real tail into 60%+, instead of clustering at 20-30%.
+4. **Matching-radius bug treated two very differently-scaled confound
+ columns with one shared distance metric.** `attackerarmyscaler`
+ spans 0-31 (`ASSERT`-confirmed hard clamp ceiling), `minorsathome`
+ spans 0-4. A single Chebyshev radius meant a "tight" radius-2 match
+ still let `minorsathome` drift by up to 2 -- literally half its
+ entire range. This is why `minorsathome` was STILL the #1 "signal"
+ even after an initial matching attempt. Fixed: `match_controls.py`
+ now takes `--exact-cols` (must match exactly, no radius search --
+ `minorsathome` defaults here) and `--radius-cols` (allowed to expand
+ outward, `attackerarmyscaler` defaults here) as separate concepts.
+5. **KDTree-based nearest-neighbor matching matched almost nothing** (308
+ of 43,181 positives) because the match columns are low-cardinality
+ integers -- huge exact-value clusters mean a KDTree's "k nearest" are
+ massively tied, so the first few positives processed in a cluster
+ exhaust the shared candidate set and everyone else in that cluster
+ fails to match. Replaced with exact-bucket + expanding-integer-radius
+ matching (see `match_controls.py`), which has no such collision
+ problem.
+6. **`_EvalKing`'s per-square ring scan implicitly assumed all 8 slots
+ were geometrically equivalent for feature purposes**; slots 3/4
+ (side squares) vs. the diagonal slots turned out to behave
+ differently enough that per-slot resolution on the diagonals was
+ pure noise at the sample sizes available -- see "Feature schema"
+ above.
+7. **Board-diagram spam.** `setboard` prints a full ASCII board dump on
+ every call by default (`g_Options.fShouldPost`, see `DumpPosition` in
+ gamelist.c), which dominated per-row I/O cost during feature
+ extraction. Fixed by prepending `nopost` to every extraction script.
+8. **Two separate concurrent-process races during interactive mining**,
+ both self-inflicted mid-session: (a) `$!` after a `cd ... && nohup
+ ...` pipeline captured the wrong PID, leaving a stale duplicate
+ process racing on the same output file as a supposedly-fresh rerun;
+ (b) killing that process didn't fully stop it before a new one
+ started. No lasting damage (caught via `ps aux` before trusting any
+ output), but: **always verify a background job is actually singular
+ via `ps aux | grep <script>` before trusting its output**, especially
+ after a kill+relaunch sequence.
+9. **`sizeof(KING_DEFECT_DUMP)` name collision with the
+ `-DKING_DEFECT_DUMP` build flag.** The build flag is defined as
+ literal `1` by the compiler; naming the struct type the same thing
+ caused the preprocessor to mangle the struct declaration wherever the
+ type name appeared. Renamed the struct to `KING_DEFECT_DUMP_INFO`.
+ If you ever add a new `#ifdef SOME_NAME` build flag AND a same-named
+ type/variable in the same session, expect this failure mode.
+10. **`gmake` doesn't recompile on a flag-only change.** After the struct
+ rename (#9) and other `chess.h` edits, a `gmake KING_DEFECT_DUMP=1`
+ without a preceding `gmake clean` silently reused stale `.o` files
+ from an earlier flag-less build, producing a binary that didn't
+ actually have `evalkingdump` registered, with no compile error at
+ all. Always `gmake clean` first when a build-flag combination
+ changes, not just when headers change (this is already called out
+ in `CLAUDE.md` for header changes, but the flag-only case is easy to
+ forget too).
+
+## Performance notes for rerunning
+
+- `pgn/twic_filtered.pgn` (642,268 games) mines in a few minutes when
+ sharded 12-24 ways via `shard_mine.py` (byte-range sharding on real
+ game boundaries, found via a fast raw-text `[Event ` scan). Don't run
+ it unsharded -- a single-process full scan took ~40 minutes.
+- Feature extraction (`extract_king_features.py`) is the slow stage:
+ ~54 minutes for 172,735 rows sharded 18 ways on a contended 24-core
+ box. `nopost` (fix #7 above) is essential -- without it, board-diagram
+ printing dominates I/O cost.
+- `match_controls.py` is pure in-memory bucket lookups, seconds even for
+ the full dataset -- never worth sharding.
+- If `twic_filtered.pgn` (642k games, >=2400 Elo / >40 ply filtered) ever
+ proves insufficient for match density, `pgn/twic.pgn` is 7-8x larger
+ (4,099,094 games, unfiltered) and available as a fallback source --
+ not needed this run (100% match rate achieved on `twic_filtered.pgn`
+ once the above bugs were fixed).
+
+## Suggested next steps
+
+1. Pick 2-3 concrete constant changes from Tier 1/2 above (start with
+ `KingFileDefects`/`KingStormingPawnDefects` -- lowest risk).
+2. Land them as a normal `eval.c` diff, run `precommit_check.sh`.
+3. `sd10` comparison across the three curated suites against
+ `head_reference/`.
+4. `match_play.py` gated head-to-head match against the pre-change DNA.
+5. Only keep the change if `LOWER95 >= 0.5`.
+6. Once a change lands, consider whether `CountKingSafetyDefects` (lazy
+ eval) needs corresponding rework -- deferred by design until a real
+ change is validated (see project goal section above).
diff --git a/src/eval_tune/match_controls.py b/src/eval_tune/match_controls.py
new file mode 100644
index 0000000..10bb821
--- /dev/null
+++ b/src/eval_tune/match_controls.py
@@ -0,0 +1,168 @@
+#!/usr/bin/env python3
+"""Case-control matching to remove the attacker-material/development
+confound found in the naive full-population regression: attackerarmyscaler
+alone got AUC 0.99 distinguishing mate-precursor positions from
+draw-derived controls, causing quasi-complete separation that destabilizes
+every other coefficient in the joint fit. Fix: match each positive to the
+nearest available control by (attackerarmyscaler, minorsathome) instead of
+using controls sampled freely from the draw pool. See CLAUDE.md's
+king-safety discussion for the full methodology.
+
+Uses exact-bucket + expanding-radius matching, not a KDTree: the match
+columns are low-cardinality integers, so exact-value clusters are huge and
+a KDTree's k nearest neighbors are highly tied -- every positive in a big
+cluster sees almost the same "nearest k" candidates, so the first few
+positions processed exhaust that shared candidate set and everyone else in
+the cluster fails to match (confirmed: a k=50 KDTree approach matched only
+308 of 43181 positives). Bucketing by exact integer key, popping
+candidates on use, and expanding search radius only when a bucket is
+empty has no such collision problem and is O(1) amortized per positive.
+
+--exact-cols vs --radius-cols, NOT a single shared radius across all match
+columns: a first version used one Chebyshev radius across both
+attackerarmyscaler (range 0-31) and minorsathome (range 0-4) -- "radius <=
+2" is tight for army-scaler (~6% of its range) but is HALF of
+minorsathome's entire range, so positives kept matching against controls
+with a wildly different minorsathome just to satisfy the harder
+army-scaler dimension. Confirmed by inspecting matched pairs: mean
+minorsathome gap of 1.3 survived a nominally "tight" radius-2 match,
+leaving minorsathome as the regression's #1 feature for reasons that were
+substantially still the sampling artifact, not real signal. Fix: columns
+in --exact-cols must match exactly (no radius search at all -- if a
+positive's exact value has no candidates left, it's unmatched, full
+stop); only --radius-cols expand outward. minorsathome defaults to exact
+since its range is small enough that "close" isn't actually close.
+"""
+import argparse
+import csv
+import itertools
+import random
+import sys
+from collections import defaultdict
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("features_csv")
+ ap.add_argument("out_csv")
+ ap.add_argument("--exact-cols", nargs="+", default=["minorsathome"],
+ help="Match columns that must match EXACTLY -- no "
+ "radius search. Use for low-cardinality columns "
+ "where 'close' isn't actually close.")
+ ap.add_argument("--radius-cols", nargs="+", default=["attackerarmyscaler"],
+ help="Match columns allowed to expand outward by "
+ "--max-radius if no exact match is available.")
+ ap.add_argument("--max-radius", type=int, default=30,
+ help="Max Chebyshev distance across --radius-cols to "
+ "search outward before giving up on a positive")
+ ap.add_argument("--seed", type=int, default=42)
+ args = ap.parse_args()
+
+ rng = random.Random(args.seed)
+
+ with open(args.features_csv, newline="") as f:
+ reader = csv.DictReader(f)
+ assert reader.fieldnames is not None
+ fieldnames = list(reader.fieldnames)
+ rows = list(reader)
+
+ positives = [r for r in rows if r["label"] == "1"]
+ candidates = [r for r in rows if r["label"] == "0"]
+ print(f"Loaded {len(positives)} positives, {len(candidates)} control "
+ f"candidates", file=sys.stderr)
+ print(f"Exact-match columns: {args.exact_cols}", file=sys.stderr)
+ print(f"Radius-search columns (max radius {args.max_radius}): "
+ f"{args.radius_cols}", file=sys.stderr)
+
+ def exact_key_of(r):
+ return tuple(int(round(float(r[c]))) for c in args.exact_cols)
+
+ def radius_key_of(r):
+ return tuple(int(round(float(r[c]))) for c in args.radius_cols)
+
+ # Two-level index: exact key -> radius key -> list of candidate rows.
+ # A positive only ever searches within its own exact-key bucket, and
+ # within that, expands outward over the radius-cols only.
+ buckets = defaultdict(lambda: defaultdict(list))
+ for r in candidates:
+ buckets[exact_key_of(r)][radius_key_of(r)].append(r)
+ for ekey in buckets:
+ for rkey in buckets[ekey]:
+ rng.shuffle(buckets[ekey][rkey])
+
+ ndim = len(args.radius_cols)
+
+ def offsets_at_radius(radius):
+ if radius == 0:
+ return [(0,) * ndim]
+ rng_vals = range(-radius, radius + 1)
+ return [combo for combo in itertools.product(rng_vals, repeat=ndim)
+ if max(abs(v) for v in combo) == radius]
+
+ shells = [offsets_at_radius(r) for r in range(args.max_radius + 1)]
+
+ matched_pairs = []
+ unmatched = 0
+ unmatched_no_exact_bucket = 0
+ radius_hist = defaultdict(int)
+
+ for pos_row in positives:
+ ekey = exact_key_of(pos_row)
+ exact_bucket = buckets.get(ekey)
+ if exact_bucket is None:
+ unmatched += 1
+ unmatched_no_exact_bucket += 1
+ continue
+ pkey = radius_key_of(pos_row)
+ found = None
+ found_radius = None
+ for radius, shell in enumerate(shells):
+ for off in shell:
+ cand_key = tuple(pkey[i] + off[i] for i in range(ndim))
+ cand_list = exact_bucket.get(cand_key)
+ if cand_list:
+ found = cand_list.pop()
+ found_radius = radius
+ break
+ if found is not None:
+ break
+ if found is None:
+ unmatched += 1
+ continue
+ radius_hist[found_radius] += 1
+ matched_pairs.append((pos_row, found))
+
+ print(f"Matched {len(matched_pairs)} pairs, {unmatched} positives "
+ f"left unmatched ({unmatched_no_exact_bucket} had no candidates "
+ f"at all with that exact-column value, {unmatched - unmatched_no_exact_bucket} "
+ f"exhausted radius {args.max_radius} on the radius columns)",
+ file=sys.stderr)
+ print("Match radius histogram (0 = exact match on everything):",
+ file=sys.stderr)
+ for r in sorted(radius_hist):
+ print(f" radius {r}: {radius_hist[r]}", file=sys.stderr)
+
+ with open(args.out_csv, "w", newline="") as f:
+ w = csv.DictWriter(f, fieldnames=fieldnames)
+ w.writeheader()
+ for pos_row, ctl_row in matched_pairs:
+ w.writerow(pos_row)
+ w.writerow(ctl_row)
+
+ print(f"Wrote {2 * len(matched_pairs)} matched rows to {args.out_csv}",
+ file=sys.stderr)
+
+ import numpy as np
+ all_cols = args.exact_cols + args.radius_cols
+ def full_key_of(r):
+ return tuple(int(round(float(r[c]))) for c in all_cols)
+ matched_pos_vals = np.array([full_key_of(p) for p, _ in matched_pairs])
+ matched_ctl_vals = np.array([full_key_of(c) for _, c in matched_pairs])
+ for i, col in enumerate(all_cols):
+ diff = np.abs(matched_pos_vals[:, i] - matched_ctl_vals[:, i])
+ print(f" {col}: mean|diff|={diff.mean():.2f} max|diff|={diff.max():.2f}",
+ file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/eval_tune/mine_king_mates.py b/src/eval_tune/mine_king_mates.py
new file mode 100644
index 0000000..7b5866b
--- /dev/null
+++ b/src/eval_tune/mine_king_mates.py
@@ -0,0 +1,315 @@
+#!/usr/bin/env python3
+"""Mine middlegame checkmate positions + matched controls for king-safety
+defect regression. See CLAUDE.md / conversation context: _EvalKing's
+counted "defects" are summed into one scalar and run through one lookup
+table with no per-defect weighting; this pulls training data to fit
+relative weights instead of hand-picking them.
+
+Output: a CSV of (fen, label, mated_color, n_back, game_id) rows. FEN
+extraction only -- no engine call here. The actual per-defect feature
+values get pulled from these FENs by the engine-side dump command
+(separate step).
+"""
+import argparse
+import chess
+import chess.pgn
+import csv
+import random
+import sys
+
+# Matches eval.c's DO_KING_SAFETY_THRESHOLD minus the VALUE_KING baseline
+# _EvalKing adds internally: real threshold is "enemy has at least a
+# rook+bishop of non-pawn, non-king material" before king safety is even
+# scored at all. No point mining data _EvalKing would ignore.
+MATING_MATERIAL_THRESHOLD_CP = 800
+
+PIECE_VALUES_CP = {
+ chess.PAWN: 0, # excluded: "non-pawn material" by definition
+ chess.KNIGHT: 300,
+ chess.BISHOP: 300,
+ chess.ROOK: 500,
+ chess.QUEEN: 975,
+ chess.KING: 0, # excluded: baseline, not a mating-material contributor
+}
+
+
+def non_pawn_material_cp(board, color):
+ total = 0
+ for piece_type in (chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN):
+ total += len(board.pieces(piece_type, color)) * PIECE_VALUES_CP[piece_type]
+ return total
+
+
+# Mirrors _EvalKing's own middlegame/endgame boundary exactly (eval.c:
+# "u = pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0]; if (u <
+# 8)") -- total non-pawn pieces, including both kings, across both sides.
+# Below this, _EvalKing itself stops caring about ring-square/file-storm
+# defects and switches to endgame-only terms (centralize king, escort
+# passer). Checking the attacker's material alone (non_pawn_material_cp)
+# is NOT sufficient on its own: a lone queen mating a bare king trivially
+# clears that floor but is a basic mating technique, not a middlegame
+# king-safety collapse -- caught by manually sampling FENs from the mined
+# set (several K+Q-vs-lone-K endings slipped through material-only
+# filtering).
+def total_non_pawn_piece_count(board):
+ count = 2 # both kings
+ for color in (chess.WHITE, chess.BLACK):
+ for piece_type in (chess.KNIGHT, chess.BISHOP, chess.ROOK, chess.QUEEN):
+ count += len(board.pieces(piece_type, color))
+ return count
+
+
+MIDDLEGAME_MIN_TOTAL_NON_PAWN_PIECES = 8
+
+
+def is_middlegame(board):
+ return total_non_pawn_piece_count(board) >= MIDDLEGAME_MIN_TOTAL_NON_PAWN_PIECES
+
+
+def mine_game(game, n_backs, min_ply_from_start=20):
+ """Yields (fen, mated_color, n_back) for a game ending in checkmate."""
+ board = game.board()
+ moves = list(game.mainline_moves())
+ if not moves:
+ return
+ # Replay to the end to confirm checkmate and find the mated side.
+ for mv in moves:
+ board.push(mv)
+ if not board.is_checkmate():
+ return
+ mated_color = board.turn # side to move at mate = side that got mated
+ total_plies = len(moves)
+
+ for n_back in n_backs:
+ idx = total_plies - n_back
+ if idx < min_ply_from_start:
+ continue
+ replay = game.board()
+ for mv in moves[:idx]:
+ replay.push(mv)
+ # Only meaningful if the mated side's king safety was even "in
+ # scope" for _EvalKing at this point -- attacker must have
+ # mating material.
+ attacker = not mated_color
+ if non_pawn_material_cp(replay, attacker) < MATING_MATERIAL_THRESHOLD_CP:
+ continue
+ if not is_middlegame(replay):
+ continue
+ yield replay.fen(), mated_color, n_back
+
+
+def is_draw(game):
+ return game.headers.get("Result", "") == "1/2-1/2"
+
+
+def mine_controls(game, n_needed, rng, min_ply=20, max_ply_from_end=15):
+ """Yields (fen, color) control samples from a non-mate game (or a mate
+ game's earlier-game portion), gated on the same material threshold,
+ kept well clear of the game's ending so a real collapse (if any) isn't
+ accidentally captured as a "stable" control.
+
+ Draw games should be called with a larger max_ply_from_end than
+ decisive/resignation games (see is_draw callers in main()) -- a draw
+ only vouches for a king position if the position stayed quiet for a
+ real stretch *after* the sample point too, not just "the game
+ eventually drew" (e.g. a middlegame attack that later got defused into
+ a simplified drawn endgame doesn't vouch for the earlier position).
+
+ min_ply must be large enough that development is realistically done
+ (move ~10, ply 20) -- NOT just "not literally move 1". A large chunk
+ of the pool's draws are short draws-by-agreement close to
+ filter_pgn.py's own 40-ply floor; combined with a big max_ply_from_end
+ and a too-small min_ply, the only plies satisfying BOTH constraints on
+ a short game land in the early opening, where minorsathome/development
+ features are legitimately still nonzero for completely ordinary
+ reasons -- not a real king-safety defect. Confirmed as the actual bug
+ behind an earlier version's controls showing minorsathome>=2 in ~85%
+ of samples: the engine's own computation was verified correct on a
+ real developed middlegame FEN, so the contamination was 100% a
+ sampling-window problem, not an eval bug.
+
+ A game too short to satisfy both min_ply and max_ply_from_end is
+ skipped entirely rather than forced into an early sample -- fewer,
+ honest control samples beat many contaminated ones."""
+ moves = list(game.mainline_moves())
+ total_plies = len(moves)
+ end_guard = max_ply_from_end
+ if total_plies < min_ply + end_guard:
+ return
+ candidates = list(range(min_ply, max(min_ply, total_plies - end_guard)))
+ if not candidates:
+ return
+ # BUG #1 (found while investigating why controls skewed early-game):
+ # shuffling `candidates` then iterating `sorted(candidates)` completely
+ # undid the shuffle, so every game always visited its window in
+ # ascending ply order and grabbed the *earliest* plies satisfying
+ # per_game_budget -- systematically starving late-middlegame coverage
+ # in any game with a wide candidate window, regardless of min_ply.
+ #
+ # BUG #2 (found immediately after "fixing" #1 with a random subsample
+ # + sort): stopping the ascending-order loop as soon as `produced >=
+ # n_needed` still biases toward whichever subsample members happen to
+ # be smallest, since ascending iteration always finds the low end of
+ # the subsample first. Measured: even after subsampling, sampled plies
+ # still clustered at the 20th-40th percentile of game length instead
+ # of spreading across the whole game.
+ #
+ # Fix for both: replay ascending through an oversampled random subset
+ # ONCE (required -- python-chess boards are only walkable forward) to
+ # collect every candidate result that passes the material/middlegame
+ # filters, then pick which ones to actually keep via `rng.sample` on
+ # that result LIST -- decoupling "must replay in order" from "which
+ # results count" is what actually gives uniform ply coverage.
+ num_to_try = min(len(candidates), max(n_needed * 4, 20))
+ sample_indices = sorted(rng.sample(candidates, num_to_try))
+ replay = game.board()
+ ply = 0
+ valid_results = []
+ for idx in sample_indices:
+ while ply < idx:
+ replay.push(moves[ply])
+ ply += 1
+ if not is_middlegame(replay):
+ continue
+ for color in (chess.WHITE, chess.BLACK):
+ attacker = not color
+ if non_pawn_material_cp(replay, attacker) >= MATING_MATERIAL_THRESHOLD_CP:
+ valid_results.append((replay.fen(), color))
+
+ if len(valid_results) > n_needed:
+ valid_results = rng.sample(valid_results, n_needed)
+ for fen, color in valid_results:
+ yield fen, color
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("pgn_file")
+ ap.add_argument("out_csv")
+ ap.add_argument("--n-backs", type=int, nargs="+", default=[4, 6, 8, 10])
+ ap.add_argument("--max-games", type=int, default=None,
+ help="Stop after scanning this many games (0/None = all)")
+ ap.add_argument("--control-ratio", type=float, default=1.0,
+ help="Controls per positive, roughly")
+ ap.add_argument("--draw-only-controls", action="store_true",
+ help="Don't fall back to decisive/resignation games "
+ "for controls even if draws don't fill the quota")
+ ap.add_argument("--draw-end-guard", type=int, default=24,
+ help="Plies of quiet play required *after* a draw "
+ "control sample, stricter than decisive games "
+ "since a draw only vouches for a position that "
+ "stayed sound past the sample point too")
+ ap.add_argument("--decisive-end-guard", type=int, default=15)
+ ap.add_argument("--min-ply", type=int, default=20,
+ help="Earliest ply eligible for any sample (positive "
+ "or control) -- must be well past normal opening "
+ "development (ply 20 = move 10), not just >0. "
+ "See mine_controls' docstring for why this "
+ "matters a lot for short draws.")
+ ap.add_argument("--seed", type=int, default=42)
+ ap.add_argument("--byte-start", type=int, default=0,
+ help="Seek here before reading (shard support -- "
+ "must be a real game-start offset, e.g. from "
+ "shard_mine.py, not an arbitrary byte)")
+ ap.add_argument("--byte-end", type=int, default=None,
+ help="Stop once past this offset (shard support)")
+ args = ap.parse_args()
+
+ rng = random.Random(args.seed)
+ positives = []
+ draw_pool_games = []
+ decisive_pool_games = []
+
+ with open(args.pgn_file, encoding="utf-8", errors="replace") as f:
+ if args.byte_start:
+ f.seek(args.byte_start)
+ n_scanned = 0
+ while True:
+ if args.max_games and n_scanned >= args.max_games:
+ break
+ if args.byte_end is not None and f.tell() >= args.byte_end:
+ break
+ game = chess.pgn.read_game(f)
+ if game is None:
+ break
+ n_scanned += 1
+ if n_scanned % 20000 == 0:
+ print(f"...scanned {n_scanned} games, "
+ f"{len(positives)} positives so far", file=sys.stderr)
+ try:
+ found_any = False
+ for fen, mated_color, n_back in mine_game(
+ game, args.n_backs, min_ply_from_start=args.min_ply):
+ positives.append((fen, 1, chess.COLOR_NAMES[mated_color],
+ n_back, n_scanned))
+ found_any = True
+ if not found_any:
+ if is_draw(game):
+ draw_pool_games.append(game)
+ else:
+ decisive_pool_games.append(game)
+ except Exception as e:
+ print(f"skip game {n_scanned}: {e}", file=sys.stderr)
+ continue
+
+ print(f"Scanned {n_scanned} games -> {len(positives)} positive samples "
+ f"from checkmate games. Control pools: {len(draw_pool_games)} "
+ f"draws, {len(decisive_pool_games)} decisive/other.",
+ file=sys.stderr)
+
+ n_controls_needed = int(len(positives) * args.control_ratio)
+ controls = []
+ per_game_budget = 2
+
+ # Draws first -- a position two comparable players couldn't crack is
+ # stronger "this was sound" evidence than a resignation, where the
+ # king position may simply never have been tested by that opponent.
+ rng.shuffle(draw_pool_games)
+ for game in draw_pool_games:
+ if len(controls) >= n_controls_needed:
+ break
+ try:
+ for fen, color in mine_controls(game, per_game_budget, rng,
+ min_ply=args.min_ply,
+ max_ply_from_end=args.draw_end_guard):
+ controls.append((fen, 0, chess.COLOR_NAMES[color], -1, None, "draw"))
+ except Exception:
+ continue
+
+ n_from_draws = len(controls)
+ print(f"Collected {n_from_draws} control samples from draws.",
+ file=sys.stderr)
+
+ if not args.draw_only_controls and len(controls) < n_controls_needed:
+ rng.shuffle(decisive_pool_games)
+ for game in decisive_pool_games:
+ if len(controls) >= n_controls_needed:
+ break
+ try:
+ for fen, color in mine_controls(game, per_game_budget, rng,
+ min_ply=args.min_ply,
+ max_ply_from_end=args.decisive_end_guard):
+ controls.append((fen, 0, chess.COLOR_NAMES[color], -1, None, "decisive"))
+ except Exception:
+ continue
+ print(f"Collected {len(controls) - n_from_draws} additional control "
+ f"samples from decisive/resignation games (quota not met by "
+ f"draws alone).", file=sys.stderr)
+
+ print(f"Collected {len(controls)} control samples total.", file=sys.stderr)
+
+ with open(args.out_csv, "w", newline="") as f:
+ w = csv.writer(f)
+ w.writerow(["fen", "label", "mated_color_or_sampled_color", "n_back",
+ "game_index", "control_source"])
+ for row in positives:
+ w.writerow(row + (None,))
+ for row in controls:
+ w.writerow(row)
+
+ print(f"Wrote {len(positives) + len(controls)} rows to {args.out_csv}",
+ file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/eval_tune/shard_mine.py b/src/eval_tune/shard_mine.py
new file mode 100644
index 0000000..6c82ed3
--- /dev/null
+++ b/src/eval_tune/shard_mine.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Parallelize mine_king_mates.py across N shards by game-boundary byte
+offset, since a full 642k-game scan is CPU-bound (python-chess replay +
+SAN parsing per game) and this box has 24 cores. Finds "[Event " line
+starts via a fast raw byte scan (not a full PGN parse) to get exact game
+boundaries, splits into N even-sized shards by game count, launches one
+mine_king_mates.py subprocess per shard with --byte-start/--byte-end, then
+concatenates the resulting CSVs (one header, rows from all shards).
+"""
+import argparse
+import csv
+import os
+import subprocess
+import sys
+import time
+
+
+def find_game_offsets(pgn_file):
+ offsets = []
+ pos = 0
+ with open(pgn_file, "rb") as f:
+ for line in f:
+ if line.startswith(b"[Event "):
+ offsets.append(pos)
+ pos += len(line)
+ return offsets, pos # pos ends up as total file size
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("pgn_file")
+ ap.add_argument("out_dir")
+ ap.add_argument("--num-shards", type=int, default=8)
+ ap.add_argument("--extra-args", default="",
+ help="Extra args passed through to mine_king_mates.py, "
+ "e.g. '--control-ratio 1.0'")
+ args = ap.parse_args()
+
+ os.makedirs(args.out_dir, exist_ok=True)
+ script_dir = os.path.dirname(os.path.abspath(__file__))
+ mine_script = os.path.join(script_dir, "mine_king_mates.py")
+
+ print("Scanning for game boundaries...", file=sys.stderr)
+ t0 = time.time()
+ offsets, file_size = find_game_offsets(args.pgn_file)
+ print(f"Found {len(offsets)} games, {file_size} bytes, "
+ f"in {time.time() - t0:.1f}s", file=sys.stderr)
+
+ n = args.num_shards
+ games_per_shard = len(offsets) // n
+ shard_bounds = []
+ for i in range(n):
+ start_idx = i * games_per_shard
+ end_idx = (i + 1) * games_per_shard if i < n - 1 else len(offsets)
+ byte_start = offsets[start_idx]
+ byte_end = offsets[end_idx] if end_idx < len(offsets) else file_size
+ shard_bounds.append((byte_start, byte_end))
+
+ procs = []
+ for i, (byte_start, byte_end) in enumerate(shard_bounds):
+ out_csv = os.path.join(args.out_dir, f"shard_{i}.csv")
+ log_file = os.path.join(args.out_dir, f"shard_{i}.log")
+ cmd = [
+ sys.executable, mine_script, args.pgn_file, out_csv,
+ "--byte-start", str(byte_start),
+ "--byte-end", str(byte_end),
+ ] + args.extra_args.split()
+ print(f"Launching shard {i}: bytes [{byte_start}, {byte_end}) -> {out_csv}",
+ file=sys.stderr)
+ with open(log_file, "w") as logf:
+ p = subprocess.Popen(cmd, stdout=logf, stderr=subprocess.STDOUT)
+ procs.append((i, p, out_csv))
+
+ failed = []
+ for i, p, out_csv in procs:
+ rc = p.wait()
+ print(f"Shard {i} exited with code {rc}", file=sys.stderr)
+ if rc != 0:
+ failed.append(i)
+
+ if failed:
+ print(f"FAILED shards: {failed} -- not merging, check logs in {args.out_dir}",
+ file=sys.stderr)
+ sys.exit(1)
+
+ merged_path = os.path.join(args.out_dir, "merged.csv")
+ header = None
+ total_rows = 0
+ with open(merged_path, "w", newline="") as out_f:
+ writer = csv.writer(out_f)
+ for i, _, out_csv in procs:
+ with open(out_csv, newline="") as in_f:
+ reader = csv.reader(in_f)
+ this_header = next(reader)
+ if header is None:
+ header = this_header
+ writer.writerow(header)
+ elif this_header != header:
+ print(f"WARNING: shard {i} header mismatch: {this_header}",
+ file=sys.stderr)
+ for row in reader:
+ writer.writerow(row)
+ total_rows += 1
+
+ print(f"Merged {total_rows} rows from {n} shards into {merged_path}",
+ file=sys.stderr)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/src/eval_tune/test_vs_head.sh b/src/eval_tune/test_vs_head.sh
index d5d8006..871b801 100755
--- a/src/eval_tune/test_vs_head.sh
+++ b/src/eval_tune/test_vs_head.sh
@@ -1,11 +1,19 @@
#!/usr/bin/env bash
-python3 ./match_play.py ../../head_reference/typhoon ../typhoon \
- --pgn ../../pgn/twic_filtered.pgn \
- --games 20000 \
- --workers 12 \
- --st 1 \
- --sprt --elo0 0 --elo1 5 \
- --scratch /usr/local/tmp/typhoon_match_overnight \
- --log overnight_match.log \
- --pgn-out ../self_play_games.pgn
+set -e
+
+CANDIDATE=/tmp/typhoon/typhoon_match_play_candidate
+HEAD=../../head_reference/typhoon
+
+cp -f ../typhoon ${CANDIDATE}
+python3 ./match_play.py \
+ ${HEAD} \
+ ${CANDIDATE} \
+ --pgn ../../pgn/twic_filtered.pgn \
+ --games 20000 \
+ --workers 12 \
+ --st 1 \
+ --sprt --elo0 0 --elo1 5 \
+ --scratch /usr/local/tmp/typhoon_match_overnight \
+ --log overnight_match.log \
+ --pgn-out ../self_play_games.pgn
diff --git a/src/eval_tune/train_king_safety_model.py b/src/eval_tune/train_king_safety_model.py
new file mode 100644
index 0000000..012e636
--- /dev/null
+++ b/src/eval_tune/train_king_safety_model.py
@@ -0,0 +1,189 @@
+#!/usr/bin/env python3
+"""Fit a logistic regression on king_features.csv's raw per-defect
+breakdown against the mate/control label, to see whether _EvalKing's
+defects should be weighted differently than the hand-picked constants
+currently baked into KING_SAFETY_BY_COUNTER / KING_COUNTER_BY_ATTACK_PATTERN
+/ EmptyAttackedSquare etc. See CLAUDE.md's king-safety discussion for the
+full methodology this is built on.
+
+Deliberately excludes `countertotal` and `safetyvalue` from the feature
+set -- those are the CURRENT engine's own derived output (the thing we're
+trying to second-guess), not a raw defect. Using them as inputs would be
+circular. `initloc` is kept as a feature (a compact king-square exposure
+signal) since, unlike the other two, it doesn't already fold in this
+sample's own defect counts.
+
+Attacker material confound (conversation context: _EvalKing scales its
+final score by REDUCED_MATERIAL_DOWN_SCALER[armyscaler], but the raw
+defect counts here are unscaled, and a real mate is naturally rarer under
+a weak attacker regardless of what the defects say) is handled two ways:
+included as a plain covariate (attackerarmyscaler), and separately
+validated by refitting on the top-tercile-army-strength subset and
+diffing coefficients against the full-population fit.
+
+Ring-square bucketing: slots 3/4 (the KingSafetyDeltas -1/+1 deltas -- the
+two SAME-RANK side squares, not "front" despite an earlier mislabeling in
+conversation) are kept individually resolved since they carried real,
+stable signal in initial runs. The other six slots (the diagonal +
+vertical squares one rank away) are collapsed into "front"/"back" bucket
+sums, color-dependent per EmptyAttackedSquare's own existing asymmetry
+(WHITE's front is slots 0/1/2, BLACK's is 5/6/7 -- see
+KingRingBucket-equivalent logic below) -- these six raw per-slot
+coefficients were the noisiest, most sign-flipping entries in every
+full-vs-top-tercile drift table, and bucketing trades a little resolution
+for a lot more stability at the sample sizes actually available here.
+`filestorm` itself is dropped as a feature in favor of its two already-
+independent sub-components (filestormopenfile, filestormpawn) which map
+directly onto separate hand-picked tables in eval.c (KingFileDefects /
+KingStormingPawnDefects) -- no reason to force a regression to re-derive
+their relative importance from one pre-summed number.
+"""
+import argparse
+import csv
+import sys
+
+import numpy as np
+from sklearn.linear_model import LogisticRegression
+from sklearn.model_selection import train_test_split
+from sklearn.preprocessing import StandardScaler
+from sklearn.metrics import roc_auc_score, accuracy_score
+
+PIECE_NAMES = ("pawn", "minor", "rook", "queen")
+PER_SQUARE_KEYS = ["offboard"] + [f"atk_{p}" for p in PIECE_NAMES] + \
+ ["undef", "def", "xray", "flight"]
+
+SIDE_SLOTS = (3, 4) # kept individually resolved
+# Color-dependent diagonal/vertical bucket membership, mirroring
+# EmptyAttackedSquare's own asymmetry (chess.h: WHITE=1, BLACK=0).
+BUCKET_SLOTS = {
+ "white": {"front": (0, 1, 2), "back": (5, 6, 7)},
+ "black": {"front": (5, 6, 7), "back": (0, 1, 2)},
+}
+
+FEATURE_COLUMNS = []
+for _sq in SIDE_SLOTS:
+ for _k in PER_SQUARE_KEYS:
+ FEATURE_COLUMNS.append(f"sq{_sq}_{_k}")
+for _bucket in ("front", "back"):
+ for _k in PER_SQUARE_KEYS:
+ FEATURE_COLUMNS.append(f"{_bucket}_{_k}")
+FEATURE_COLUMNS += ["filestormopenfile", "filestormpawn", "queenproximity",
+ "initloc", "castled", "minorsathome",
+ "misscastleshort", "misscastlelong", "attackerarmyscaler"]
+
+# Reference-only columns: the engine's OWN current derived output for this
+# position, kept for comparison but never fed into the model as inputs.
+REFERENCE_COLUMNS = ["countertotal", "safetyvalue", "filestorm"]
+
+
+def _row_to_features(r):
+ color = r["mated_color_or_sampled_color"]
+ bucket_slots = BUCKET_SLOTS[color]
+ out = []
+ for sq in SIDE_SLOTS:
+ for k in PER_SQUARE_KEYS:
+ out.append(float(r[f"sq{sq}_{k}"]))
+ for bucket_name in ("front", "back"):
+ slots = bucket_slots[bucket_name]
+ for k in PER_SQUARE_KEYS:
+ # Sum across the 3 slots in this bucket -- offboard included,
+ # since a king near the edge legitimately has fewer on-board
+ # ring squares and that's itself signal, not noise to hide.
+ out.append(sum(float(r[f"sq{s}_{k}"]) for s in slots))
+ for c in ["filestormopenfile", "filestormpawn", "queenproximity",
+ "initloc", "castled", "minorsathome",
+ "misscastleshort", "misscastlelong", "attackerarmyscaler"]:
+ out.append(float(r[c]))
+ return out
+
+
+def load_data(csv_path):
+ with open(csv_path, newline="") as f:
+ reader = csv.DictReader(f)
+ rows = list(reader)
+ y = np.array([int(r["label"]) for r in rows], dtype=int)
+ X = np.array([_row_to_features(r) for r in rows], dtype=float)
+ ref = {c: np.array([float(r[c]) for r in rows]) for c in REFERENCE_COLUMNS}
+ return X, y, ref
+
+
+def fit_and_report(X, y, label, top_n=25):
+ scaler = StandardScaler()
+ X_scaled = scaler.fit_transform(X)
+ X_train, X_test, y_train, y_test = train_test_split(
+ X_scaled, y, test_size=0.2, random_state=42, stratify=y)
+
+ clf = LogisticRegression(max_iter=2000, C=1.0)
+ clf.fit(X_train, y_train)
+
+ y_pred = clf.predict(X_test)
+ y_prob = clf.predict_proba(X_test)[:, 1]
+ acc = accuracy_score(y_test, y_pred)
+ auc = roc_auc_score(y_test, y_prob)
+
+ print(f"\n=== {label} (n={len(y)}, train={len(y_train)}, test={len(y_test)}) ===")
+ print(f"Test accuracy: {acc:.4f} Test AUC: {auc:.4f}")
+
+ # Standardized coefficients are directly comparable to each other
+ # (same units: change in log-odds per one std-dev of that feature)
+ # regardless of the feature's raw scale (a 0/1 flag vs. armyscaler's
+ # 0-62 range) -- that comparability is the entire point of this
+ # exercise, so report these, not raw unscaled coefficients.
+ coefs = clf.coef_[0]
+ order = np.argsort(-np.abs(coefs))
+ print(f"\nTop {top_n} features by |standardized coefficient| "
+ f"(positive = more mate-predictive):")
+ for i in order[:top_n]:
+ print(f" {FEATURE_COLUMNS[i]:28s} {coefs[i]:+.4f}")
+
+ return dict(zip(FEATURE_COLUMNS, coefs)), scaler, clf
+
+
+def main():
+ ap = argparse.ArgumentParser()
+ ap.add_argument("features_csv")
+ ap.add_argument("--top-n", type=int, default=25)
+ args = ap.parse_args()
+
+ X, y, ref = load_data(args.features_csv)
+ print(f"Loaded {len(y)} rows, {X.shape[1]} features, "
+ f"{y.sum()} positives / {(y == 0).sum()} controls", file=sys.stderr)
+
+ print("\n--- Reference: current engine's own derived score ---")
+ for c in REFERENCE_COLUMNS:
+ pos_mean = ref[c][y == 1].mean()
+ ctl_mean = ref[c][y == 0].mean()
+ print(f" {c}: mean(mate)={pos_mean:.1f} mean(control)={ctl_mean:.1f} "
+ f"ratio={pos_mean / max(ctl_mean, 1e-9):.2f}x")
+
+ full_coefs, _, _ = fit_and_report(X, y, "Full population", top_n=args.top_n)
+
+ # Attacker-material confound check: refit on the top tercile of
+ # attacker army strength only, where REDUCED_MATERIAL_DOWN_SCALER is
+ # near its ceiling and roughly constant across the subset -- if
+ # coefficients move a lot vs. the full-population fit, the raw defect
+ # weights aren't material-invariant and the interaction should be
+ # modeled explicitly rather than left as a plain covariate.
+ army_idx = FEATURE_COLUMNS.index("attackerarmyscaler")
+ army_vals = X[:, army_idx]
+ top_tercile_cutoff = np.percentile(army_vals, 66.7)
+ mask = army_vals >= top_tercile_cutoff
+ print(f"\nTop-tercile army strength cutoff: attackerarmyscaler >= "
+ f"{top_tercile_cutoff:.0f} ({mask.sum()} rows)", file=sys.stderr)
+ strong_coefs, _, _ = fit_and_report(
+ X[mask], y[mask], "Top-tercile attacker army strength", top_n=args.top_n)
+
+ print(f"\n=== Coefficient drift: full population vs. top-tercile army strength ===")
+ print(f"(large drift on a feature means its true weight is NOT "
+ f"material-invariant -- REDUCED_MATERIAL_DOWN_SCALER alone may "
+ f"not be enough to capture that interaction)")
+ drift = [(c, full_coefs[c], strong_coefs.get(c, 0.0),
+ abs(full_coefs[c] - strong_coefs.get(c, 0.0)))
+ for c in FEATURE_COLUMNS if c != "attackerarmyscaler"]
+ drift.sort(key=lambda t: -t[3])
+ for c, fv, sv, d in drift[:args.top_n]:
+ print(f" {c:28s} full={fv:+.4f} top_tercile={sv:+.4f} drift={d:.4f}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/ECM.md b/tests/ECM.md
new file mode 100644
index 0000000..5ff1ed0
--- /dev/null
+++ b/tests/ECM.md
@@ -0,0 +1,246 @@
+# ECM position notes
+
+Per-position metadata accumulated while investigating solve-rate losses on
+the curated suites (`ecm_ringers.ep_`, `ecm_confident_quick.ep_`,
+`ecm_hard_quick.ep_`) during the 2026-09-10/11 extension-taper/qsearch-prune
+session. IDs match the `id "ECM.###"` labels `script`/`suite_diff.pl` print
+(note: due to how the `.ep_` files pipeline `id`/`go`/`setboard`/`solution`
+-- each `id` line is followed by the *previous* block's `setboard`/
+`solution`, not the one textually after it -- so don't assume `grep`ing the
+raw `.ep_` file for an id gives you the matching FEN without checking this).
+
+## Methodology
+
+A position's HEAD-vs-candidate solve flip was investigated by:
+
+1. Reading HEAD's own `sd10` log for that position: does the PV settle on
+ one move with a stable or improving score across iterations, ideally
+ with a `++`/`+` confirmed-fail-high mark? Or does it flip between
+ different candidate moves, land on an unconfirmed score at the very
+ last iteration (especially if it abandons an already-*confirmed*
+ alternative), or return a score suspiciously close to `0.00`/negative
+ for the side that's supposedly winning?
+2. Where available, an independent second opinion: Crafty's own search
+ (`sd 15`-`sd 20`) on the same position, and/or typhoon-candidate's own
+ result at a deeper fixed depth (`sd 15`) than the suite's `sd 10`.
+3. A position only counts as **legit** if HEAD's own read was solid *and*
+ nothing independent contradicts it. A single wobble or disagreement is
+ enough to call it **flaky**.
+
+This is a real, if informal, signal -- not a substitute for actually
+re-running the suites when a change lands, just a way to avoid re-deriving
+"is this specific solve-count delta noise" from scratch every time a
+similar-shaped loss reappears (which it does -- most of the positions below
+have shown up as a flip across *multiple*, otherwise-unrelated changes this
+session).
+
+## Legit (real tactic, HEAD understood it solidly -- a genuine loss if lost)
+
+| ID | Suite | FEN | Solution | Why legit |
+|---|---|---|---|---|
+| ECM.035 | confident_quick | `2r2bk1/4qp2/3n2p1/2R1p1Np/2p1N3/r6P/1Q3PP1/3R2K1 w - -` | Rxc8 | HEAD confirms (`++`, +2.31) before finalizing at +2.79; Crafty's independent deep search converges to ~+4.6; candidate's own `sd 15` also confirms |
+| ECM.080 | confident_quick | `5r1k/pb2r1bp/1p2B1p1/n7/4qNQ1/4B3/P4PPP/2RR2K1 w - -` | Rd4 | Single clean line, +3.26, fast solve, no oscillation |
+| ECM.128 | hard_quick | `rn2qbr1/2p4k/p2p1nb1/1p1Pp2p/2P4P/2NBBPN1/PP1Q4/2KR2R1 w - -` | Nxh5 | Single clean line, +2.66, no oscillation |
+| ECM.071 | hard_quick | `b1n1r1k1/2q2p1p/p4Pp1/1pBP4/4B1P1/2P1Q3/P4K1P/4R3 w - - 0 0` | Qh6 | Confirmed (`++`, +3.34) at depth 9, climbs monotonically to +4.74 by depth 12, same move throughout. Crafty independently finds the position winning even bigger (+7-9) but via a different move (`Kg2`, a quiet consolidation) -- doesn't contradict `Qh6`'s own soundness, just suggests it isn't the single objectively-best try. Gained by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.250 | confident_quick | `2r1rbk1/5pp1/bq5p/1pnBPN2/6Q1/N7/5PPP/R2R2K1 w - -` | Bxf7+ | Single clean line, +3.31, fast solve |
+| ECM.251 | hard_quick | `1k1r1r2/2b2R2/Bppp2p1/2q4p/Q2pP3/P2P4/2P3PP/1R4K1 w - -` | Bb7 | Instant solve; Crafty's own deep search independently plays Bb7 too (+1.87) |
+| ECM.445 | confident_quick | `7r/2p1q1k1/1p3r1p/p1pPpPp1/2P1P1P1/PP6/3Q2KR/7R w - -` | Qxg5+ | HEAD never solves this at `sd10` at all (only flips under `sn5m`); Crafty confirms the move is objectively good and stable (+2.31 to +2.35 across depths 18-20) |
+| ECM.519 | confident_quick | `1k1r2r1/p1p3q1/2p2p2/3pn3/1P1b2bB/1QN5/P4PPP/1R2RBK1 b - -` | Nf3+ | Huge, unambiguous score (+7.33), no wavering at all; candidate's own deeper search confirms and strengthens it |
+| ECM.147 | confident_quick | `r2b1rk1/5qp1/2p2P2/2p3Pp/p6P/2Q5/PPP5/2KR2R1 w - - 0 0` | g6 | HEAD's own line switches ideas mid-search (Qxc5 -> g6) but the new idea gets its own `++` confirmation (depth 10, +0.75) before the final unconfirmed bump to +0.89; Crafty independently plays the same `g6` and stably converges to +0.49-0.66 across depths 19-22. Thin margin, but move-choice agreement from an independent engine is the deciding evidence -- lost by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.672 | confident_quick | `5rk1/pbrn1ppp/1ppN1q2/2P1p3/3P4/1PRB4/P3QPPP/5RK1 w - - 0 0` | Bxh7+ | HEAD confirms (`++` twice, +4.09/+5.59) with the same first move stable from depth 7 on, score climbing monotonically to +6.22 at depth 10; Crafty independently plays the same move, reaching +9.5 by depth 27. Lost by the candidate LMR-graded-curve build (2026-09-11) -- notably, the *same candidate binary* run to deeper iterative depths (not suite-capped `sd10`) does eventually find `Bxh7+` too, jumping from +2.46 (depth 12, still on a `cxb6`-based line) to +7.59/+7.90 at depths 15-16. Not a permanent blind spot: the graded curve over-reduces this specific sacrifice line early enough to delay discovery by ~7-8 plies past what `sd10` allows, not miss it outright |
+| ECM.721 | ringers | `4rrk1/2pn2pb/p1p1qp2/1pb1pN2/P3P1PN/1P1P4/1BP2PK1/R2Q3R w - -` | Nxg7 | Clean, unwavering, +3.19; candidate confirms/strengthens at deeper depth (+4.29) |
+| ECM.733 | confident_quick | `2r3k1/pbq1np1p/1p1b1Bp1/8/6QP/2P2N2/B4PP1/4R1K1 w - -` | Bxf7+ | HEAD confirms once (`++`, +1.52); same first move stable across all entries, only the deeper continuation changes -- stronger signal than a plain unconfirmed line. Reconfirmed by Crafty (stable `Bxf7+`, +4.2-4.4, depths 18-22) during the 2026-09-11 LMR graded-curve investigation, which also lost this position |
+| ECM.736 | confident_quick | `r2r2k1/1b2qpp1/1p2n3/p1b1p2p/4P1n1/PP1B2PP/1B1NQP2/R3NRK1 b - - 0 0` | Nxf2 | HEAD abandons a confirmed `Nd4` (`++`, +1.51) for `Nxf2`, unconfirmed and briefly dipping (+2.05 -> +1.93) -- looked flaky from HEAD's log alone. Crafty independently and stably plays the same `Nxf2` across 5 straight iterations (depths 24-28, -1.43 to -1.90 from Black's perspective), confirming the move choice despite HEAD's own wobble. Lost by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.734 | hard_quick | `r3r1k1/1bqn1ppp/1pp2p2/8/3P4/1B4N1/PP3PPP/R2QR1K1 w - -` | Bxf7+ | Same shape as ECM.733: confirmed once (`++`, +1.74), same top move stable throughout even as the deeper line's score settles lower |
+| ECM.789 | hard_quick | `2r1r1k1/pb1n1pp1/1p1qpn1p/4N1B1/2PP4/3B4/P2Q1PPP/3RR1K1 w - - 0 0` | Bxh6 | Never gets an explicit `++` mark but the move is stable across the last 3 iterations with the score climbing (1.06 -> 1.27 -> 1.69). Crafty independently confirms the same move, also growing (1.89 -> 2.06 -> 2.13). Gained by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.846 | hard_quick | `3qr1k1/rp3pbp/2p3p1/3p1b2/N1nP4/P2NP3/1PQ2PPP/1BRR2K1 b - -` | Rxe3 | **Reclassified from Flaky (2026-09-11)**: this entry previously read "two unconfirmed lines, both under +0.5" from an earlier build. The candidate LMR-graded-curve build, run to real (non-node-capped) depth, stabilizes firmly on `Rxe3` from depth 11 through 15 (+0.31 -> +0.67, no further wobble). Crafty independently and stably confirms the same move across depths 25-30 (-0.82 to -1.12, its own sign convention). Two-engine, deep, stable agreement overturns the old flaky call -- the earlier read was likely just a shallower/different build catching this position mid-wobble before it settles |
+| ECM.735 | hard_quick | `r3r1k1/pp3ppp/8/1Nbp3q/3NnBn1/2P5/PPQ2PPP/3R1RK1 b - -` | Ngxf2 | HEAD's own line is clean and solid; Crafty's defense uses the *other* knight (Nexf2) but that's a "which piece" quibble, not a refutation of the idea |
+| ECM.803 | hard_quick | `2b2r1k/2p3pp/2Nb4/pP5q/2PP4/P4pP1/5P1P/R1BQ2K1 b - -` | Bxg3 | HEAD's line is clean; Crafty independently finds **mate in 10** starting with Bxg3 -- about as solid as confirmation gets |
+| ECM.442 | hard_quick | `4qn1k/1b3rpp/pb2pB2/1pp1P2P/3p1NQ1/P2P2PB/1PP4K/5R2 w - -` | Ng6+ | Not actually a miss -- the king-file-storm-defect candidate (2026-09-14) settles on `Bh4` at `sd10`, but its own deeper iterations (depth 14-15) find and `++`-confirm `Ng6+` (+1.94, climbing to +2.47 by depth 16). A depth-budget delay, not a blind spot; see the 2026-09-14 session note below |
+
+## Flaky / garbage (suite noise -- don't read a loss here as a real regression)
+
+| ID | Suite | FEN | "Solution" | Why flaky |
+|---|---|---|---|---|
+| ECM.213 | ringers | `k3r3/pR5p/PppR1p1p/4nP2/1PP5/8/4B1rP/2K5 w - -` | Re7 | Came off a draw-by-repetition line (`Kd1 ... <=>`) at only +0.37, unconfirmed |
+| ECM.270 | ringers | `6k1/1p3p2/4r1p1/1PPrbq1p/2QN4/P2R1P1P/R4KP1/8 b - -` | Bg3+ | Score exactly `0.00`, single unconfirmed line; Crafty's own `sd20` search prefers Rxc5 instead |
+| ECM.336 | hard_quick | `r1bqr1k1/pp3pp1/2p4p/3PN1n1/3P1b2/2NB4/PPQ2PPP/3R1RK1 b - -` | Rxe5 | Abandons a *twice*-confirmed cxd5 (swung -0.81 -> -0.16 -> +1.34) for an unconfirmed Rxe5 at the very end |
+| ECM.337 | confident_quick | `r3r1k1/pbqn1ppp/1p3n2/1Pb5/2p5/5N1P/PBQ1BPP1/1N1R1RK1 b - -` | Rxe2 | Cycles through 3 different first moves within depth 10, lands unconfirmed |
+| ECM.340 | confident_quick | `1k2r2r/1bpQN2p/1p6/P3p2q/4np2/2P2N2/2P2PPP/R3R1K1 b - -` | Rxe7 | Abandons a *thrice*-confirmed Qh6 (+2.49) for a near-identical unconfirmed Rxe7 (+2.76) |
+| ECM.370 | hard_quick | `1r2r1k1/1pqb2pp/p2bp1n1/2p3BQ/3pN3/1P1P2P1/P1P3BP/4RRK1 w - -` | Bd8 | Score exactly `0.00` on a repetition line; Crafty prefers a different move at only +0.15 |
+| ECM.443 | hard_quick | `2b5/2qrrpk1/5Rp1/2p4p/1pB1PR1P/1P1P2P1/5Q1K/8 w - -` | Qb2 | Two unconfirmed lines, thin margins (+0.92, +1.10) |
+| ECM.637 | hard_quick | `r3qr1k/1b4pp/p2RQ3/Bp2p3/4p3/6P1/PPP2P1P/3R2K1 b - -` | Qh5 | Swapped a confirmed e3 (+2.18) for an unconfirmed Qh5 (+2.19) -- a coin-flip; Crafty's own eval also wobbled between -2.33 and -0.80 across nearby depths |
+| ECM.715 | confident_quick | `3r1rk1/5pp1/pq1n1n1p/2pPR3/2B2P2/1PBQ2RP/P5P1/6K1 w - -` | Rxg7+ | Abandons a *twice*-confirmed Rge3 (+4.41) for a lower-scoring, unconfirmed Rxg7+ (+3.05) |
+| ECM.771 | hard_quick | `3qr1k1/1br1bp1p/p3p1pB/1p1nN3/3PB3/7Q/PP3PPP/3RR1K1 w - -` | Nxf7 | HEAD never breaks +0.56, unconfirmed; Crafty's independent top choice is a different move near dead equal |
+| ECM.813 | confident_quick | `r2q1rk1/pp2ppb1/3pn1pQ/3R4/2P3B1/4BR2/PP4PP/6K1 w - -` | Qxg6 | Both of HEAD's candidate lines score *negative* (-0.74, -0.69) -- HEAD's own eval doesn't think its own "solution" is good for the side supposedly winning |
+| ECM.422 | hard_quick | `6k1/rnqb3p/5ppQ/2pPp3/p1N1P3/2PB3P/5PP1/1R4K1 w - - 0 0` | f4 | Same move (`f4`) held across 5 iterations but never gets a `++` confirm, thin margin (+0.70 to +0.95). Crafty independently prefers a different move (`Qh4`) at a similarly thin margin (+0.68 to +1.06) -- a genuine disagreement on a razor-thin position, not corroboration. Gained by the candidate LMR-graded-curve build (2026-09-11) but not a confident real solve |
+| ECM.830 | confident_quick | `1n2r3/p1pq1kp1/1b1pNpp1/3P4/5RP1/3Q3P/1B3P2/6K1 w - -` | Bxf6 | Oscillates Nxg7/Bxf6 the whole way, no confirmation either side |
+| ECM.842 | confident_quick | `r1b1k1nr/3n1p1p/1qpBp1p1/pp2b3/4PN2/PBN2Q2/1PP2PPP/2KR3R w kq - 0 0` | Nxe6 | HEAD abandons a confirmed `Bxe6` (`++`, +1.14) for an unconfirmed `Nxe6`, never reconfirmed. Crafty's own search at `sd 22` doesn't corroborate either -- wobbles between `Qg3` and `Bxe6` across depths 20-22, never playing `Nxe6`. Independent check does not support HEAD's answer, unlike ECM.147/672/736 below -- lost by the candidate LMR-graded-curve build (2026-09-11) but stays classified flaky |
+| ECM.875 | hard_quick | `2kr1br1/ppp2p1p/2n1p1q1/2Nn4/3P4/1Q2PB2/PP1B1P1P/2R1K2R b K - 0 0` | Qg1+ | Score exactly `0.00` on a repetition line (`<=>`) across 6 straight iterations (depths 8-13) -- textbook flaky signature. Contradicts the "Gains seen" table's loose note calling this Crafty-confirmed legit in an earlier review; that note appears to have been wrong, or about a different search reaching a different (non-drawing) conclusion. Gained by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.879 | hard_quick | `r2r2k1/pp3ppp/2q5/2p5/2P1P1n1/P4P1b/1BQ1BP1P/R3R2K b - - 0 0` | Rd2 | Score exactly `0.00` on a repetition line (`<=>`) -- same signature as ECM.875. Gained by the candidate LMR-graded-curve build (2026-09-11) |
+| ECM.866 | hard_quick | `7R/1p1r4/2b2p2/2Pp1qk1/3P1bp1/3N4/2Q2PB1/6K1 w - -` | Rg8+ | HEAD's own score is negative (-0.12 to -0.15); Crafty's independent top choice (Qd1) is also near-zero. **Note**: has shown up as a *gain* (candidate solves it, HEAD doesn't) in at least one later run too -- consistent with a genuinely coin-flip position, not a stable read either direction |
+
+## Gains seen (candidate solves, HEAD doesn't) -- not deeply vetted
+
+These showed up as improvements across various runs. Only spot-checked, not
+put through the full methodology above -- listed for reference, not as a
+confirmed-legit set the way the "Legit" table above is.
+
+| ID | Suite | Notes |
+|---|---|---|
+| ECM.193 | hard_quick | Crafty-confirmed legit in earlier review |
+| ECM.476 | hard_quick | Crafty-confirmed legit in earlier review |
+| ECM.544 | hard_quick | Crafty-confirmed legit in earlier review |
+| ECM.613 | hard_quick | Seen as a gain across multiple runs |
+| ECM.716 | confident_quick | Seen as a gain across multiple runs |
+| ECM.779 | hard_quick | Seen as a gain across multiple runs |
+| ECM.822 | hard_quick | Seen as a gain across multiple runs |
+| ECM.875 | hard_quick | Seen as a gain across multiple runs |
+| ECM.489 | hard_quick | Seen once |
+
+## 2026-09-11 LMR graded-curve session
+
+A separate investigation replaced typhoon's flat `-ONE_PLY` LMR reduction
+with a graded depth/move-number curve (fit to this engine's own measured
+tail-fail-high-rate data, not a borrowed formula -- see `dynamic.c`'s
+`InitLMRTable`), plus a QSearch-drop guard in `GetLMRReduction` (never
+reduce far enough to land below `THREE_QUARTERS_PLY` remaining depth).
+Measured against `head_reference` (`ddf932b`) on `confident_quick`:
+`sd10` net -4/+2 (net -3 across all three suites combined), `sn5000000`
+net -3/+3 depth-adjusted (net +4 combined, `hard_quick` alone +7 with
++0.5 extra ply -- a real depth-payoff signal).
+
+Re-investigated all 6 `confident_quick` losses (identical set at both
+`sd10` and `sn5m` -- a reproducible effect, not scattered noise) and both
+gains against `head_reference`'s own log plus a fresh independent Crafty
+check (`sd 22`, not run through the full methodology before this
+session). Net result: **4 of 6 losses are legit** (733, 672, 147, 736 --
+147 and 736 were previously unclassified and looked flaky from HEAD's
+log alone, but Crafty's independent move-choice agreement overturned
+that read), only 2 are flaky (715, 842). Both gains (519, 035) are
+already-established legit. That's a real, non-noise net-negative
+tactical cost on `confident_quick` specifically -- the aggregate
+suite-level numbers understated this at first glance (`sd10` -4 raw
+looked mostly explainable as noise until the per-position Crafty check
+was actually run).
+
+ECM.672 is worth remembering for future LMR work specifically: the same
+candidate binary, given enough iterative depth (not suite-capped
+`sd10`), *does* eventually find the correct `Bxh7+` sacrifice (depth
+15-16, jumping from +2.46 to +7.59/+7.90) after several iterations
+spent on an unrelated `cxb6`-based line. The graded curve isn't blind to
+this tactic, it's delaying discovery by roughly 7-8 plies past what a
+depth-10 budget allows -- a genuine cost at fixed/shallow search depths,
+but not the kind of permanent blind spot a bad exemption rule would
+produce. Worth checking whether this delay is concentrated in
+sacrifice/attacking lines specifically (where the "safe to reduce"
+heuristics this curve is built from may be least trustworthy) before
+concluding the curve needs a different shape rather than just deeper
+search to pay off.
+
+**`hard_quick`'s `sn5000000` gains investigated too** (11 gains, 4 losses
+-- 193, 476, 779, 866, not re-investigated here since they're losses on
+the suite that was already net-positive). Of the 11 gains: **7 are legit**
+(071, 251, 734, 735, 789, 803, 846 -- 846 is a reclassification, see its
+updated Legit-table entry above: an earlier build's shallower read
+tagged it flaky, but this candidate's deeper search plus Crafty both
+stably converge on the same move), **3 are flaky** (422 -- genuine
+engine-vs-engine disagreement on a thin margin, not corroboration; 875,
+879 -- both exact `0.00` on a repetition line), and **1 is genuinely
+unresolved** (640): the candidate's own extended search twice-confirms
+`Rxb2` at +3.66 (the `sn5m`-capped run's PV/played-move mismatch was
+just the node budget cutting off *before* this switch, not a bug), but
+Crafty independently searches to depth 30 and never once prefers
+`Rxb2`, staying on a different move (`Bd8`) near dead equal. Two
+real-depth searches disagreeing after real depth is a different, more
+interesting kind of uncertainty than the usual flaky signature (score
+exactly 0, unconfirmed thin margins) -- worth a dedicated line-by-line
+look at the `Rxb2` continuation before counting it either way.
+
+**Combined verdict across both suites, real (non-flaky) positions only**:
+9 legit gains (035, 519, 071, 251, 734, 735, 789, 803, 846) vs. 4 legit
+losses (733, 672, 147, 736), with ECM.640 unresolved. A materially more
+favorable picture than the raw suite aggregates suggested on their own
+-- this is why the per-position Crafty-corroboration check matters more
+than trusting `sd10`/`sn5000000` solve counts at face value, especially
+at this suite size.
+
+## 2026-09-14 king-safety file-storm-defect session
+
+Retuned `eval.c`'s `KingFileDefects`/`KingStormingPawnDefects` tables
+(inside `_ComputeFileStormDefects`) per `eval_tune/king_safety_data/HANDOFF.md`'s
+tier-1 finding: `{+2,+1,+0}` -> `{+4,+2,+0}` and
+`{+0,+4,+3,+1,+0,+0,+0,+0}` -> `{+0,+7,+5,+2,+0,+0,+0,+0}`, a real-game
+mate-mining logistic regression's two most dominant, most stable
+predictors, previously weighted the same as every other hand-picked
+king-safety defect. Measured against `head_reference` (`ddf932b`) at
+`sd10`: `ecm_ringers` -2, `ecm_confident_quick` -1, `ecm_hard_quick` -1.
+
+Per-position investigation (same methodology as the LMR session above)
+found almost none of this is real:
+- `ECM.715`, `ECM.866` (losses) and `ECM.813`, `ECM.443`, `ECM.771`
+ (gains) were already tagged flaky above.
+- `ECM.716`, `ECM.193`, `ECM.476` (losses) were already listed in the
+ "Gains seen" table from other sessions -- a position that has already
+ flip-flopped direction before isn't new evidence of a regression.
+- `ECM.501`, `ECM.801` (ringers losses, previously unclassified): both
+ show HEAD *itself* wobbling between candidate moves and landing on its
+ final answer only at the last iteration with no `++` confirmation
+ (`ECM.501`: Rf7 -> Rc8 -> Rf7 -> Rb8 -> Nxf3; `ECM.801`: Nd4 held
+ stably for 3 iterations then an unconfirmed jump to Nh6+ only at
+ depth 10) -- textbook flaky signature, not a real loss either
+ direction.
+- `ECM.442` (hard_quick loss): not a miss at all -- see its updated
+ entry in the Legit table above. The candidate's own deeper iterations
+ (depth 14-15) find and `++`-confirm the same `Ng6+` HEAD plays at
+ `sd10`; the `sd10` "loss" is purely a depth-budget delay (climbing
+ from +0.91 at depth 10 to a confirmed +1.94 by depth 14), the same
+ shape as `ECM.672`'s entry above.
+- `ECM.298` (hard_quick loss, previously unclassified): HEAD converges
+ cleanly on `Re6`/`Rf6` (+2.7 to +2.9) and the suite's labeled solution
+ is `Re6`, but the candidate instead finds and `++`-confirms `h4`
+ (+5.49 by depth 11, climbing to +8.87 by depth 17) -- and Crafty's own
+ independent deep search (`sd 27`, ~9.3s to switch onto it after
+ spending depths 20-23 stuck on `Rf6` at only +2.9-3.2) agrees `h4` is
+ objectively much stronger (+10.17) than the suite's labeled answer.
+ Notably the candidate found this line in 2.5s vs. Crafty's 9.3s.
+ Not a regression -- if anything, evidence the candidate is finding a
+ better resource than the suite's own labeled solution, faster than an
+ independent reference engine. Worth relabeling this suite's expected
+ answer if it's ever revisited, rather than trusting `Re6` as ground
+ truth.
+
+**Net read**: of the 4 nominal net solve losses (-2/-1/-1 across the
+three suites), 7 of 9 individual position losses were already-known or
+freshly-confirmed noise, one (`ECM.442`) is a depth-budget artifact, and
+the last (`ECM.298`) is arguably a candidate improvement mislabeled as a
+loss by the suite's fixed expected answer. No surviving evidence of a
+real tactical regression from this change at the position level, despite
+the suite-aggregate solve counts looking mildly negative -- another data
+point for not trusting raw `sd10` solve deltas on suites this size
+without the per-position check.
+
+Self-play (`match_play.py`, ongoing) at 110 games: -29 =47 +34
+(score 0.523, llr=+0.11, still far from either SPRT bound) -- too early
+to call, but notably `avg depth: candidate=9.96 head=10.25` (candidate
+searching ~0.3 ply shallower at the same `st` budget) with
+`avg ebf: candidate=4.442 head=4.396` (branching factor slightly
+*higher*, i.e. the tree got slightly fatter, not narrower) -- a
+non-trivial shape change in the search tree from what was purely an
+eval-weight retune around king safety, presumably via move-ordering/
+pruning decisions that read king danger differently now (e.g. anything
+gated on `KING_SAFETY_BY_COUNTER`'s bucket or a null-move/LMR
+verification threshold sensitive to eval score swings). Worth keeping
+in mind for future king-safety retunes: even a "just the numbers"
+change to an eval sub-term can visibly shift average search depth, not
+just leaf-node scores.
+
+## Caveat
+
+This list is a snapshot from one session's worth of investigation, not a
+permanent classification -- a position flagged "flaky" here reflects HEAD's
+behavior *at the commit this session started from* (`9d1e32d`) plus
+whatever candidate builds were tested through it. If the eval or search
+changes significantly later, re-derive rather than trust this table blindly
+-- especially the "Gains" table, which was never run through the full
+methodology in the first place.