1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
# King-safety defect regression: handoff
## Goal
`_EvalKing` (eval.c) sums a pile of hand-picked "defect" constants into one
scalar (`uCounter`), then runs that through a single lookup table
(`KING_SAFETY_BY_COUNTER`). The hand-picked weights assume every defect
counts equally, which is suspect -- e.g. a queen touching an undefended
ring square and a knight touching one add the exact same `+1`. This
project mines real games to see which defects actually predict getting
mated, so `eval.c`'s weights can be informed by data instead of guesswork.
**Method**: find real middlegame checkmates in human games, look at the
losing king's position N plies earlier, and compare its raw defect
pattern against a matched "sound" control position (drawn games) with the
same attacker material and development level. Fit a logistic regression;
features with a real, stable coefficient are candidates for reweighting
in `eval.c`. See `CLAUDE.md`'s "King-safety eval improvement project"
section (search for it) for the full narrative this file summarizes.
## Where everything is
- **Final training data**: `remine_v4/king_features_matched_v4.csv` --
86,362 rows (43,181 matched positive/control pairs). **Use this one.**
- `remine_v4/merged.csv` -- raw mined FENs before feature extraction
(172,735 rows: 43,181 positives + 129,554 control candidates).
- `remine_v4/king_features_v4.csv` -- all 172,735 rows with full
`_EvalKing` defect breakdown, before matching trims it down to the
86,362 confound-controlled rows.
- Pipeline scripts live in `eval_tune/` (sibling of this dir), not here:
`mine_king_mates.py`, `shard_mine.py`, `extract_king_features.py`,
`match_controls.py`, `train_king_safety_model.py`.
- Engine side: `chess.h`'s `KING_DEFECT_DUMP_INFO` struct,
`_EvalKing`'s `#ifdef KING_DEFECT_DUMP` instrumentation in eval.c, and
command.c's `evalkingdump` command -- all gated behind the
`KING_DEFECT_DUMP=1` gmake build flag (NOT part of a normal build).
`../typhoon_kingdump` is a saved binary built with that flag; rebuild
it (`gmake clean` first -- flag changes don't trigger recompilation on
their own, see "Gotchas" below) if `eval.c`/`chess.h`/`command.c`
change.
## Final result (from `king_features_matched_v4.csv`)
Logistic regression, 80/20 train/test split, standardized coefficients.
Test AUC 0.91 (plausible for "engineered danger features vs. proven-sound
draws", not suspiciously perfect). Ranked by |coefficient|, most-to-least
trustworthy tier first (stability = full-population vs. top-tercile-army-
strength fits agreeing):
**Tier 1 -- dominant, very stable**: `filestormopenfile` (+0.57 to +0.80),
`filestormpawn` (+0.51 to +0.54). These are eval.c's `KingFileDefects[3]`
and `KingStormingPawnDefects[8]` tables (open/half-open file near the
king; how far an enemy pawn has advanced up that file), already
independent tables -- lowest-risk place to start a real retune.
**Tier 2 -- strong, stable**: `front_offboard`, `front_def`, `sq4_offboard`,
`front_undef`, `front_atk_minor`, `front_atk_rook`, `misscastleshort`,
`front_flight` (+0.33 to +0.45). "front" = the diagonal/vertical ring
squares one rank away, in the direction each side attacks toward (see
"Feature schema" below for the exact slot mapping -- do NOT assume this
means "in front of the king" the way a human would first guess; see the
correction note there). `sq3`/`sq4` are the two same-rank side squares,
kept individually resolved.
**Tier 3 -- moderate, less stable**: `back_offboard` (drifts the most of
any surviving feature, treat with suspicion), `sq3_undef`,
`attackerarmyscaler`, `queenproximity`, `back_atk_minor`, `front_atk_queen`.
**Confirmed noise -- do not retune based on this data**: `minorsathome`
(coefficient ~0.001, was previously the #1 "signal" before two sampling
bugs got fixed -- see "Bugs found" below), `initloc`, `castled` (exactly
0.0000, fully collinear with `misscastleshort`/`misscastlelong`).
## Translating this into eval.c -- confidence and plan
This is NOT a "throw away eval.c and use the model" situation. The plan
is targeted retuning of existing constants/tables, informed by relative
coefficient *ratios*, not literal log-odds-to-centipawn conversion:
1. **`KingFileDefects`/`KingStormingPawnDefects` deserve a real weight
increase relative to everything else** -- they're the strongest
signal by a clear margin and already live in their own small,
isolated tables (`_ComputeFileStormDefects` in eval.c). Lowest risk,
highest expected payoff, start here.
2. **`EmptyAttackedSquare`/`EmptyUnattackedSquare`'s front/back asymmetry
is directionally right but probably too weak a spread** -- front-
bucket features consistently outrank back-bucket ones here by a good
margin.
3. **`misscastleshort`'s constant likely deserves a bump.**
4. **Don't touch `KING_INITIAL_COUNTER_BY_LOCATION` or the undeveloped-
minors term** based on this data -- both came back as noise.
5. Neither `KingFileDefects`/`KingStormingPawnDefects` nor the
ring-scan tables are currently `DNA_ARRAY`-registered (checked: only
`KING_SAFETY_BY_COUNTER` and `KING_QUEEN_PROXIMITY_DANGER` are under
king-safety). Retuning them means a manual edit (optionally promoting
them to file scope + registering as DNA arrays for future tuning),
not "rerun the tuner."
6. **Absolute magnitude has to be calibrated the normal way** -- run any
change through `sd10` ECM suite comparison against `head_reference/`
and `match_play.py`'s gated head-to-head match, same as any other
hand-tuned eval constant. The regression tells you *shape*
(which defects matter more relative to each other), not the right
centipawn magnitude for a chess engine's scoring scale.
7. `CountKingSafetyDefects` (the cheap lazy-eval proxy) stays untouched
until a real change lands and is validated -- agreed early in this
project, still the right call.
## Feature schema (read this before extending the pipeline)
`KingSafetyDeltas` in eval.c (`-17,-16,-15,-1,+1,+15,+16,+17`) defines 8
ring slots, indices 0-7 in that order. **Correction made mid-project**:
slots 3 and 4 (deltas -1/+1) are the two SAME-RANK side squares, not
"front of the king" -- an earlier description in conversation got this
wrong. The diagonal/vertical slots (0,1,2 and 5,6,7) are the ones with a
real front/back asymmetry, matching `EmptyAttackedSquare`'s own existing
color-dependent weighting (WHITE's front is slots 0/1/2, BLACK's is
5/6/7 -- see `_EvalKing`'s comments and `KING_RING_FRONT`-style logic).
`extract_king_features.py` dumps full per-slot resolution (all 8 slots x
4 piece types x several state flags) since the engine-side cost of doing
so is free once you're calling `evalkingdump` anyway. `train_king_safety_model.py`
is where slots 0/1/2/5/6/7 get bucketed into "front"/"back" sums
(color-dependent per row's `mated_color_or_sampled_color`), while slots 3/4
stay individually resolved -- this bucketing was added in this session
after the diagonal per-slot coefficients turned out to be the noisiest,
most sign-flipping entries in every full-vs-subset drift comparison.
`filestorm` is deliberately NOT used as a training feature -- its two
independent sub-components (`filestormopenfile`, `filestormpawn`) are
used instead, since they map onto two already-separate eval.c tables and
there's no reason to make the regression re-derive their relative
importance from one pre-summed number.
## Bugs found and fixed this session (read before trusting old data)
If you find any `.csv` file in this project OTHER than
`remine_v4/*_v4.csv` and its merged.csv, IT PREDATES ONE OR MORE OF THESE
FIXES AND SHOULD NOT BE USED. (Older files were deleted during cleanup,
but if this pipeline gets rerun and produces intermediate files again,
this is the checklist for whether they're trustworthy.)
1. **Bare-king endgame mates leaking into "middlegame" positives.** Only
checking the attacker's material (`>= 800cp`, roughly rook+bishop) let
trivial K+Q-vs-lone-K technique mates through. Fixed by also requiring
`total_non_pawn_piece_count(board) >= 8` (mirrors `_EvalKing`'s own
`pos->uNonPawnCount[WHITE][0] + pos->uNonPawnCount[BLACK][0] < 8`
endgame boundary exactly) -- see `is_middlegame()` in
`mine_king_mates.py`.
2. **`min_ply` too low (10, i.e. move 5) for control sampling.** A large
fraction of the pool's draws are short draws-by-agreement close to
`filter_pgn.py`'s own 40-ply floor. Combined with a large end-guard
(staying clear of the game's end), the only valid sampling window on a
short draw was the early opening -- where minors are legitimately
still undeveloped for completely ordinary reasons. This alone caused
`minorsathome` to look like a massive, spurious "signal" (controls
showed `minorsathome>=2` in ~85% of samples). Verified the engine's
own computation was correct on a real developed-middlegame FEN --
100% a sampling-window bug, not an eval bug. Fixed: `min_ply` raised
to 20 (move 10), and a game too short to satisfy both `min_ply` and
the end-guard is now skipped entirely rather than force-sampled early.
3. **Control-sampling ply selection secretly always picked the earliest
plies in a game's valid window.** `rng.shuffle(candidates)` followed
by `for idx in sorted(candidates)` completely undid the shuffle.
Fixed in two steps (the first fix wasn't sufficient on its own --
see the code comment in `mine_controls()` for why): randomly
subsample which plies to try, replay ascending through all of them
ONCE (required -- boards can only be walked forward), collect every
result that passes filters into a list, THEN randomly choose which
results to actually keep via `rng.sample` on that list. Verified:
sampled-ply-as-fraction-of-game-length now has a median around 0.46
with real tail into 60%+, instead of clustering at 20-30%.
4. **Matching-radius bug treated two very differently-scaled confound
columns with one shared distance metric.** `attackerarmyscaler`
spans 0-31 (`ASSERT`-confirmed hard clamp ceiling), `minorsathome`
spans 0-4. A single Chebyshev radius meant a "tight" radius-2 match
still let `minorsathome` drift by up to 2 -- literally half its
entire range. This is why `minorsathome` was STILL the #1 "signal"
even after an initial matching attempt. Fixed: `match_controls.py`
now takes `--exact-cols` (must match exactly, no radius search --
`minorsathome` defaults here) and `--radius-cols` (allowed to expand
outward, `attackerarmyscaler` defaults here) as separate concepts.
5. **KDTree-based nearest-neighbor matching matched almost nothing** (308
of 43,181 positives) because the match columns are low-cardinality
integers -- huge exact-value clusters mean a KDTree's "k nearest" are
massively tied, so the first few positives processed in a cluster
exhaust the shared candidate set and everyone else in that cluster
fails to match. Replaced with exact-bucket + expanding-integer-radius
matching (see `match_controls.py`), which has no such collision
problem.
6. **`_EvalKing`'s per-square ring scan implicitly assumed all 8 slots
were geometrically equivalent for feature purposes**; slots 3/4
(side squares) vs. the diagonal slots turned out to behave
differently enough that per-slot resolution on the diagonals was
pure noise at the sample sizes available -- see "Feature schema"
above.
7. **Board-diagram spam.** `setboard` prints a full ASCII board dump on
every call by default (`g_Options.fShouldPost`, see `DumpPosition` in
gamelist.c), which dominated per-row I/O cost during feature
extraction. Fixed by prepending `nopost` to every extraction script.
8. **Two separate concurrent-process races during interactive mining**,
both self-inflicted mid-session: (a) `$!` after a `cd ... && nohup
...` pipeline captured the wrong PID, leaving a stale duplicate
process racing on the same output file as a supposedly-fresh rerun;
(b) killing that process didn't fully stop it before a new one
started. No lasting damage (caught via `ps aux` before trusting any
output), but: **always verify a background job is actually singular
via `ps aux | grep <script>` before trusting its output**, especially
after a kill+relaunch sequence.
9. **`sizeof(KING_DEFECT_DUMP)` name collision with the
`-DKING_DEFECT_DUMP` build flag.** The build flag is defined as
literal `1` by the compiler; naming the struct type the same thing
caused the preprocessor to mangle the struct declaration wherever the
type name appeared. Renamed the struct to `KING_DEFECT_DUMP_INFO`.
If you ever add a new `#ifdef SOME_NAME` build flag AND a same-named
type/variable in the same session, expect this failure mode.
10. **`gmake` doesn't recompile on a flag-only change.** After the struct
rename (#9) and other `chess.h` edits, a `gmake KING_DEFECT_DUMP=1`
without a preceding `gmake clean` silently reused stale `.o` files
from an earlier flag-less build, producing a binary that didn't
actually have `evalkingdump` registered, with no compile error at
all. Always `gmake clean` first when a build-flag combination
changes, not just when headers change (this is already called out
in `CLAUDE.md` for header changes, but the flag-only case is easy to
forget too).
## Performance notes for rerunning
- `pgn/twic_filtered.pgn` (642,268 games) mines in a few minutes when
sharded 12-24 ways via `shard_mine.py` (byte-range sharding on real
game boundaries, found via a fast raw-text `[Event ` scan). Don't run
it unsharded -- a single-process full scan took ~40 minutes.
- Feature extraction (`extract_king_features.py`) is the slow stage:
~54 minutes for 172,735 rows sharded 18 ways on a contended 24-core
box. `nopost` (fix #7 above) is essential -- without it, board-diagram
printing dominates I/O cost.
- `match_controls.py` is pure in-memory bucket lookups, seconds even for
the full dataset -- never worth sharding.
- If `twic_filtered.pgn` (642k games, >=2400 Elo / >40 ply filtered) ever
proves insufficient for match density, `pgn/twic.pgn` is 7-8x larger
(4,099,094 games, unfiltered) and available as a fallback source --
not needed this run (100% match rate achieved on `twic_filtered.pgn`
once the above bugs were fixed).
## Suggested next steps
1. Pick 2-3 concrete constant changes from Tier 1/2 above (start with
`KingFileDefects`/`KingStormingPawnDefects` -- lowest risk).
2. Land them as a normal `eval.c` diff, run `precommit_check.sh`.
3. `sd10` comparison across the three curated suites against
`head_reference/`.
4. `match_play.py` gated head-to-head match against the pre-change DNA.
5. Only keep the change if `LOWER95 >= 0.5`.
6. Once a change lands, consider whether `CountKingSafetyDefects` (lazy
eval) needs corresponding rework -- deferred by design until a real
change is validated (see project goal section above).
|
