New markdown.

2 files changed, 486 insertions, 0 deletions

diff --git a/src/ANALYZE_GAME.md b/src/ANALYZE_GAME.md
new file mode 100644
index 0000000..2576a1e
--- /dev/null
+++ b/src/ANALYZE_GAME.md
@@ -0,0 +1,254 @@
+# Analyzing a game from blunders.pgn
+
+A working playbook for pulling a specific game out of `blunders.pgn` and
+figuring out *where the eval/search actually went wrong*, as opposed to
+just noticing that the final result was bad. Written after a session spent
+doing exactly this on the 2026-10-06 `typhoon vs Crafty-25.2` loss (a
+KRP-vs-KQ endgame that looked like it had a scoring bug but turned out to
+be legitimately lost, plus two real engine bugs found along the way).
+
+## 0. What's actually in blunders.pgn
+
+Each game's move comments are **only** `{score/depth time}`, e.g.
+`{-5.09/34 6}` -- score in pawns from White's perspective (not side-to-move;
+confirmed by the sign trending consistently negative as White loses, across
+both White's and Black's own move comments), search depth reached, and
+wall-clock seconds spent on that move. **There is no PV text and no engine
+tag in the PGN at all** -- not `<REC>`, not `<EGTB>`, not `<#>`, nothing.
+Whatever wrapper plays these self-play/vs-Crafty games strips the PV down to
+just the eval number before writing the comment. If you want to see *why*
+a move was chosen -- the actual PV, or whether a recognizer/tablebase hit
+produced the score -- you have to replay the position through a live engine
+yourself; the PGN alone can't tell you.
+
+Don't go looking for `<EGTB>`/`<#>` tags in old PGNs for another reason too:
+those sentinels (`EGTBMOVE` plus the `RECOGN_EGTB` wiring) are new, added in
+the 2026-10-06 session that produced this document. Any game recorded before
+that commit landed couldn't have produced them even if the PGN format did
+carry PV text.
+
+## 1. Pulling the game and exact FENs with python-chess
+
+`python3 -c "import chess"` works in this environment (python-chess is
+installed) -- use it rather than hand-parsing SAN, especially for games deep
+enough that move numbers and ply counts diverge from intuition.
+
+```python
+import chess.pgn
+f = open('blunders.pgn')
+games = []
+g = chess.pgn.read_game(f)
+while g:
+    games.append(g)
+    g = chess.pgn.read_game(f)
+last = games[-1]            # or games[N] for a specific game
+board = last.board()
+moves = list(last.mainline_moves())
+```
+
+**Getting a FEN at a specific PGN move number**: ply count and PGN move
+number are NOT the same thing and it's easy to get this wrong (did exactly
+this once already this session -- cost a wasted round-trip). The mapping:
+
+```python
+def ply(movenum, color):           # color: 'w' or 'b'
+    return 2*(movenum-1) + (1 if color == 'w' else 2)
+```
+
+`moves[ply(69, 'w') - 1]` is White's 69th move (0-indexed list, 1-indexed
+ply). Push moves up through the target ply and read `board.fen()` --
+`board.halfmove_clock` is directly available too and matters a lot (see
+"fifty-move-rule proximity" below).
+
+To scan a whole stretch and watch halfmove_clock / material change move by
+move (useful for finding exactly where a pawn/capture resets the fifty-move
+counter, or where material transitions from one tablebase signature to
+another):
+
+```python
+for i, m in enumerate(moves, start=1):
+    before = board.copy()
+    board.push(m)
+    if lo <= i <= hi:
+        print(i, before.san(m), 'halfmove_clock=', board.halfmove_clock,
+              'fen=', board.fen())
+```
+
+## 2. Replaying a position through typhoon
+
+Use `--batch --command` with `setboard`, not interactive stdin piping (per
+CLAUDE.md, piping commands over stdin to a live REPL doesn't reliably
+complete a multi-step script). Always pass `--egtbpath` explicitly if you
+want to be sure which EGTB directory is in play (the hardcoded default is
+`/zscratch/egtb`, which currently has full WDL+DTZ coverage through 6-man --
+see CLAUDE.md -- but don't assume that silently when writing up findings;
+state it).
+
+```sh
+./typhoon --cpus 1 --hash 256m --egtbpath /zscratch/egtb \
+  --logfile /tmp/typhoon/<name>.log \
+  --batch --command "force; setboard <fen>; eval; sd <depth>; go"
+```
+
+- `eval` alone (no search) gives the raw static eval -- useful as a sanity
+  baseline before trusting a search score.
+- `sd <depth>` fixes search depth; prefer this over `st <seconds>` for
+  reproducibility (machine load varies a lot on this shared box -- see
+  CLAUDE.md's ECM section).
+- Watch stdout for these diagnostic lines, which tell you *how* the move was
+  chosen, not just what it was:
+  - `ONLY MOVE --> stop searching now` -- forced, no real search happened.
+  - `TABLEBASE MOVE --> stop searching now` -- root EGTB (DTZ) root-move
+    short-circuit; score will show as `PV=<EGTB>` and a flat `±NMATE`-ish
+    "mate" display that is **not** a real mate distance (see section 4).
+  - `FORCED MATE --> stop searching now` -- the iterative loop detected a
+    mate score at the current depth and is stopping; the printed PV comes
+    from `szLastPV`, which on a **root fail-high** is just the bare move
+    (`MoveToSan`), not a real PV walk -- see section 5, don't expect `<#>`
+    here.
+  - `DEPTH LIMIT --> stop searching now` -- ordinary completion, nothing
+    special.
+- Grep the output/log for `tellothers` to get the final
+  `score, time, nps, PV=...` summary line in one place.
+
+## 3. Cross-checking with Crafty
+
+`/usr/local/bin/crafty` is available and gives a genuinely independent
+second opinion -- different engine, different (or absent) EGTB usage,
+different search. Use it exactly the way you'd use typhoon:
+
+```sh
+echo "setboard <fen>
+sd <depth>
+go
+quit" | timeout 20 /usr/local/bin/crafty 2>&1 | tail -30
+```
+
+Crafty's output is far chattier (full eval component breakdown, PV with
+move count, nodes/nps) -- look for the `depth ... score ... variation`
+block and the `note: scores are for the white side` line (confirms sign
+convention, which is directly comparable to typhoon's White-POV scores).
+
+**Why this matters**: if typhoon reports something wild (a huge mate-ish
+score with a PV that goes nowhere, or a score frozen for 20+ plies in a row)
+and Crafty reports an ordinary, non-mate score with a PV that's actually
+converging, that's your signal the *typhoon* number is suspect, not that the
+position itself is unusual. This exact comparison is what caught the
+`ProbeEGTBRoot` DTZ-as-mate-distance bug (section 4) -- typhoon said
+`-MATE4`, Crafty's independent 16-ply search said `-7.38` with no mate
+anywhere in a 13-move PV.
+
+## 4. Things that look like bugs but are fifty-move-rule artifacts (and one that was a real bug)
+
+**Watch `halfmove_clock` whenever a long endgame grind produces a
+suspicious score.** Syzygy tables report **WDL** (win/draw/loss) and
+**DTZ** (distance to the next fifty-move-rule-*zeroing* move -- a capture or
+pawn push), not DTM (distance to mate). This project's EGTB directory has
+**zero `.rtbm` files** -- DTM tables were never downloaded, and Fathom's DTM
+code (`probe_dtm_table`, `TB_probe_dtm`, etc.) is present in the source but
+dead weight without data behind it. So:
+
+- A genuinely won endgame can sit at a **small DTZ** while still being many
+  plies from actual mate, especially once `halfmove_clock` is climbing
+  toward 100 (a reset becomes forced soon regardless of how far mate
+  actually is).
+- A score that stays flat for many consecutive moves, or a score that
+  jumps to a huge "mate-ish" magnitude with a PV that doesn't resolve
+  anywhere close to that many plies later, is the signature of this --
+  **not** evidence of missing/corrupt tablebase files by itself. Check
+  `ls /zscratch/egtb | grep <material-signature>` to rule out a genuine
+  coverage gap first, but if the files are present, suspect the
+  DTZ-vs-mate-distance conflation next.
+- This was a **real, fixed bug** as of 2026-10-06: `probe.c`'s
+  `ProbeEGTBRoot` used to compute `*piScore = INFINITY_SCORE ∓ uDtz` and
+  report it through the mate-score display path -- i.e. it fabricated a
+  "mate in N" number out of a quantity that was never a mate distance. Fixed
+  to report a flat `+NMATE`/`-NMATE` (same convention `ProbeEGTB`/
+  `RecognLookup` already use at interior nodes) instead. If you're
+  diagnosing a *pre-fix* game (anything before this fix landed -- check
+  `git log probe.c`), expect to see exactly this artifact and don't chase
+  it as a new finding.
+- The **move** Fathom picks at the root is unaffected by any of this --
+  `tb_probe_root()` does its own internal DTZ-based ranking before
+  returning, so the chosen move is already DTZ-optimal regardless of
+  whether the *score* we derived from it was sane. Don't second-guess the
+  move choice because the displayed score looked wrong; check the score
+  logic and the move logic as separate questions.
+
+## 5. Reading PV sentinels (and when you won't see them)
+
+`WalkPV` (`util.c`) terminates a printed PV early and appends a tag when it
+hits one of five sentinel "moves" (`data.c`): `HASHMOVE` → `<TT>`,
+`RECOGNMOVE` → `<REC>`, `DRAWMOVE` → `<=>`, `MATEMOVE` → `<#>`, `EGTBMOVE` →
+`<EGTB>`. These mark *why the PV stopped*, not what the move was:
+
+- `<REC>` / `<EGTB>`: an interior-node `RecognLookup` hit resolved the
+  score outright. `RECOGN_EGTB` (→ `<EGTB>`) means an actual on-disk Syzygy
+  WDL probe succeeded; `RECOGN_EXACT` (→ `<REC>`) means one of `recogn.c`'s
+  hand-written classic recognizers (KPK, KBKP, etc.) fired instead. Before
+  2026-10-06 these were indistinguishable in the PV (both reported
+  `RECOGNMOVE`) -- `RecognLookup` had an `RECOGN_EGTB` enum value sitting
+  unused the whole time, always returning `RECOGN_EXACT` even on a genuine
+  Fathom hit. If you're looking at an old log/build, `<REC>` could mean
+  either; `<EGTB>` specifically didn't exist yet.
+- `<#>`: search found an actual forced checkmate (zero legal moves + in
+  check) at a PV leaf.
+- `<=>`: a drawn leaf (zero legal moves, not in check == stalemate; this
+  sentinel is also used for recognized/forced draws elsewhere).
+- **Root tablebase moves get `<EGTB>` in `szLastPV` directly** (as a literal
+  string, not via `WalkPV` -- `ProbeEGTBRoot`'s success path short-circuits
+  the whole iterative loop and never calls `WalkPV` at all for that move).
+
+**The big gotcha**: `PrintPV` (`util.c`) only calls the real `WalkPV` walk
+-- the one that can surface any of these tags -- when the reported score is
+a genuine PV node, `iAlpha < iScore < iBeta`. On a **root fail-high**
+(`iScore > iBeta`, i.e. the aspiration window is still widening upward),
+it falls back to printing just the bare best move via `MoveToSan`, with no
+PV walk and therefore no sentinel, ever -- even if the position is a
+dead-obvious mate-in-1 whose leaf genuinely wrote `MATEMOVE` two plies down.
+So: **don't read "no `<#>` in the output" as "no mate was found"** -- check
+whether the line says `FORCED MATE` (a fail-high-driven stop) vs. a normal
+depth-complete report before concluding anything about whether the engine
+actually resolved the mating line as an exact PV.
+
+## 6. The timer-elapsed-time gotcha
+
+If you see `Searched for -<huge number> seconds` in output, that's not a
+real timing anomaly to chase -- or rather, it *was* a real bug (fixed
+2026-10-06): `Iterate()` (`root.c`) used to stamp
+`g_MoveTimer.dEndTime = SystemTimeStamp()` partway through the function,
+*before* the shared `end:` label that every early-exit path (`ONLY MOVE`,
+`TABLEBASE MOVE`, pre-move terminal states) jumps to via `goto end`. Those
+paths skipped the stamp entirely, leaving `dEndTime` stale (0, or whatever
+the previous unrelated search last set it to), which produces exactly this
+kind of nonsense negative "elapsed time" when the end-of-search report
+computes `dEndTime - dStartTime`. Fixed by moving the stamp to right after
+the `end:` label itself. If you're diagnosing a pre-fix build and see this,
+it's cosmetic (doesn't affect move choice or score), but it's also a tell
+that you're looking at a root short-circuit path (tablebase or only-move),
+which is useful context for *why* no real iterative search happened at that
+node.
+
+## 7. General workflow summary
+
+1. Identify the game (usually "last game in the file") and get the exact
+   FEN at the move(s) you care about via python-chess -- don't eyeball SAN
+   move numbers against ply counts by hand.
+2. Check `board.halfmove_clock` at that FEN before anything else if it's an
+   endgame -- rules out/in the fifty-move-rule class of artifact immediately.
+3. Replay the FEN through `./typhoon --batch --command "force; setboard
+   <fen>; sd <depth>; go"`, read the diagnostic `--> stop searching now`
+   line to understand *which* code path produced the move, and check
+   `tellothers`'s `PV=` field.
+4. Get a second opinion from `/usr/local/bin/crafty` on the same FEN at
+   comparable depth. Large disagreement (especially "my engine claims near-
+   mate, Crafty's PV goes nowhere near that far") is the signal to dig into
+   the score-computation code, not the position.
+5. If something still looks wrong, read the actual source for the code path
+   the diagnostic line pointed at (`root.c`'s `Iterate`, `probe.c`'s
+   `ProbeEGTB`/`ProbeEGTBRoot`, `recogn.c`'s `RecognLookup`) rather than
+   guessing from behavior alone -- every bug found this way so far
+   (`ProbeEGTBRoot`'s DTZ-as-mate-score, the `dEndTime` placement, the
+   dead `RECOGN_EGTB` branch) was a straightforward, small, findable-in-
+   minutes fix once the actual code was in front of us; none of them
+   required deep speculation.
diff --git a/tests/CROSS_SECTIONAL_ALL.md b/tests/CROSS_SECTIONAL_ALL.md
new file mode 100644
index 0000000..4f702b0
--- /dev/null
+++ b/tests/CROSS_SECTIONAL_ALL.md
@@ -0,0 +1,232 @@
+# Plan: `cross_sectional_all.ep_` -- a larger successor to `cross_sectional_200k.ep_`
+
+Status: **planned, not yet run**. Deliberately deferred until the current
+16-shard joint-table collection run (`../joint_table_collection/`) finishes --
+this is a multi-hour CPU job on the same shared box and shouldn't compete
+with that run for cores. See "Sequencing" at the end.
+
+## Motivation
+
+`tests/cross_sectional_200k.ep_` (400,000 lines = 200,000 `setboard`/`go`
+pairs) is the cross-sectional real-game position pool used to seed the
+joint-table collection runs (and other real-game-position needs). It
+predates the current PGN corpus in `../pgn/`:
+
+- `twic.pgn` -- 8,452,495 games (raw TWIC archive)
+- `twic_filtered.pgn` -- 1,337,714 games (pre-filtered subset of the above --
+  header inspection confirms Elo/ply filtering already applied, presumably
+  via `../src/eval_tune/filter_pgn.py`'s defaults, `--min-elo 2400 --min-ply
+  20`; worth confirming exact args before relying on that filter alone --
+  see "Open decisions")
+- `lichess_elite_*.pgn` (2013-09 through 2020-05) -- 3,819,130 games, of
+  which 3,819,129 (99.99997%) already have both players rated >2000 (the
+  "elite" filter is already essentially an Elo>2000 gate)
+
+Measured directly (this session, via a streaming `gawk` pass over every
+`[WhiteElo]`/`[BlackElo]` pair, keyed on `[Event` boundaries):
+
+| source | total games | both players >2000 |
+|---|---:|---:|
+| `twic.pgn` | 8,452,495 | 2,948,359 |
+| `lichess_elite_*.pgn` | 3,819,130 | 3,819,129 |
+| **combined** | **12,271,625** | **6,767,488** |
+
+So the source corpus can support a cross-sectional pool far larger than
+200k even under a strict >2000 rating floor -- the constraint is entirely
+"how many do we want," not "is there enough data."
+
+## Sampling method
+
+**5 positions per qualifying game, at fixed fractional points in the game
+(not random plies).** This is the key design choice: dividing by game
+*length* rather than drawing one random ply per game structurally
+guarantees every kept game contributes exactly one position to each of 5
+game-stage buckets, instead of leaving stage balance to chance (uniform-
+random-ply sampling is naturally skewed toward the middlegame, since games
+spend more plies there than grinding out short endgames -- this was the
+same problem `sample_eval_calibration_positions.py`'s `--stratify` flag
+was built to correct for, after the fact, by material bucket; sampling at
+fixed fractions avoids needing that correction in the first place).
+
+- Take the game's full mainline ply count `N`.
+- Sample at **6 equal segments, using the 5 *interior* boundaries**
+  (1/6, 2/6, 3/6, 4/6, 5/6 of `N`) -- not literal 1/5..5/5. The 5/5 point
+  is the final move of the game, which is frequently checkmate, stalemate,
+  or immediately pre-resignation -- `go` can't search a terminal board
+  meaningfully. Using interior sixths avoids this without losing the
+  "5 stages" framing.
+- Label the 5 samples `early`, `early_mid`, `mid`, `mid_late`, `late`.
+- Skip a boundary (and count it) if it lands on a game-over board, or if
+  rounding collapses two boundaries onto the same ply (only possible in
+  games just barely over the `--min-ply` floor).
+
+## Filtering: "long enough to be interesting"
+
+Two independent filters, both required:
+
+1. **Rating floor**: both `WhiteElo` and `BlackElo` present and above
+   threshold. Recommend **2000** (matches the rating question this plan
+   originated from) -- but see "Open decisions" below re: consistency with
+   `twic_filtered.pgn`'s own pre-baked filter.
+2. **Minimum ply count**: reject games shorter than `--min-ply` (default
+   **30** plies = 15 full moves each side). Rationale: with 5 sample points
+   needed, very short games (10-move draws by repetition, early
+   resignations) either fail outright or produce 5 samples crammed into a
+   handful of real plies, which defeats the point of stage separation.
+   30 plies gives ~5-6 plies of spacing between consecutive sample points
+   even in a borderline-length game.
+
+## Deduplication
+
+Positions are deduped on a **normalized FEN key** -- board placement, side
+to move, castling rights, en passant square -- explicitly dropping the
+halfmove-clock and fullmove-number fields, which would otherwise make two
+functionally-identical positions compare as distinct. Dedup is global
+across the whole run (one `seen_keys` set for the whole corpus, not
+per-file), since the same well-known opening line recurs across many
+games regardless of source file.
+
+Measured on a 776-game smoke-test slice (`lichess_elite_2020-05.pgn`,
+first 20,000 lines): overall duplicate rate 0.4%, concentrated almost
+entirely in the `early` stage (1.8% there vs ~0% by `mid`). This matches
+expectation -- opening theory repeats verbatim across games; middlegame
+and endgame positions are close to unique per game. For reference, the
+existing `cross_sectional_200k.ep_` itself has ~8.5% duplicate positions
+(182,999 unique boards out of 200,000 lines) -- this plan's dedup should
+produce a cleaner pool than the file it's replacing, not just a bigger one.
+
+Dedup is a soft design choice, not a hard requirement -- see "Open
+decisions."
+
+## Quiescence: deliberately NOT filtered
+
+Considered and rejected. These positions are **search roots** (`setboard`
++ `go`), not eval-only calibration points. The engine's own qsearch
+resolves tactics at every node of the tree it searches from these roots
+regardless of whether the root itself is quiet -- filtering for quiescent
+roots would just mean throwing away the (accurate) tactical-position
+share of real games, biasing the pool toward "calm" positions in a way
+that doesn't match what the engine's search tree actually looks like
+mid-game. This mirrors the reasoning already written into
+`sample_eval_calibration_positions.py`'s docstring, which explicitly does
+NOT filter on SEE/hanging-material/qsearch-agreement for the same reason.
+(Note: this is the opposite choice from `tune_eval_dna.py`'s *quiet*-
+position pools, which exist for Texel eval tuning, a genuinely different
+use case where the eval function has to approximate game outcome without
+search resolving anything.)
+
+## Stats to compute and report (already implemented in the prototype script)
+
+- games scanned / kept / rejected (split by reason: Elo, too-short)
+- positions written, duplicate-FEN count and rate (overall and per-stage)
+- game-over boundaries skipped
+- result distribution (`1-0` / `0-1` / `1/2-1/2`) of kept games -- reported,
+  not forced balanced by default (see "Open decisions" on whether to add
+  explicit result-balancing quotas)
+- per-source-file game counts (to see the twic vs. lichess mix in the
+  final pool, since lichess alone vastly outnumbers twic in recent years
+  and a naive concat would otherwise skew heavily toward recent online
+  blitz)
+- per-stage written count and dup rate (the 5 buckets above)
+- material-bucket histogram (0 = bare kings .. 7 = full army; same
+  bucketing as `eval.c`'s `_MaterialBucket` and the calibration script's
+  own copy) -- a secondary phase-sanity check independent of the
+  ply-fraction stage labels, since a long grinding decisive endgame can
+  still have plenty of material on the board at its "late" sample point
+- side-to-move balance (white/black to move) in the final pool
+
+Smoke-test run on 776 games already confirms the script produces sane
+numbers on all of the above (376/344/43 result split, bucket histogram
+skewed toward full-army as expected for `early`, 54%/46% white/black to
+move).
+
+## Performance and timing
+
+Measured: **~197 games/sec single-threaded** (python-chess's PGN parser is
+the bottleneck, not file I/O).
+
+| corpus choice | games | est. wall-clock (1 core) |
+|---|---:|---:|
+| raw `twic.pgn` + `lichess_elite_*.pgn` | 12,271,625 | ~17.3 hours |
+| `twic_filtered.pgn` + `lichess_elite_*.pgn` | ~5,156,844 | ~7.3 hours |
+
+Recommend the second option (`twic_filtered.pgn`, not raw `twic.pgn`) --
+it's already pre-filtered and 6x smaller, and the raw `twic.pgn` pass
+would just spend most of its time discarding sub-2000 games anyway.
+
+**Parallelization is easy and should be used.** Each source file (1
+`twic_filtered.pgn` + ~80 monthly `lichess_elite_*.pgn` files) is
+independent input; duplicate rate is low (0.4% measured) and concentrated
+in the `early` stage, so running each file as its own process (one
+`seen_keys` set per process, no cross-process coordination during the
+scan) and doing a **cheap post-hoc FEN-dedup merge pass** at the end loses
+only a small amount of extra dedup accuracy in exchange for an
+~N-core speedup. With even 8 cores this turns a 7.3-hour job into under
+an hour; with more cores (or renting cloud CPU -- see the unrelated
+earlier discussion on cloud compute cost in this project, same
+economics apply: cheap, embarrassingly-parallel, CPU-only) it could run
+in minutes.
+
+## Script
+
+Prototype already written and smoke-tested:
+`../src/eval_tune/build_cross_sectional_all.py`
+
+Current interface:
+```
+build_cross_sectional_all.py PGN_PATH [PGN_PATH ...] \
+    --min-elo 2000 --min-ply 30 --out tests/cross_sectional_all.ep_
+```
+
+Runs sequentially over the given files today. Before the real run, it
+should gain:
+- a `--jobs N` (or a thin wrapper shell script) to fan out per-file
+  processing across cores, plus a merge step that re-applies the
+  normalized-FEN dedup across all per-file outputs before writing the
+  final `.ep_`
+- (optional) a `--balance-results` quota mechanism if the natural result
+  distribution turns out to be too skewed once measured on the real
+  corpus (the 776-game smoke test's 376/344/43 split is close to even
+  between decisive results already at this sample size; whether that
+  holds at the full-corpus scale is worth checking on the real numbers
+  before deciding this is needed)
+
+## Open decisions (to settle before the real run)
+
+1. **Rating floor: 2000, or match `twic_filtered.pgn`'s existing filter?**
+   `filter_pgn.py`'s own default is `--min-elo 2400`; if `twic_filtered.pgn`
+   was built with that default, using `--min-elo 2000` downstream doesn't
+   loosen twic's effective floor (it's already baked in at 2400) but does
+   matter for `lichess_elite` positions, which aren't otherwise filtered
+   at build time. Worth explicitly checking what args built
+   `twic_filtered.pgn` before treating 2000 and 2400 as interchangeable.
+2. **Keep dedup, or drop it?** Dedup is cheap and clearly improves pool
+   quality (removes the same 8.5%-ish redundancy the current 200k file
+   already has), but it does mean the final position count depends on
+   corpus overlap, not just games-kept x 5 -- worth deciding whether an
+   exact target size (e.g. "2 million positions") matters enough to
+   oversample games and truncate, vs. just taking whatever the filtered +
+   deduped pool naturally yields.
+3. **Result balancing**: leave natural, or enforce quotas?** Leaning
+   toward "measure first, decide after" -- forcing balance before knowing
+   whether the real corpus is actually skewed risks discarding good data
+   for no reason.
+4. **Final target size**: not yet fixed. At 5,156,844 candidate games (the
+   `twic_filtered` + `lichess_elite` combination) x 5 positions/game minus
+   whatever dedup removes, the ceiling is in the ~20-25 million range --
+   almost certainly far more than needed. Worth picking a concrete target
+   (e.g. 1-2 million) and either subsampling games up front (cheaper, fewer
+   games to parse) or subsampling the output pool after the fact
+   (slower, but preserves the natural stage/result/material distribution
+   measured on the full run).
+
+## Sequencing
+
+**Do not run until the current joint-table collection run
+(`../joint_table_collection/`) finishes.** That run uses all 16 cores this
+box supports (`--cpus 1` per shard, deliberately single-threaded,
+deliberately using independent shards rather than threads for
+parallelism) for what's currently budgeted as a multi-hour run. This job,
+even parallelized, would contend directly with it for the same cores.
+Revisit this plan and confirm the open decisions above once that run has
+completed and its results have been baked into `data.c`.