# Plan: `cross_sectional_all.ep_` -- a larger successor to `cross_sectional_200k.ep_` Status: **planned, not yet run**. Deliberately deferred until the current 16-shard joint-table collection run (`../joint_table_collection/`) finishes -- this is a multi-hour CPU job on the same shared box and shouldn't compete with that run for cores. See "Sequencing" at the end. ## Motivation `tests/cross_sectional_200k.ep_` (400,000 lines = 200,000 `setboard`/`go` pairs) is the cross-sectional real-game position pool used to seed the joint-table collection runs (and other real-game-position needs). It predates the current PGN corpus in `../pgn/`: - `twic.pgn` -- 8,452,495 games (raw TWIC archive) - `twic_filtered.pgn` -- 1,337,714 games (pre-filtered subset of the above -- header inspection confirms Elo/ply filtering already applied, presumably via `../src/eval_tune/filter_pgn.py`'s defaults, `--min-elo 2400 --min-ply 20`; worth confirming exact args before relying on that filter alone -- see "Open decisions") - `lichess_elite_*.pgn` (2013-09 through 2020-05) -- 3,819,130 games, of which 3,819,129 (99.99997%) already have both players rated >2000 (the "elite" filter is already essentially an Elo>2000 gate) Measured directly (this session, via a streaming `gawk` pass over every `[WhiteElo]`/`[BlackElo]` pair, keyed on `[Event` boundaries): | source | total games | both players >2000 | |---|---:|---:| | `twic.pgn` | 8,452,495 | 2,948,359 | | `lichess_elite_*.pgn` | 3,819,130 | 3,819,129 | | **combined** | **12,271,625** | **6,767,488** | So the source corpus can support a cross-sectional pool far larger than 200k even under a strict >2000 rating floor -- the constraint is entirely "how many do we want," not "is there enough data." ## Sampling method **5 positions per qualifying game, at fixed fractional points in the game (not random plies).** This is the key design choice: dividing by game *length* rather than drawing one random ply per game structurally guarantees every kept game contributes exactly one position to each of 5 game-stage buckets, instead of leaving stage balance to chance (uniform- random-ply sampling is naturally skewed toward the middlegame, since games spend more plies there than grinding out short endgames -- this was the same problem `sample_eval_calibration_positions.py`'s `--stratify` flag was built to correct for, after the fact, by material bucket; sampling at fixed fractions avoids needing that correction in the first place). - Take the game's full mainline ply count `N`. - Sample at **6 equal segments, using the 5 *interior* boundaries** (1/6, 2/6, 3/6, 4/6, 5/6 of `N`) -- not literal 1/5..5/5. The 5/5 point is the final move of the game, which is frequently checkmate, stalemate, or immediately pre-resignation -- `go` can't search a terminal board meaningfully. Using interior sixths avoids this without losing the "5 stages" framing. - Label the 5 samples `early`, `early_mid`, `mid`, `mid_late`, `late`. - Skip a boundary (and count it) if it lands on a game-over board, or if rounding collapses two boundaries onto the same ply (only possible in games just barely over the `--min-ply` floor). ## Filtering: "long enough to be interesting" Two independent filters, both required: 1. **Rating floor**: both `WhiteElo` and `BlackElo` present and above threshold. Recommend **2000** (matches the rating question this plan originated from) -- but see "Open decisions" below re: consistency with `twic_filtered.pgn`'s own pre-baked filter. 2. **Minimum ply count**: reject games shorter than `--min-ply` (default **30** plies = 15 full moves each side). Rationale: with 5 sample points needed, very short games (10-move draws by repetition, early resignations) either fail outright or produce 5 samples crammed into a handful of real plies, which defeats the point of stage separation. 30 plies gives ~5-6 plies of spacing between consecutive sample points even in a borderline-length game. ## Deduplication Positions are deduped on a **normalized FEN key** -- board placement, side to move, castling rights, en passant square -- explicitly dropping the halfmove-clock and fullmove-number fields, which would otherwise make two functionally-identical positions compare as distinct. Dedup is global across the whole run (one `seen_keys` set for the whole corpus, not per-file), since the same well-known opening line recurs across many games regardless of source file. Measured on a 776-game smoke-test slice (`lichess_elite_2020-05.pgn`, first 20,000 lines): overall duplicate rate 0.4%, concentrated almost entirely in the `early` stage (1.8% there vs ~0% by `mid`). This matches expectation -- opening theory repeats verbatim across games; middlegame and endgame positions are close to unique per game. For reference, the existing `cross_sectional_200k.ep_` itself has ~8.5% duplicate positions (182,999 unique boards out of 200,000 lines) -- this plan's dedup should produce a cleaner pool than the file it's replacing, not just a bigger one. Dedup is a soft design choice, not a hard requirement -- see "Open decisions." ## Quiescence: deliberately NOT filtered Considered and rejected. These positions are **search roots** (`setboard` + `go`), not eval-only calibration points. The engine's own qsearch resolves tactics at every node of the tree it searches from these roots regardless of whether the root itself is quiet -- filtering for quiescent roots would just mean throwing away the (accurate) tactical-position share of real games, biasing the pool toward "calm" positions in a way that doesn't match what the engine's search tree actually looks like mid-game. This mirrors the reasoning already written into `sample_eval_calibration_positions.py`'s docstring, which explicitly does NOT filter on SEE/hanging-material/qsearch-agreement for the same reason. (Note: this is the opposite choice from `tune_eval_dna.py`'s *quiet*- position pools, which exist for Texel eval tuning, a genuinely different use case where the eval function has to approximate game outcome without search resolving anything.) ## Stats to compute and report (already implemented in the prototype script) - games scanned / kept / rejected (split by reason: Elo, too-short) - positions written, duplicate-FEN count and rate (overall and per-stage) - game-over boundaries skipped - result distribution (`1-0` / `0-1` / `1/2-1/2`) of kept games -- reported, not forced balanced by default (see "Open decisions" on whether to add explicit result-balancing quotas) - per-source-file game counts (to see the twic vs. lichess mix in the final pool, since lichess alone vastly outnumbers twic in recent years and a naive concat would otherwise skew heavily toward recent online blitz) - per-stage written count and dup rate (the 5 buckets above) - material-bucket histogram (0 = bare kings .. 7 = full army; same bucketing as `eval.c`'s `_MaterialBucket` and the calibration script's own copy) -- a secondary phase-sanity check independent of the ply-fraction stage labels, since a long grinding decisive endgame can still have plenty of material on the board at its "late" sample point - side-to-move balance (white/black to move) in the final pool Smoke-test run on 776 games already confirms the script produces sane numbers on all of the above (376/344/43 result split, bucket histogram skewed toward full-army as expected for `early`, 54%/46% white/black to move). ## Performance and timing Measured: **~197 games/sec single-threaded** (python-chess's PGN parser is the bottleneck, not file I/O). | corpus choice | games | est. wall-clock (1 core) | |---|---:|---:| | raw `twic.pgn` + `lichess_elite_*.pgn` | 12,271,625 | ~17.3 hours | | `twic_filtered.pgn` + `lichess_elite_*.pgn` | ~5,156,844 | ~7.3 hours | Recommend the second option (`twic_filtered.pgn`, not raw `twic.pgn`) -- it's already pre-filtered and 6x smaller, and the raw `twic.pgn` pass would just spend most of its time discarding sub-2000 games anyway. **Parallelization is easy and should be used.** Each source file (1 `twic_filtered.pgn` + ~80 monthly `lichess_elite_*.pgn` files) is independent input; duplicate rate is low (0.4% measured) and concentrated in the `early` stage, so running each file as its own process (one `seen_keys` set per process, no cross-process coordination during the scan) and doing a **cheap post-hoc FEN-dedup merge pass** at the end loses only a small amount of extra dedup accuracy in exchange for an ~N-core speedup. With even 8 cores this turns a 7.3-hour job into under an hour; with more cores (or renting cloud CPU -- see the unrelated earlier discussion on cloud compute cost in this project, same economics apply: cheap, embarrassingly-parallel, CPU-only) it could run in minutes. ## Script Prototype already written and smoke-tested: `../src/eval_tune/build_cross_sectional_all.py` Current interface: ``` build_cross_sectional_all.py PGN_PATH [PGN_PATH ...] \ --min-elo 2000 --min-ply 30 --out tests/cross_sectional_all.ep_ ``` Runs sequentially over the given files today. Before the real run, it should gain: - a `--jobs N` (or a thin wrapper shell script) to fan out per-file processing across cores, plus a merge step that re-applies the normalized-FEN dedup across all per-file outputs before writing the final `.ep_` - (optional) a `--balance-results` quota mechanism if the natural result distribution turns out to be too skewed once measured on the real corpus (the 776-game smoke test's 376/344/43 split is close to even between decisive results already at this sample size; whether that holds at the full-corpus scale is worth checking on the real numbers before deciding this is needed) ## Open decisions (to settle before the real run) 1. **Rating floor: 2000, or match `twic_filtered.pgn`'s existing filter?** `filter_pgn.py`'s own default is `--min-elo 2400`; if `twic_filtered.pgn` was built with that default, using `--min-elo 2000` downstream doesn't loosen twic's effective floor (it's already baked in at 2400) but does matter for `lichess_elite` positions, which aren't otherwise filtered at build time. Worth explicitly checking what args built `twic_filtered.pgn` before treating 2000 and 2400 as interchangeable. 2. **Keep dedup, or drop it?** Dedup is cheap and clearly improves pool quality (removes the same 8.5%-ish redundancy the current 200k file already has), but it does mean the final position count depends on corpus overlap, not just games-kept x 5 -- worth deciding whether an exact target size (e.g. "2 million positions") matters enough to oversample games and truncate, vs. just taking whatever the filtered + deduped pool naturally yields. 3. **Result balancing**: leave natural, or enforce quotas?** Leaning toward "measure first, decide after" -- forcing balance before knowing whether the real corpus is actually skewed risks discarding good data for no reason. 4. **Final target size**: not yet fixed. At 5,156,844 candidate games (the `twic_filtered` + `lichess_elite` combination) x 5 positions/game minus whatever dedup removes, the ceiling is in the ~20-25 million range -- almost certainly far more than needed. Worth picking a concrete target (e.g. 1-2 million) and either subsampling games up front (cheaper, fewer games to parse) or subsampling the output pool after the fact (slower, but preserves the natural stage/result/material distribution measured on the full run). ## Sequencing **Do not run until the current joint-table collection run (`../joint_table_collection/`) finishes.** That run uses all 16 cores this box supports (`--cpus 1` per shard, deliberately single-threaded, deliberately using independent shards rather than threads for parallelism) for what's currently budgeted as a multi-hour run. This job, even parallelized, would contend directly with it for the same cores. Revisit this plan and confirm the open decisions above once that run has completed and its results have been baked into `data.c`.