New markdown.

1 files changed, 232 insertions, 0 deletions

diff --git a/tests/CROSS_SECTIONAL_ALL.md b/tests/CROSS_SECTIONAL_ALL.md
new file mode 100644
index 0000000..4f702b0
--- /dev/null
+++ b/tests/CROSS_SECTIONAL_ALL.md
@@ -0,0 +1,232 @@
+# Plan: `cross_sectional_all.ep_` -- a larger successor to `cross_sectional_200k.ep_`
+
+Status: **planned, not yet run**. Deliberately deferred until the current
+16-shard joint-table collection run (`../joint_table_collection/`) finishes --
+this is a multi-hour CPU job on the same shared box and shouldn't compete
+with that run for cores. See "Sequencing" at the end.
+
+## Motivation
+
+`tests/cross_sectional_200k.ep_` (400,000 lines = 200,000 `setboard`/`go`
+pairs) is the cross-sectional real-game position pool used to seed the
+joint-table collection runs (and other real-game-position needs). It
+predates the current PGN corpus in `../pgn/`:
+
+- `twic.pgn` -- 8,452,495 games (raw TWIC archive)
+- `twic_filtered.pgn` -- 1,337,714 games (pre-filtered subset of the above --
+  header inspection confirms Elo/ply filtering already applied, presumably
+  via `../src/eval_tune/filter_pgn.py`'s defaults, `--min-elo 2400 --min-ply
+  20`; worth confirming exact args before relying on that filter alone --
+  see "Open decisions")
+- `lichess_elite_*.pgn` (2013-09 through 2020-05) -- 3,819,130 games, of
+  which 3,819,129 (99.99997%) already have both players rated >2000 (the
+  "elite" filter is already essentially an Elo>2000 gate)
+
+Measured directly (this session, via a streaming `gawk` pass over every
+`[WhiteElo]`/`[BlackElo]` pair, keyed on `[Event` boundaries):
+
+| source | total games | both players >2000 |
+|---|---:|---:|
+| `twic.pgn` | 8,452,495 | 2,948,359 |
+| `lichess_elite_*.pgn` | 3,819,130 | 3,819,129 |
+| **combined** | **12,271,625** | **6,767,488** |
+
+So the source corpus can support a cross-sectional pool far larger than
+200k even under a strict >2000 rating floor -- the constraint is entirely
+"how many do we want," not "is there enough data."
+
+## Sampling method
+
+**5 positions per qualifying game, at fixed fractional points in the game
+(not random plies).** This is the key design choice: dividing by game
+*length* rather than drawing one random ply per game structurally
+guarantees every kept game contributes exactly one position to each of 5
+game-stage buckets, instead of leaving stage balance to chance (uniform-
+random-ply sampling is naturally skewed toward the middlegame, since games
+spend more plies there than grinding out short endgames -- this was the
+same problem `sample_eval_calibration_positions.py`'s `--stratify` flag
+was built to correct for, after the fact, by material bucket; sampling at
+fixed fractions avoids needing that correction in the first place).
+
+- Take the game's full mainline ply count `N`.
+- Sample at **6 equal segments, using the 5 *interior* boundaries**
+  (1/6, 2/6, 3/6, 4/6, 5/6 of `N`) -- not literal 1/5..5/5. The 5/5 point
+  is the final move of the game, which is frequently checkmate, stalemate,
+  or immediately pre-resignation -- `go` can't search a terminal board
+  meaningfully. Using interior sixths avoids this without losing the
+  "5 stages" framing.
+- Label the 5 samples `early`, `early_mid`, `mid`, `mid_late`, `late`.
+- Skip a boundary (and count it) if it lands on a game-over board, or if
+  rounding collapses two boundaries onto the same ply (only possible in
+  games just barely over the `--min-ply` floor).
+
+## Filtering: "long enough to be interesting"
+
+Two independent filters, both required:
+
+1. **Rating floor**: both `WhiteElo` and `BlackElo` present and above
+   threshold. Recommend **2000** (matches the rating question this plan
+   originated from) -- but see "Open decisions" below re: consistency with
+   `twic_filtered.pgn`'s own pre-baked filter.
+2. **Minimum ply count**: reject games shorter than `--min-ply` (default
+   **30** plies = 15 full moves each side). Rationale: with 5 sample points
+   needed, very short games (10-move draws by repetition, early
+   resignations) either fail outright or produce 5 samples crammed into a
+   handful of real plies, which defeats the point of stage separation.
+   30 plies gives ~5-6 plies of spacing between consecutive sample points
+   even in a borderline-length game.
+
+## Deduplication
+
+Positions are deduped on a **normalized FEN key** -- board placement, side
+to move, castling rights, en passant square -- explicitly dropping the
+halfmove-clock and fullmove-number fields, which would otherwise make two
+functionally-identical positions compare as distinct. Dedup is global
+across the whole run (one `seen_keys` set for the whole corpus, not
+per-file), since the same well-known opening line recurs across many
+games regardless of source file.
+
+Measured on a 776-game smoke-test slice (`lichess_elite_2020-05.pgn`,
+first 20,000 lines): overall duplicate rate 0.4%, concentrated almost
+entirely in the `early` stage (1.8% there vs ~0% by `mid`). This matches
+expectation -- opening theory repeats verbatim across games; middlegame
+and endgame positions are close to unique per game. For reference, the
+existing `cross_sectional_200k.ep_` itself has ~8.5% duplicate positions
+(182,999 unique boards out of 200,000 lines) -- this plan's dedup should
+produce a cleaner pool than the file it's replacing, not just a bigger one.
+
+Dedup is a soft design choice, not a hard requirement -- see "Open
+decisions."
+
+## Quiescence: deliberately NOT filtered
+
+Considered and rejected. These positions are **search roots** (`setboard`
++ `go`), not eval-only calibration points. The engine's own qsearch
+resolves tactics at every node of the tree it searches from these roots
+regardless of whether the root itself is quiet -- filtering for quiescent
+roots would just mean throwing away the (accurate) tactical-position
+share of real games, biasing the pool toward "calm" positions in a way
+that doesn't match what the engine's search tree actually looks like
+mid-game. This mirrors the reasoning already written into
+`sample_eval_calibration_positions.py`'s docstring, which explicitly does
+NOT filter on SEE/hanging-material/qsearch-agreement for the same reason.
+(Note: this is the opposite choice from `tune_eval_dna.py`'s *quiet*-
+position pools, which exist for Texel eval tuning, a genuinely different
+use case where the eval function has to approximate game outcome without
+search resolving anything.)
+
+## Stats to compute and report (already implemented in the prototype script)
+
+- games scanned / kept / rejected (split by reason: Elo, too-short)
+- positions written, duplicate-FEN count and rate (overall and per-stage)
+- game-over boundaries skipped
+- result distribution (`1-0` / `0-1` / `1/2-1/2`) of kept games -- reported,
+  not forced balanced by default (see "Open decisions" on whether to add
+  explicit result-balancing quotas)
+- per-source-file game counts (to see the twic vs. lichess mix in the
+  final pool, since lichess alone vastly outnumbers twic in recent years
+  and a naive concat would otherwise skew heavily toward recent online
+  blitz)
+- per-stage written count and dup rate (the 5 buckets above)
+- material-bucket histogram (0 = bare kings .. 7 = full army; same
+  bucketing as `eval.c`'s `_MaterialBucket` and the calibration script's
+  own copy) -- a secondary phase-sanity check independent of the
+  ply-fraction stage labels, since a long grinding decisive endgame can
+  still have plenty of material on the board at its "late" sample point
+- side-to-move balance (white/black to move) in the final pool
+
+Smoke-test run on 776 games already confirms the script produces sane
+numbers on all of the above (376/344/43 result split, bucket histogram
+skewed toward full-army as expected for `early`, 54%/46% white/black to
+move).
+
+## Performance and timing
+
+Measured: **~197 games/sec single-threaded** (python-chess's PGN parser is
+the bottleneck, not file I/O).
+
+| corpus choice | games | est. wall-clock (1 core) |
+|---|---:|---:|
+| raw `twic.pgn` + `lichess_elite_*.pgn` | 12,271,625 | ~17.3 hours |
+| `twic_filtered.pgn` + `lichess_elite_*.pgn` | ~5,156,844 | ~7.3 hours |
+
+Recommend the second option (`twic_filtered.pgn`, not raw `twic.pgn`) --
+it's already pre-filtered and 6x smaller, and the raw `twic.pgn` pass
+would just spend most of its time discarding sub-2000 games anyway.
+
+**Parallelization is easy and should be used.** Each source file (1
+`twic_filtered.pgn` + ~80 monthly `lichess_elite_*.pgn` files) is
+independent input; duplicate rate is low (0.4% measured) and concentrated
+in the `early` stage, so running each file as its own process (one
+`seen_keys` set per process, no cross-process coordination during the
+scan) and doing a **cheap post-hoc FEN-dedup merge pass** at the end loses
+only a small amount of extra dedup accuracy in exchange for an
+~N-core speedup. With even 8 cores this turns a 7.3-hour job into under
+an hour; with more cores (or renting cloud CPU -- see the unrelated
+earlier discussion on cloud compute cost in this project, same
+economics apply: cheap, embarrassingly-parallel, CPU-only) it could run
+in minutes.
+
+## Script
+
+Prototype already written and smoke-tested:
+`../src/eval_tune/build_cross_sectional_all.py`
+
+Current interface:
+```
+build_cross_sectional_all.py PGN_PATH [PGN_PATH ...] \
+    --min-elo 2000 --min-ply 30 --out tests/cross_sectional_all.ep_
+```
+
+Runs sequentially over the given files today. Before the real run, it
+should gain:
+- a `--jobs N` (or a thin wrapper shell script) to fan out per-file
+  processing across cores, plus a merge step that re-applies the
+  normalized-FEN dedup across all per-file outputs before writing the
+  final `.ep_`
+- (optional) a `--balance-results` quota mechanism if the natural result
+  distribution turns out to be too skewed once measured on the real
+  corpus (the 776-game smoke test's 376/344/43 split is close to even
+  between decisive results already at this sample size; whether that
+  holds at the full-corpus scale is worth checking on the real numbers
+  before deciding this is needed)
+
+## Open decisions (to settle before the real run)
+
+1. **Rating floor: 2000, or match `twic_filtered.pgn`'s existing filter?**
+   `filter_pgn.py`'s own default is `--min-elo 2400`; if `twic_filtered.pgn`
+   was built with that default, using `--min-elo 2000` downstream doesn't
+   loosen twic's effective floor (it's already baked in at 2400) but does
+   matter for `lichess_elite` positions, which aren't otherwise filtered
+   at build time. Worth explicitly checking what args built
+   `twic_filtered.pgn` before treating 2000 and 2400 as interchangeable.
+2. **Keep dedup, or drop it?** Dedup is cheap and clearly improves pool
+   quality (removes the same 8.5%-ish redundancy the current 200k file
+   already has), but it does mean the final position count depends on
+   corpus overlap, not just games-kept x 5 -- worth deciding whether an
+   exact target size (e.g. "2 million positions") matters enough to
+   oversample games and truncate, vs. just taking whatever the filtered +
+   deduped pool naturally yields.
+3. **Result balancing**: leave natural, or enforce quotas?** Leaning
+   toward "measure first, decide after" -- forcing balance before knowing
+   whether the real corpus is actually skewed risks discarding good data
+   for no reason.
+4. **Final target size**: not yet fixed. At 5,156,844 candidate games (the
+   `twic_filtered` + `lichess_elite` combination) x 5 positions/game minus
+   whatever dedup removes, the ceiling is in the ~20-25 million range --
+   almost certainly far more than needed. Worth picking a concrete target
+   (e.g. 1-2 million) and either subsampling games up front (cheaper, fewer
+   games to parse) or subsampling the output pool after the fact
+   (slower, but preserves the natural stage/result/material distribution
+   measured on the full run).
+
+## Sequencing
+
+**Do not run until the current joint-table collection run
+(`../joint_table_collection/`) finishes.** That run uses all 16 cores this
+box supports (`--cpus 1` per shard, deliberately single-threaded,
+deliberately using independent shards rather than threads for
+parallelism) for what's currently budgeted as a multi-hour run. This job,
+even parallelized, would contend directly with it for the same cores.
+Revisit this plan and confirm the open decisions above once that run has
+completed and its results have been baked into `data.c`.