path: root/tests/CROSS_SECTIONAL_ALL.md

# Plan: `cross_sectional_all.ep_` -- a larger successor to `cross_sectional_200k.ep_`

Status: **planned, not yet run**. Deliberately deferred until the current
16-shard joint-table collection run (`../joint_table_collection/`) finishes --
this is a multi-hour CPU job on the same shared box and shouldn't compete
with that run for cores. See "Sequencing" at the end.

## Motivation

`tests/cross_sectional_200k.ep_` (400,000 lines = 200,000 `setboard`/`go`
pairs) is the cross-sectional real-game position pool used to seed the
joint-table collection runs (and other real-game-position needs). It
predates the current PGN corpus in `../pgn/`:

- `twic.pgn` -- 8,452,495 games (raw TWIC archive)
- `twic_filtered.pgn` -- 1,337,714 games (pre-filtered subset of the above --
  header inspection confirms Elo/ply filtering already applied, presumably
  via `../src/eval_tune/filter_pgn.py`'s defaults, `--min-elo 2400 --min-ply
  20`; worth confirming exact args before relying on that filter alone --
  see "Open decisions")
- `lichess_elite_*.pgn` (2013-09 through 2020-05) -- 3,819,130 games, of
  which 3,819,129 (99.99997%) already have both players rated >2000 (the
  "elite" filter is already essentially an Elo>2000 gate)

Measured directly (this session, via a streaming `gawk` pass over every
`[WhiteElo]`/`[BlackElo]` pair, keyed on `[Event` boundaries):

| source | total games | both players >2000 |
|---|---:|---:|
| `twic.pgn` | 8,452,495 | 2,948,359 |
| `lichess_elite_*.pgn` | 3,819,130 | 3,819,129 |
| **combined** | **12,271,625** | **6,767,488** |

So the source corpus can support a cross-sectional pool far larger than
200k even under a strict >2000 rating floor -- the constraint is entirely
"how many do we want," not "is there enough data."

## Sampling method

**5 positions per qualifying game, at fixed fractional points in the game
(not random plies).** This is the key design choice: dividing by game
*length* rather than drawing one random ply per game structurally
guarantees every kept game contributes exactly one position to each of 5
game-stage buckets, instead of leaving stage balance to chance (uniform-
random-ply sampling is naturally skewed toward the middlegame, since games
spend more plies there than grinding out short endgames -- this was the
same problem `sample_eval_calibration_positions.py`'s `--stratify` flag
was built to correct for, after the fact, by material bucket; sampling at
fixed fractions avoids needing that correction in the first place).

- Take the game's full mainline ply count `N`.
- Sample at **6 equal segments, using the 5 *interior* boundaries**
  (1/6, 2/6, 3/6, 4/6, 5/6 of `N`) -- not literal 1/5..5/5. The 5/5 point
  is the final move of the game, which is frequently checkmate, stalemate,
  or immediately pre-resignation -- `go` can't search a terminal board
  meaningfully. Using interior sixths avoids this without losing the
  "5 stages" framing.
- Label the 5 samples `early`, `early_mid`, `mid`, `mid_late`, `late`.
- Skip a boundary (and count it) if it lands on a game-over board, or if
  rounding collapses two boundaries onto the same ply (only possible in
  games just barely over the `--min-ply` floor).

## Filtering: "long enough to be interesting"

Two independent filters, both required:

1. **Rating floor**: both `WhiteElo` and `BlackElo` present and above
   threshold. Recommend **2000** (matches the rating question this plan
   originated from) -- but see "Open decisions" below re: consistency with
   `twic_filtered.pgn`'s own pre-baked filter.
2. **Minimum ply count**: reject games shorter than `--min-ply` (default
   **30** plies = 15 full moves each side). Rationale: with 5 sample points
   needed, very short games (10-move draws by repetition, early
   resignations) either fail outright or produce 5 samples crammed into a
   handful of real plies, which defeats the point of stage separation.
   30 plies gives ~5-6 plies of spacing between consecutive sample points
   even in a borderline-length game.

## Deduplication

Positions are deduped on a **normalized FEN key** -- board placement, side
to move, castling rights, en passant square -- explicitly dropping the
halfmove-clock and fullmove-number fields, which would otherwise make two
functionally-identical positions compare as distinct. Dedup is global
across the whole run (one `seen_keys` set for the whole corpus, not
per-file), since the same well-known opening line recurs across many
games regardless of source file.

Measured on a 776-game smoke-test slice (`lichess_elite_2020-05.pgn`,
first 20,000 lines): overall duplicate rate 0.4%, concentrated almost
entirely in the `early` stage (1.8% there vs ~0% by `mid`). This matches
expectation -- opening theory repeats verbatim across games; middlegame
and endgame positions are close to unique per game. For reference, the
existing `cross_sectional_200k.ep_` itself has ~8.5% duplicate positions
(182,999 unique boards out of 200,000 lines) -- this plan's dedup should
produce a cleaner pool than the file it's replacing, not just a bigger one.

Dedup is a soft design choice, not a hard requirement -- see "Open
decisions."

## Quiescence: deliberately NOT filtered

Considered and rejected. These positions are **search roots** (`setboard`
+ `go`), not eval-only calibration points. The engine's own qsearch
resolves tactics at every node of the tree it searches from these roots
regardless of whether the root itself is quiet -- filtering for quiescent
roots would just mean throwing away the (accurate) tactical-position
share of real games, biasing the pool toward "calm" positions in a way
that doesn't match what the engine's search tree actually looks like
mid-game. This mirrors the reasoning already written into
`sample_eval_calibration_positions.py`'s docstring, which explicitly does
NOT filter on SEE/hanging-material/qsearch-agreement for the same reason.
(Note: this is the opposite choice from `tune_eval_dna.py`'s *quiet*-
position pools, which exist for Texel eval tuning, a genuinely different
use case where the eval function has to approximate game outcome without
search resolving anything.)

## Stats to compute and report (already implemented in the prototype script)

- games scanned / kept / rejected (split by reason: Elo, too-short)
- positions written, duplicate-FEN count and rate (overall and per-stage)
- game-over boundaries skipped
- result distribution (`1-0` / `0-1` / `1/2-1/2`) of kept games -- reported,
  not forced balanced by default (see "Open decisions" on whether to add
  explicit result-balancing quotas)
- per-source-file game counts (to see the twic vs. lichess mix in the
  final pool, since lichess alone vastly outnumbers twic in recent years
  and a naive concat would otherwise skew heavily toward recent online
  blitz)
- per-stage written count and dup rate (the 5 buckets above)
- material-bucket histogram (0 = bare kings .. 7 = full army; same
  bucketing as `eval.c`'s `_MaterialBucket` and the calibration script's
  own copy) -- a secondary phase-sanity check independent of the
  ply-fraction stage labels, since a long grinding decisive endgame can
  still have plenty of material on the board at its "late" sample point
- side-to-move balance (white/black to move) in the final pool

Smoke-test run on 776 games already confirms the script produces sane
numbers on all of the above (376/344/43 result split, bucket histogram
skewed toward full-army as expected for `early`, 54%/46% white/black to
move).

## Performance and timing

Measured: **~197 games/sec single-threaded** (python-chess's PGN parser is
the bottleneck, not file I/O).

| corpus choice | games | est. wall-clock (1 core) |
|---|---:|---:|
| raw `twic.pgn` + `lichess_elite_*.pgn` | 12,271,625 | ~17.3 hours |
| `twic_filtered.pgn` + `lichess_elite_*.pgn` | ~5,156,844 | ~7.3 hours |

Recommend the second option (`twic_filtered.pgn`, not raw `twic.pgn`) --
it's already pre-filtered and 6x smaller, and the raw `twic.pgn` pass
would just spend most of its time discarding sub-2000 games anyway.

**Parallelization is easy and should be used.** Each source file (1
`twic_filtered.pgn` + ~80 monthly `lichess_elite_*.pgn` files) is
independent input; duplicate rate is low (0.4% measured) and concentrated
in the `early` stage, so running each file as its own process (one
`seen_keys` set per process, no cross-process coordination during the
scan) and doing a **cheap post-hoc FEN-dedup merge pass** at the end loses
only a small amount of extra dedup accuracy in exchange for an
~N-core speedup. With even 8 cores this turns a 7.3-hour job into under
an hour; with more cores (or renting cloud CPU -- see the unrelated
earlier discussion on cloud compute cost in this project, same
economics apply: cheap, embarrassingly-parallel, CPU-only) it could run
in minutes.

## Script

Prototype already written and smoke-tested:
`../src/eval_tune/build_cross_sectional_all.py`

Current interface:
```
build_cross_sectional_all.py PGN_PATH [PGN_PATH ...] \
    --min-elo 2000 --min-ply 30 --out tests/cross_sectional_all.ep_
```

Runs sequentially over the given files today. Before the real run, it
should gain:
- a `--jobs N` (or a thin wrapper shell script) to fan out per-file
  processing across cores, plus a merge step that re-applies the
  normalized-FEN dedup across all per-file outputs before writing the
  final `.ep_`
- (optional) a `--balance-results` quota mechanism if the natural result
  distribution turns out to be too skewed once measured on the real
  corpus (the 776-game smoke test's 376/344/43 split is close to even
  between decisive results already at this sample size; whether that
  holds at the full-corpus scale is worth checking on the real numbers
  before deciding this is needed)

## Open decisions (to settle before the real run)

1. **Rating floor: 2000, or match `twic_filtered.pgn`'s existing filter?**
   `filter_pgn.py`'s own default is `--min-elo 2400`; if `twic_filtered.pgn`
   was built with that default, using `--min-elo 2000` downstream doesn't
   loosen twic's effective floor (it's already baked in at 2400) but does
   matter for `lichess_elite` positions, which aren't otherwise filtered
   at build time. Worth explicitly checking what args built
   `twic_filtered.pgn` before treating 2000 and 2400 as interchangeable.
2. **Keep dedup, or drop it?** Dedup is cheap and clearly improves pool
   quality (removes the same 8.5%-ish redundancy the current 200k file
   already has), but it does mean the final position count depends on
   corpus overlap, not just games-kept x 5 -- worth deciding whether an
   exact target size (e.g. "2 million positions") matters enough to
   oversample games and truncate, vs. just taking whatever the filtered +
   deduped pool naturally yields.
3. **Result balancing**: leave natural, or enforce quotas?** Leaning
   toward "measure first, decide after" -- forcing balance before knowing
   whether the real corpus is actually skewed risks discarding good data
   for no reason.
4. **Final target size**: not yet fixed. At 5,156,844 candidate games (the
   `twic_filtered` + `lichess_elite` combination) x 5 positions/game minus
   whatever dedup removes, the ceiling is in the ~20-25 million range --
   almost certainly far more than needed. Worth picking a concrete target
   (e.g. 1-2 million) and either subsampling games up front (cheaper, fewer
   games to parse) or subsampling the output pool after the fact
   (slower, but preserves the natural stage/result/material distribution
   measured on the full run).

## Sequencing

**Do not run until the current joint-table collection run
(`../joint_table_collection/`) finishes.** That run uses all 16 cores this
box supports (`--cpus 1` per shard, deliberately single-threaded,
deliberately using independent shards rather than threads for
parallelism) for what's currently budgeted as a multi-hour run. This job,
even parallelized, would contend directly with it for the same cores.
Revisit this plan and confirm the open decisions above once that run has
completed and its results have been baked into `data.c`.