# Four-axis joint FH% table -- next steps, 2026-09-28 Checkpoint: `git stash` (`stash@{0}`, message starting "four-axis (dumb/gap/ late/depth) marginal calibration instrumentation...") holds the working instrumentation this plan builds on: `fPruneReduceEligible` (search.c), the `bbMinorAttacks`/`bbRookAttacks` snapshot fix, and four independently-wired marginal calibration axes (dumb, gap, late, depth), each confirmed clean at real volume on `tests/ecm_ringers.ep_`, `sd 14`, single position family. Pop that stash to resume from exactly this point. ## Where each axis landed (marginal, single-position-family sample) All four measured against the *eligible* leftover baseline (0.31% FH, matching HEAD's own EFP/LMR eligibility checklist plus losing-capture admission via cached SEE sign, plus `iAlpha < +NMATE`): - **dumb** (material hung, `EstimateHangingMaterialCp`): every bucket below baseline except 1-100cp (~3x baseline, traced to SEE-scored losing captures + underpromotions -- a real SEE limitation, not a bug; plan is to exclude this bucket from pruning/reduction eligibility, not fix SEE). - **gap** (`iAlpha - iEval`): clean, ~4 orders of magnitude, 3.80% (gap<=0) down to ~0.00% (900cp+), strictly decreasing. - **late** (this move's ordinal position, `uLegalMoves`): clean, ~23x decline move 2 (1.61%) to move 27 (0.07%), small uptick in the deep tail (raw-move- number-vs-fraction-of-list issue, already known from earlier work). - **depth** (`uDepth / ONE_PLY` at this node, before this move's own extension/reduction): clean, ~10-15x decline depth 0 (0.44%) to depth 9-13 (~0.00-0.06%). Hypothesis: shallow subtrees are noisier -- the opponent hasn't had enough plies to find its real defense, so a "leftover" move spuriously crosses beta more often near the horizon than deep, where the subtree has room to converge to the move's true (usually bad) value. This is in tension with the traditional "EFP is safest near the horizon" argument, which is about *cost per mistake* (small subtree), not *frequency of mistakes* -- our data says frequency is actually *highest* right where the traditional argument calls it safest. The tradeoff is real, not resolved by this data alone. ## Why a joint table now, not a cheap combined estimate Discussed and rejected the "naive-Bayes combine the four marginals, skip the joint collection" shortcut as the first move: cheaper, but doesn't address the actual open question, which is what to do about the fact that *every* prior attempt in this project to turn a calibrated signal into a live pruning/reduction win has landed at "roughly even," never a clear win, regardless of how correctness-verified the signal is. That pattern won't be fixed by better precision; if it's real, it's more likely explained by information the search loses when a move gets skipped (history/TT/killer contributions later parts of the tree would have used) or by distribution shift (the table is trained under full search, deployed under a search that now prunes/reduces differently based on it). Building the joint table is still worth doing -- Scott wants the real thing, not an approximation, and a short test run is cheap -- but the *first live test* of it must be via **reduction, not pruning** (see "Consumption, phase 2" below), specifically because reduction is self-correcting (existing re-search-on-unexpected- raise machinery) in a way pruning structurally cannot be. If reduction *also* lands at "even," that is real evidence the ceiling is structural, not a signal-quality problem, and no further table refinement will fix it. ## Table shape Same four axes as the marginal work, but joint, cache-shaped, and coarsened where the marginal curves already went flat: | axis | marginal buckets | shipped buckets | boundary notes | |-------|-----------------:|----------------:|-----------------| | depth | 25 | **12** | exact 0-9 (all real signal/volume lives here), then 10-13 grouped, 14+ overflow | | dumb | 11 | **11** | unchanged -- already tight, semantically meaningful (0=not dumb, 1-9=100cp bins, 10=901cp+) | | gap | 11 | **11** | unchanged -- already tight (0=gap<=0, 1-9=100cp bins, 10=901cp+) | | late | 33 | **15** | raw move 1 is never produced (`uMoveNum` is 1-based) and raw move 2 is the real floor (the `fPruneReduceEligible`/collection gate requires `uLegalMoves > 1`) -- exact moves 2-13 (biggest swings live here), then 14-21 / 22-31 grouped, 32+ overflow. See 2026-09-30 correction note below: the original bucketing shifted by only 1 (assuming raw 1 was the floor), leaving shipped bucket 0 permanently empty. | Total shipped cells: 12 x 11 x 11 x 15 = **21,780**. **Axis order in the C array: depth outermost.** Depth is the one axis fixed for an entire node's move loop; dumb/gap/late all vary move-to-move within that same loop. Declaring the array as `[SEARCH_DEPTH_BUCKETS][DUMB][GAP] [LATE]` means every lookup during one node's whole move loop stays inside one depth-slice: 11 x 11 x 15 = 1,815 cells. At 2 bytes/cell (see below) that slice is **~3.5KB** -- small enough to stay resident in L1 for the entire node, not just L2 for the whole table. First touch per node pays one cache-miss; everything else in that node is hot. The internal order of dumb/gap/late doesn't matter much for locality (all three vary per-move similarly) -- pick whatever's most readable; `[depth][dumb][gap][late]` matches the order we normally discuss them in. ## Cell encoding: thousandths of a percent, not hundredths `USHORT`, storing `FH% * 1000` (so 1 == 0.001%, 1000 == 1.000%). Max representable ~65.535%, far more than needed (every real cell we've seen is under ~6%) -- the range headroom isn't the point, the *low-end* precision is: real cells run from ~0.00% up through maybe 2-4%, and confirmed-zero tests and dumb/gap tail buckets both live down in the 0.00-0.10% range where hundredths-of-a-percent (the old table's unit) was already too coarse to distinguish "genuinely near-zero" from "true zero." Thousandths gives 10x finer resolution exactly where decisions get made. No special encoding needed for the high end -- Scott's "anything over ~0.39% is useless to us anyway" observation is true (pruning/reduction decisions never care whether a bad cell is 2% or 6%), but it doesn't require clamping or a different scale; a linear thousandths encoding already wastes no precision there, because we simply never read the top of the range for a decision. Leave it be. Sentinel: `0xFFFF` (`FH_TABLE_SENTINEL`-equivalent) means "below this run's sample floor, don't trust this cell yet" -- see below. ## Sample floor, this run only This is a 90-minute preliminary collection, not the eventual continuously- updated table. For *this* pass, bake with a `MIN_SAMPLES` floor (propose 500, tune after seeing real per-cell counts) -- any cell below the floor gets the sentinel instead of a real percentage. A consuming (reduction) lookup that hits a sentinel cell should fall back to the coarser gap-only or dumb-only marginal estimate rather than treat "insufficient data" as "confirmed safe." Once this table is run for real (a week, or continuously in the background, matching the original project's stated intent), the floor can loosen or come off entirely -- Scott's point that "we don't need to care whether 0.0% came from 10k or 10B tries" is correct *for a table that keeps accumulating*, not for a single 90-minute snapshot. ## How to declare and populate it 1. **New raw collection counters** (chess.h `COUNTERS.tree`, mirrored into `g_Options` the same way the four marginal axes already are): a single joint 4D array at *marginal* (uncoarsened) resolution -- `UINT64 u64JointTries[SEARCH_DEPTH_BUCKETS_RAW(25)][HANG_CP_BUCKETS(11)] [GAP_CP_BUCKETS(11)][MOVE_NUM_BUCKETS_RAW(33)]` and the matching `u64JointFH` -- 25*11*11*33 = 99,825 cells * 16 bytes (two UINT64) = ~1.6MB per counter pair. This is collection-only, lives in `SEARCHER_THREAD_CONTEXT` (already ~75MB), and is never shipped -- size doesn't matter here, only in the final baked table. Collecting at full marginal resolution (not pre-coarsened) preserves the option to try a different final bucket shape later without re-collecting, matching how the gap-bucket rework reused the same joint-run pickle by re-bucketing at bake time instead of re-running collection. 2. **Wiring in `search.c`**: reuse the exact same `fPruneReduceEligible`-gated block the four marginal axes already sit in. Compute all four bucket indices (`_HangCpBucket`, `_GapCpBucket`, `_MoveNumBucket`, `_DepthBucket` -- already written, unchanged) and increment the single joint cell `u64JointTries[depthBucket][dumbBucket] [gapBucket][lateBucket]` once per eligible move, `u64JointFH[...]` once per eligible fail-high, in addition to (not instead of) the four existing marginal increments -- keep the marginals live during this run as a cheap cross-check that the joint reduces to what we already understand. 3. **Reassembly format**: a new `FOURAXISRAW depth dumb gap late fh tries` line, printed the same checkpoint-friendly way as the old `FHGAPRAW`/`FHJOINTRAW` lines were (full cumulative dump every N positions via the suite-report path, last-checkpoint-only on reassembly, never summed across checkpoints). 4. **Shard/collect**: 16 shards, `--cpus 1` each (matches every prior collection run in this project -- split.c's parallel helpers don't feed these counters), `sd 16` or deeper (need real depth to populate the depth 8-13 buckets at all -- shallower runs structurally can't reach them), 90 minutes wall-clock per shard via `timeout`, mixed real-game (`cross_sectional_200k.ep_`) + modest tactical (`ecm.ep_`) position pool, same ~90/10 blend as prior runs. Output to `/usr/local/tmp/typhoon/` (persistent, not `/tmp` -- see the disk-full incident earlier this project). **Do the same smoke-test discipline as every prior run**: 3-5 minute all-16-shards test first, confirm 0% pruned/no crashes/sane disk growth, *then* commit to the full 90 minutes. 5. **Bake**: python script reads each shard's last checkpoint block, aggregates `FOURAXISRAW` lines into a dict keyed by the *raw* (25 x 11 x 11 x 33) indices, re-buckets depth and late down to the shipped (12, 15) shape (dumb and gap pass through unchanged), applies the sample floor, converts surviving cells to thousandths-of-a-percent, emits a C initializer for `g_uJointFHThousandths[12][11][11][15]` matching the depth-outermost declaration above, and splices it into `data.c` the same brace-depth-tracked way prior bakes did (never a fixed-line-count assumption -- table bodies are huge and shift size on every rebuild). ## Consumption, phase 2 (after the table exists -- not this pass) Replace the current flat one-ply LMR with a graduated additive term at the same call site, using the *already-computed* eligibility gate and bucket indices (no new per-move cost beyond the table lookup itself, since `fPruneReduceEligible` and all four bucket functions already run on every candidate move once this lands): ``` uPredictedFH = LookupJointTable(depthBucket, dumbBucket, gapBucket, lateBucket); extra_reduction_plies = f(-ln(uPredictedFH)) // more permissive than pruning // ever was -- see the doc's // own "on reduction vs // pruning" section for why // reduction can tolerate a // much less conservative // threshold (self- // correcting via re-search, // unlike a prune) ``` Exclude the dumb axis's 1-100cp bucket from ever triggering *extra* reduction beyond whatever the other three axes independently justify -- that bucket is understood to be SEE's blind spot (structural undervaluing of certain sacrifices/underpromotions), not a real "this move is fine" signal, and treating it as "safe to reduce more" would be trusting exactly the population we already know fools the estimate. First real test: self-play against `head_reference` (current flat one-ply LMR), same protocol as every prior match in this project (LOWER95 >= 0.5 gate, not raw score; fixed `sd`/`sn` matched to what's actually being tested). Watch specifically for whether this breaks the "always even" pattern -- if it doesn't, that's the signal to stop iterating on this approach rather than build a bigger/more precise table. ## Correction, 2026-09-30: late-bucket-0 was permanently dead, now fixed Found while reviewing coverage on a fresh 16-shard collection run: `_ShipLateBucket`/`ship_late_bucket`'s `-1` shift (search.c, dashboard.py) assumed the real floor for raw `late` was 1 (`uMoveNum` is 1-based). It isn't -- the collection/consultation gate itself (`fPruneReduceEligible`, `search.c`, requires `uLegalMoves > 1`) means raw `late` is never less than 2. So shipped bucket 0 (meant for raw==1) could never receive a sample: it read `0xFFFF` (sentinel) in every baked table to date, not because of insufficient collection time but because the bucketing wasted a slot on an unreachable raw value. This alone accounted for 1,452 of the 21,780 shipped cells (a full `depth x dumb x gap` slice) permanently reading "no data," inflating the apparent coverage gap by ~6.7 percentage points in every prior coverage report. **Fix**: shift by 2 instead of 1, anchored on the real floor (raw >= 2): raw 2-13 -> shipped 0-11 (singleton), 14-21 -> 12, 22-31 -> 13, 32+ -> 14. Same bucket widths as before, just correctly anchored. Applied in lockstep to `search.c`'s `_ShipLateBucket` (live lookup path), `dashboard.py`'s `ship_late_bucket` (coverage reporting), and the newly-written `joint_table_collection/bake_joint.py` (which imports `dashboard.py`'s version rather than redefining it, so it can't drift out of sync). The table in `data.c` was rebaked from the 2026-09-30 collection run with the corrected bucketing; real coverage came out to 18,775/21,780 (86.2%) populated, vs. 17,444/21,780 (80.1%) that the old (wrong) bucketing would have reported on the same data. No prior collection data needed to be discarded for this -- it's purely a bucketing/reporting bug, not a population/gate problem (a separate concern raised and ruled out the same day; see SPEC.md's corresponding note).