1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
|
# Four-axis joint FH% table -- next steps, 2026-09-28
Checkpoint: `git stash` (`stash@{0}`, message starting "four-axis (dumb/gap/
late/depth) marginal calibration instrumentation...") holds the working
instrumentation this plan builds on: `fPruneReduceEligible` (search.c), the
`bbMinorAttacks`/`bbRookAttacks` snapshot fix, and four independently-wired
marginal calibration axes (dumb, gap, late, depth), each confirmed clean at
real volume on `tests/ecm_ringers.ep_`, `sd 14`, single position family. Pop
that stash to resume from exactly this point.
## Where each axis landed (marginal, single-position-family sample)
All four measured against the *eligible* leftover baseline (0.31% FH,
matching HEAD's own EFP/LMR eligibility checklist plus losing-capture
admission via cached SEE sign, plus `iAlpha < +NMATE`):
- **dumb** (material hung, `EstimateHangingMaterialCp`): every bucket below
baseline except 1-100cp (~3x baseline, traced to SEE-scored losing
captures + underpromotions -- a real SEE limitation, not a bug; plan is to
exclude this bucket from pruning/reduction eligibility, not fix SEE).
- **gap** (`iAlpha - iEval`): clean, ~4 orders of magnitude, 3.80% (gap<=0)
down to ~0.00% (900cp+), strictly decreasing.
- **late** (this move's ordinal position, `uLegalMoves`): clean, ~23x decline
move 2 (1.61%) to move 27 (0.07%), small uptick in the deep tail (raw-move-
number-vs-fraction-of-list issue, already known from earlier work).
- **depth** (`uDepth / ONE_PLY` at this node, before this move's own
extension/reduction): clean, ~10-15x decline depth 0 (0.44%) to depth 9-13
(~0.00-0.06%). Hypothesis: shallow subtrees are noisier -- the opponent
hasn't had enough plies to find its real defense, so a "leftover" move
spuriously crosses beta more often near the horizon than deep, where the
subtree has room to converge to the move's true (usually bad) value. This
is in tension with the traditional "EFP is safest near the horizon"
argument, which is about *cost per mistake* (small subtree), not
*frequency of mistakes* -- our data says frequency is actually *highest*
right where the traditional argument calls it safest. The tradeoff is
real, not resolved by this data alone.
## Why a joint table now, not a cheap combined estimate
Discussed and rejected the "naive-Bayes combine the four marginals, skip
the joint collection" shortcut as the first move: cheaper, but doesn't
address the actual open question, which is what to do about the fact that
*every* prior attempt in this project to turn a calibrated signal into a
live pruning/reduction win has landed at "roughly even," never a clear win,
regardless of how correctness-verified the signal is. That pattern won't be
fixed by better precision; if it's real, it's more likely explained by
information the search loses when a move gets skipped (history/TT/killer
contributions later parts of the tree would have used) or by distribution
shift (the table is trained under full search, deployed under a search that
now prunes/reduces differently based on it). Building the joint table is
still worth doing -- Scott wants the real thing, not an approximation, and
a short test run is cheap -- but the *first live test* of it must be via
**reduction, not pruning** (see "Consumption, phase 2" below), specifically
because reduction is self-correcting (existing re-search-on-unexpected-
raise machinery) in a way pruning structurally cannot be. If reduction
*also* lands at "even," that is real evidence the ceiling is structural,
not a signal-quality problem, and no further table refinement will fix it.
## Table shape
Same four axes as the marginal work, but joint, cache-shaped, and
coarsened where the marginal curves already went flat:
| axis | marginal buckets | shipped buckets | boundary notes |
|-------|-----------------:|----------------:|-----------------|
| depth | 25 | **12** | exact 0-9 (all real signal/volume lives here), then 10-13 grouped, 14+ overflow |
| dumb | 11 | **11** | unchanged -- already tight, semantically meaningful (0=not dumb, 1-9=100cp bins, 10=901cp+) |
| gap | 11 | **11** | unchanged -- already tight (0=gap<=0, 1-9=100cp bins, 10=901cp+) |
| late | 33 | **15** | raw move 1 is never produced (`uMoveNum` is 1-based) and raw move 2 is the real floor (the `fPruneReduceEligible`/collection gate requires `uLegalMoves > 1`) -- exact moves 2-13 (biggest swings live here), then 14-21 / 22-31 grouped, 32+ overflow. See 2026-09-30 correction note below: the original bucketing shifted by only 1 (assuming raw 1 was the floor), leaving shipped bucket 0 permanently empty. |
Total shipped cells: 12 x 11 x 11 x 15 = **21,780**.
**Axis order in the C array: depth outermost.** Depth is the one axis fixed
for an entire node's move loop; dumb/gap/late all vary move-to-move within
that same loop. Declaring the array as `[SEARCH_DEPTH_BUCKETS][DUMB][GAP]
[LATE]` means every lookup during one node's whole move loop stays inside
one depth-slice: 11 x 11 x 15 = 1,815 cells. At 2 bytes/cell (see below)
that slice is **~3.5KB** -- small enough to stay resident in L1 for the
entire node, not just L2 for the whole table. First touch per node pays one
cache-miss; everything else in that node is hot. The internal order of
dumb/gap/late doesn't matter much for locality (all three vary per-move
similarly) -- pick whatever's most readable; `[depth][dumb][gap][late]`
matches the order we normally discuss them in.
## Cell encoding: thousandths of a percent, not hundredths
`USHORT`, storing `FH% * 1000` (so 1 == 0.001%, 1000 == 1.000%). Max
representable ~65.535%, far more than needed (every real cell we've seen is
under ~6%) -- the range headroom isn't the point, the *low-end* precision
is: real cells run from ~0.00% up through maybe 2-4%, and confirmed-zero
tests and dumb/gap tail buckets both live down in the 0.00-0.10% range
where hundredths-of-a-percent (the old table's unit) was already too coarse
to distinguish "genuinely near-zero" from "true zero." Thousandths gives
10x finer resolution exactly where decisions get made. No special encoding
needed for the high end -- Scott's "anything over ~0.39% is useless to us
anyway" observation is true (pruning/reduction decisions never care whether
a bad cell is 2% or 6%), but it doesn't require clamping or a different
scale; a linear thousandths encoding already wastes no precision there,
because we simply never read the top of the range for a decision. Leave it
be.
Sentinel: `0xFFFF` (`FH_TABLE_SENTINEL`-equivalent) means "below this
run's sample floor, don't trust this cell yet" -- see below.
## Sample floor, this run only
This is a 90-minute preliminary collection, not the eventual continuously-
updated table. For *this* pass, bake with a `MIN_SAMPLES` floor (propose
500, tune after seeing real per-cell counts) -- any cell below the floor
gets the sentinel instead of a real percentage. A consuming (reduction)
lookup that hits a sentinel cell should fall back to the coarser gap-only
or dumb-only marginal estimate rather than treat "insufficient data" as
"confirmed safe." Once this table is run for real (a week, or continuously
in the background, matching the original project's stated intent), the
floor can loosen or come off entirely -- Scott's point that "we don't need
to care whether 0.0% came from 10k or 10B tries" is correct *for a table
that keeps accumulating*, not for a single 90-minute snapshot.
## How to declare and populate it
1. **New raw collection counters** (chess.h `COUNTERS.tree`, mirrored into
`g_Options` the same way the four marginal axes already are): a single
joint 4D array at *marginal* (uncoarsened) resolution --
`UINT64 u64JointTries[SEARCH_DEPTH_BUCKETS_RAW(25)][HANG_CP_BUCKETS(11)]
[GAP_CP_BUCKETS(11)][MOVE_NUM_BUCKETS_RAW(33)]` and the matching
`u64JointFH` -- 25*11*11*33 = 99,825 cells * 16 bytes (two UINT64) =
~1.6MB per counter pair. This is collection-only, lives in
`SEARCHER_THREAD_CONTEXT` (already ~75MB), and is never shipped -- size
doesn't matter here, only in the final baked table. Collecting at full
marginal resolution (not pre-coarsened) preserves the option to try a
different final bucket shape later without re-collecting, matching how
the gap-bucket rework reused the same joint-run pickle by re-bucketing
at bake time instead of re-running collection.
2. **Wiring in `search.c`**: reuse the exact same
`fPruneReduceEligible`-gated block the four marginal axes already sit
in. Compute all four bucket indices (`_HangCpBucket`, `_GapCpBucket`,
`_MoveNumBucket`, `_DepthBucket` -- already written, unchanged) and
increment the single joint cell `u64JointTries[depthBucket][dumbBucket]
[gapBucket][lateBucket]` once per eligible move, `u64JointFH[...]` once
per eligible fail-high, in addition to (not instead of) the four
existing marginal increments -- keep the marginals live during this run
as a cheap cross-check that the joint reduces to what we already
understand.
3. **Reassembly format**: a new `FOURAXISRAW depth dumb gap late fh tries`
line, printed the same checkpoint-friendly way as the old
`FHGAPRAW`/`FHJOINTRAW` lines were (full cumulative dump every N
positions via the suite-report path, last-checkpoint-only on
reassembly, never summed across checkpoints).
4. **Shard/collect**: 16 shards, `--cpus 1` each (matches every prior
collection run in this project -- split.c's parallel helpers don't feed
these counters), `sd 16` or deeper (need real depth to populate the
depth 8-13 buckets at all -- shallower runs structurally can't reach
them), 90 minutes wall-clock per shard via `timeout`, mixed real-game
(`cross_sectional_200k.ep_`) + modest tactical (`ecm.ep_`) position pool,
same ~90/10 blend as prior runs. Output to `/usr/local/tmp/typhoon/`
(persistent, not `/tmp` -- see the disk-full incident earlier this
project). **Do the same smoke-test discipline as every prior run**: 3-5
minute all-16-shards test first, confirm 0% pruned/no crashes/sane disk
growth, *then* commit to the full 90 minutes.
5. **Bake**: python script reads each shard's last checkpoint block,
aggregates `FOURAXISRAW` lines into a dict keyed by the *raw* (25 x 11 x
11 x 33) indices, re-buckets depth and late down to the shipped (12,
15) shape (dumb and gap pass through unchanged), applies the sample
floor, converts surviving cells to thousandths-of-a-percent, emits a C
initializer for `g_uJointFHThousandths[12][11][11][15]` matching the
depth-outermost declaration above, and splices it into `data.c` the same
brace-depth-tracked way prior bakes did (never a fixed-line-count
assumption -- table bodies are huge and shift size on every rebuild).
## Consumption, phase 2 (after the table exists -- not this pass)
Replace the current flat one-ply LMR with a graduated additive term at the
same call site, using the *already-computed* eligibility gate and bucket
indices (no new per-move cost beyond the table lookup itself, since
`fPruneReduceEligible` and all four bucket functions already run on every
candidate move once this lands):
```
uPredictedFH = LookupJointTable(depthBucket, dumbBucket, gapBucket, lateBucket);
extra_reduction_plies = f(-ln(uPredictedFH)) // more permissive than pruning
// ever was -- see the doc's
// own "on reduction vs
// pruning" section for why
// reduction can tolerate a
// much less conservative
// threshold (self-
// correcting via re-search,
// unlike a prune)
```
Exclude the dumb axis's 1-100cp bucket from ever triggering *extra*
reduction beyond whatever the other three axes independently justify --
that bucket is understood to be SEE's blind spot (structural undervaluing
of certain sacrifices/underpromotions), not a real "this move is fine"
signal, and treating it as "safe to reduce more" would be trusting exactly
the population we already know fools the estimate.
First real test: self-play against `head_reference` (current flat one-ply
LMR), same protocol as every prior match in this project (LOWER95 >= 0.5
gate, not raw score; fixed `sd`/`sn` matched to what's actually being
tested). Watch specifically for whether this breaks the "always even"
pattern -- if it doesn't, that's the signal to stop iterating on this
approach rather than build a bigger/more precise table.
## Correction, 2026-09-30: late-bucket-0 was permanently dead, now fixed
Found while reviewing coverage on a fresh 16-shard collection run:
`_ShipLateBucket`/`ship_late_bucket`'s `-1` shift (search.c, dashboard.py)
assumed the real floor for raw `late` was 1 (`uMoveNum` is 1-based). It
isn't -- the collection/consultation gate itself (`fPruneReduceEligible`,
`search.c`, requires `uLegalMoves > 1`) means raw `late` is never less
than 2. So shipped bucket 0 (meant for raw==1) could never receive a
sample: it read `0xFFFF` (sentinel) in every baked table to date, not
because of insufficient collection time but because the bucketing wasted
a slot on an unreachable raw value. This alone accounted for 1,452 of the
21,780 shipped cells (a full `depth x dumb x gap` slice) permanently
reading "no data," inflating the apparent coverage gap by ~6.7 percentage
points in every prior coverage report.
**Fix**: shift by 2 instead of 1, anchored on the real floor (raw >= 2):
raw 2-13 -> shipped 0-11 (singleton), 14-21 -> 12, 22-31 -> 13, 32+ -> 14.
Same bucket widths as before, just correctly anchored. Applied in lockstep
to `search.c`'s `_ShipLateBucket` (live lookup path), `dashboard.py`'s
`ship_late_bucket` (coverage reporting), and the newly-written
`joint_table_collection/bake_joint.py` (which imports `dashboard.py`'s
version rather than redefining it, so it can't drift out of sync). The
table in `data.c` was rebaked from the 2026-09-30 collection run with the
corrected bucketing; real coverage came out to 18,775/21,780 (86.2%)
populated, vs. 17,444/21,780 (80.1%) that the old (wrong) bucketing would
have reported on the same data.
No prior collection data needed to be discarded for this -- it's purely a
bucketing/reporting bug, not a population/gate problem (a separate concern
raised and ruled out the same day; see SPEC.md's corresponding note).
|
