1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
# Lazy/super-lazy eval margin redesign (2026-09-17)
## What changed and why
Original goal: speed up `Eval()` (eval.c) by letting more nodes exit
through the cheap "super lazy" (material-only, runs before `_EvalPawns`)
and "lazy" (material + pawns + passers + bad-trade + bishop-pairs) tiers
instead of paying for full eval, motivated by comparing against Crafty's
much higher NPS.
The old margin construction (still visible via `git show HEAD:src/eval.c`
at commit `4a7ec9a`) was a patchwork: a flat `LAZY_EVAL_BASE_MARGIN` (75)
seed, widened by a `CountKingSafetyDefects`-indexed king term (`_EvalKing`'s
own slow bitboard scan, called *again* just to estimate a margin) plus a
flat residual constant, then `MAX()`'d against a separately-measured
flat floor per material bucket. Super-lazy was a single flat
`SUPER_LAZY_MARGIN_BY_ARMY[8]` lookup, no per-position information at all.
Both had real problems: `CountKingSafetyDefects` is slow despite being
bitboard-based (defeats the point of being in the *cheap* path), and the
`MAX()`-against-a-flat-floor pattern silently overrode the
position-specific estimate in most cases (found and fixed twice this
session, once per tier).
## The new construction
For **both** tiers, at every node, cheap **O(1) table lookups only** (no
`CountKingSafetyDefects`, no attack-table generation):
```
margin_a = MAX0( (king_p_hi[my_bucket] - king_p_lo[opp_bucket])
+ (resid_p_hi[opp_bucket] - resid_p_lo[my_bucket]) )
margin_b = MAX0( (king_p_hi[opp_bucket] - king_p_lo[my_bucket])
+ (resid_p_hi[my_bucket] - resid_p_lo[opp_bucket]) )
```
- `margin_a` gates the fail-high check (`score - margin_a > beta`):
soundness requires assuming the most pessimistic-for-me,
optimistic-for-opponent outcome for each term.
- `margin_b` gates the fail-low check (`score + margin_b < alpha`):
the mirror image.
- **King term**: bucketed by the **attacker's** own army scaler
(`MINU(7, armyScaler/4)`, 8 buckets, 0-31 scaler range) -- danger to a
king is a function of the attacker's material, not the king-owner's.
Shared between both tiers (`iKingP2ByAttackerArmyOnly` /
`iKingP98ByAttackerArmyOnly` in eval.c).
- **Residual term** (mobility/trapped/passers-not-yet-counted/etc.):
bucketed by **each side's own** army scaler. Two separate tables
because the two tiers have different baselines to estimate *from*:
- `iResidualP2/98ByOwnArmy` -- regular lazy's baseline already
includes pawns/passers/bad-trade/bishop-pairs, so this residual
only needs to cover what's left (mobility, trapped pieces, etc).
- `iSuperLazyResidualP2/98ByOwnArmy` -- super-lazy's baseline is pure
material, so this residual is much wider (includes everything
regular lazy's own residual excludes).
`_hi`/`_lo` are currently p98/p2 as shipped, but genuinely **per-bucket
tunable** -- see "How to keep tuning" below. The king table has very
little spread above p90 in every bucket (already checked against the
full histogram) and was deliberately left alone throughout tuning;
all of today's percentile-adjustment work only ever touched the two
residual tables.
`iPositional[side]` (the value fed back to QSearch's futility pruning,
`search.c`'s `FUTILITY_BASE_MARGIN_BY_SOURCE[tier] + iPositional[side]`)
is now a **separate**, deliberately different statistic: a **p50**
(median, not tail-bound) typical-value estimate of that side's own
missing terms, matching what full-eval's own `iPositional` actually is
(a real point estimate, not a safety margin). See `iKingP50By...` /
`iResidualP50By...` / `iSuperLazyResidualP50By...` in eval.c. Don't
confuse these with the p98/p2 margin tables -- different statistic,
different purpose (bias correction vs. soundness bound).
## Where the histogram data lives
`eval.c`'s `_PersistRawHistogramsToFile()` (called automatically every
time a `DUMP_RESIDUAL_SAMPLES` build runs `calibrate residual`) writes
**every raw bin** of all three histograms (residual, sl_residual, king)
to `/var/tmp/typhoon/residual_raw_histograms.txt` -- not just the
handful of percentile columns that happen to be printed to the console.
This directory (`src/eval_tune/lazy_eval_margins/`) has copies from the
2026-09-17 "guess #2" 30k-position calibration run
(`residual_raw_histograms_guess2_30k.txt`) plus the reader script
(`hist_percentile.py`) -- **copy the fresh file here after any future
30k run you want to keep**, since `/var/tmp` is scratch and could be
cleared.
To read *any* percentile out of a saved histogram, without re-running
anything:
```sh
python3 hist_percentile.py residual_raw_histograms_guess2_30k.txt # default spread
python3 hist_percentile.py residual_raw_histograms_guess2_30k.txt 0.93 0.965 # exact percentiles you want
```
## How to keep tuning
1. **Build**: `gmake clean && gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 DUMP_RESIDUAL_SAMPLES=1 CALIBRATE_MARGIN_SAFETY=1`
2. **Run the 30k stratified pool** (both commands in one invocation --
`calibrate residual` persists the histogram file, `calibrate
marginsafety` prints the real exceeded rate per **material** bucket
for both tiers):
```sh
./typhoon --cpus 1 --hash 256m --logfile /var/tmp/typhoon/run.log \
--batch --command "force; book name /nonexistent.book.bin; sd 6; \
script ../tests/eval_calibration_sample_strat30k.ep_; \
calibrate residual; calibrate marginsafety" \
> /var/tmp/typhoon/run.stdout 2>&1
```
Takes roughly 20-25 minutes. Run it in the background
(`run_in_background`/`&`), don't block on it.
3. **Read the exceeded rate per material bucket** from the
`calibrate marginsafety` output (`exceeded / n`, printed separately
for "Regular lazy" and "Super lazy", by **material** bucket --
combined army scaler / 8, a *different* bucketing than the
percentile tables' per-side scaler/4 buckets -- see the caveat
below).
4. **Copy the fresh `/var/tmp/typhoon/residual_raw_histograms.txt`**
into this directory before it can get lost.
5. **Pick new percentiles per per-side bucket** using
`hist_percentile.py` against the fresh file, and edit
`iResidualP2/98ByOwnArmy` / `iSuperLazyResidualP2/98ByOwnArmy` in
eval.c directly (keep each bucket's low/high pair **symmetric** --
e.g. p95/p5 together, never p98 mixed with p10 in the same bucket).
6. **Rebuild and re-run step 2** to verify the real effect.
### The two gotchas that cost real time this session
- **The 500-position "quick check" sample is unreliable for this
specific measurement.** It's fine for cheap direction-of-travel
checks on most things, but exceeded-rate is a rare-event percentage,
and 500 positions (not even 500 independent draws -- a single game's
search tree can dominate a material bucket's node count) can be off
by 2-5x from the true 30k-pool rate. One bucket (super-lazy material
bucket 1) got tightened based on a 500-position reading of 0.035%,
then the real 30k number came back at 1.13% -- a dangerous-direction
miss. **Always confirm with the full 30k pool before trusting a
number**, especially anything under ~1%.
- **Changing one bucket's margin changes measured behavior in OTHER
buckets too**, even ones whose own table entries never moved. This
isn't a bug -- `Eval()`'s exit decisions are real search behavior, so
widening/narrowing any margin reshapes the actual search tree
(different alpha/beta bounds propagate differently), which changes
which exact nodes land in *every* material bucket, not just the one
you touched. Material bucket 3's exceeded rate moved from 0.21% to
0.94% to 0.65% to 1.44% across different runs despite its own table
entries never changing once. Don't chase a bucket that "moved" if you
didn't touch its inputs -- check what else changed first.
- **Material buckets and per-side buckets are different bucketing
schemes.** `calibrate marginsafety` reports by **material bucket**
(`_MaterialBucket`: combined army scaler for both sides, /8). The
percentile tables you actually edit are indexed by **per-side**
bucket (`armyScaler/4`, one bucket per side, looked up twice -- once
as "my bucket", once as "opponent's bucket"). A single material
bucket (e.g. combined scaler 24-31) is populated by many different
per-side bucket combinations (3+4, 2+5, 1+6, 0+7...), so there's no
clean 1:1 way to "fix material bucket 3" by editing "per-side bucket
3" alone -- expect diffuse, imprecise control, not a dial.
## On the "insane" 2% idea
Structurally straightforward: pick a looser matched percentile pair
(e.g. p90/p10, or go outside the currently-recorded columns using
`hist_percentile.py` against a fresh histogram) for whichever buckets
you want at ~2%, following the same edit-rebuild-reverify loop above.
Two things worth checking once you have it:
1. **NPS gain isn't guaranteed to scale linearly with the exceeded-rate
relaxation** -- this session found the NPS win from the whole
redesign was much smaller than expected (see the `git log`/session
history around `EvalLookForDrawishSituations` getting added to both
cheap tiers, and `fBareKingEitherSide`'s two unconditional
`TotalNonPawnCount` calls on every node -- both are new fixed costs
that eat into any margin-driven exit-rate gain). Worth re-checking
the `Eval exit breakdown` / new `calibrate exitrate` numbers
(below) alongside NPS, not just NPS alone.
2. **`FUTILITY_BASE_MARGIN_LAZY`/`SUPERLAZY`** (search.c/chess.h) were
calibrated against *this* `iPositional` distribution
(`CALIBRATE_QSEARCH_FUTILITY`, see chess.h's derivation comment near
those two `#define`s) -- if the eval-exit margins move a lot, especially
in a direction that changes how often/how confidently each tier
exits, it's worth re-running that calibration too rather than
assuming the existing 350/175 values still hold.
## New tooling added this session (always available, no build flag)
- **`calibrate exitrate`** -- prints eval exit rate (% super lazy / %
lazy / % full) **by material bucket**, not just the flat aggregate
"Eval exit breakdown" line every search already prints. Complements
the exceeded-rate data: a bucket with low exit rate AND low exceeded
rate has real untapped speed headroom; a bucket already near its
target exceeded rate with a healthy exit rate is well-calibrated.
Implemented in eval.c (`g_u64SuperLazyEvalsByBucket` /
`g_u64LazyEvalsByBucket` / `g_u64FullEvalsByBucket`,
`DumpEvalExitRateByBucket`) and wired into command.c.
|
