typhoon.git/src/TODO, branch master

NNUE search integration: FH% tables, NODE_TYPE refinement, TREEDUMP rewrite

2026-10-23T20:19:17Z

- data.c/chess.h: widen the joint FH% table's late axis 16->17, dedicating the new bucket 0 to a whole-node ("will something here fail high at all") sample rather than sharing it with the first-leftover-move curve. Add an NNUE-eval counterpart table (g_uJointFHThousandths_NNUE) and an NNUE counterpart of the beta-side table, selected at runtime based on whether NNUE eval is active. - search.c/searchsup.c: add _RefineNodeTypeGuess, which uses the FH% tables (via g_NodeTypeGuessTable) to refine a node's NODE_TYPE guess instead of the old heuristic; thread uParentMoveFHEstimate through PLY_INFO so a node can see its parent's prediction for the move that led to it. - generate.c: initialize sKillers[2]/[3].uBonus unconditionally (an uninitialized-stack read caught by valgrind under LTO; harmless today but a real bug). - dumptree.c/chess.h/GNUmakefile: replace the long-dead DUMP_TREE XML tree dump (written for a GUI this project no longer has) with TREEDUMP, a flat, path-keyed move/result trace meant to diff against an identically-instrumented Stockfish build -- see docs/TREEDUMP_FORMAT.md. - docs/DEEPER_PRUNING.md: notes on closing the search-efficiency gap vs Stockfish. - crafty_egtb/stockfish_xboard/typhoon_nnue: wrapper scripts so match_play.py's Engine class can drive Crafty, Stockfish (via PolyGlot), and an NNUE-enabled typhoon the same way it drives HCE typhoon. - tests/lop: rename lop.txt -> lop.ep_; update compare_lop.cpp. - annotate_pgns: add aggregate_marginsafety.py, dedup_pgn.py, fit_lazy_probe.py; update annotate.py/run_nnue_pipeline.py for the NNUE training pipeline. - .gitignore: ignore compile_commands.json (clangd build DB) and eval.nnue (a symlink into the untracked head_reference/ comparison baseline), neither of which belongs in the repo. Co-Authored-By: Claude Sonnet 5

Wire NNUE eval into search, build the training pipeline, and add a fast

2026-10-20T16:26:44Z

move/eval-agreement benchmark against Stockfish (tests/lop) Implements the architecture decisions in src/docs/NNUE.md: HalfKAv2_hm feature set, int16/int8 quantization, per-ply accumulator backup for king moves, and a lazy-eval-ready integration into search.c/eval.c. Adds the full annotate_pgns/ training pipeline (extraction, K-fit/ labeling, quiet filtering, tensor building, training, export) plus an end-to-end orchestrator (run_nnue_pipeline.py) and validation tooling (sanity_check_net.py, compare_vs_stockfish_net.py). Adds tests/lop: a 10k-position list-of-positions benchmark comparing engine eval rank and move choice against pre-computed Stockfish/lc0 references, much cheaper than a full self-play match for day-to-day progress checks. First sd10-matched self-play result against classic HCE: LLR +2.28 (H1=+2.94), -72=119+169 over 360 games, confirming the net is "smarter per node" at matched depth. A real-time (--st 4) match is currently weaker (candidate loses ~3 plies of depth to a ~3.3-3.7x nps gap against an unrecalibrated search), consistent with NNUE.md's adoption-criterion section -- pruning/reduction constants still need retuning for the new eval's score scale before a fair real-time comparison is possible. Co-Authored-By: Claude Sonnet 5

NNUE accumulator incremental maintenance: MakeMove/UnmakeMove hookup

2026-10-15T22:21:26Z

Wires the live NNUE accumulator (SEARCHER_THREAD_CONTEXT.sNnueAccum[2], added in the previous commit) into real incremental maintenance, hooked directly into the piece-movement primitives rather than re-deriving move type from MOVE in a separate pass: - chess.h: shared HalfKAv2_hm feature-index formula/constants (NnueFeatureIndex and friends), moved here from nnuefeatures.c so the reference dump and the real hot-path code can never drift apart. Prototypes for NnueAccumAddFeature/RemoveFeature/Recompute and the (still placeholder, zero-initialized) embedding table. LiftPiece/PlacePiece/SlidePiece/SlidePawn/SlidePieceWithoutSigs signatures gain an NNUE_ACCUMULATOR sNnueAccum[2] parameter. - nnueaccum.c (new): the embedding table storage and the three accumulator operations. Written as straight-line 256-float loops, structured for -O3 -ffast-math auto-vectorization on this box's AVX2+FMA target; explicit SIMD intrinsics and real weight loading are separate, deliberately deferred follow-ups. - move.c: all 8 Lift/Place/Slide(WithoutSigs) primitives now keep sNnueAccum incrementally correct for both perspectives on every call -- except when the piece is a king, where each primitive skips its own-color perspective entirely (that perspective needs a full recompute, not a delta, since every HalfKAv2_hm feature in it is own-king-relative; computing an incremental update for it only to discard it unread would be wasted work). MakeMove backs up that perspective into the current ply's PLY_INFO.sNnueAccumBackup before a king moves (castling or a plain step) and overwrites it via NnueAccumRecompute once the board reflects the final position; UnmakeMove restores it from the same backup slot. DEBUG builds tag the backup with the color it belongs to and poison it after consumption, release builds carry no overhead for either. - nnuefeatures.c: now a thin board-walking wrapper around the shared chess.h formula instead of its own duplicated copy. - testmove.c: TestLiftPlaceSlidePiece passes a local (unread) dummy accumulator pair -- the only caller of these primitives outside move.c, and it never needed NNUE awareness, just a real argument. Verified via the full precommit_check.sh (all 4 gates, including the new combined DEBUG=1 TEST=1 self-test, which exercises MakeMove/ UnmakeMove heavily via TestMoveGenerator's perft counts and TestLegalMoveGenerator) -- confirms nothing crashes and no existing invariant breaks. Does NOT yet confirm the actual property that matters (incrementally-maintained accumulator == recompute-from- scratch) -- no test exercises that yet; next step. Co-Authored-By: Claude Sonnet 5

Finish work to unify search. Play tested, even as expected.

2026-10-15T01:54:36Z

Unfortunately slightly slower.

Loosen LMP pruning thresholds; trim stale comments/dead code in search.c

2026-10-14T17:49:20Z

Raise uLMPStaticMaxPctThousandths across the board (200/200/150/75/50/ 25/12/6/3/1/0 -> 325/300/250/200/150/100/80/25/6/1/0) and LMP_DYNAMIC_MAX_PCT_THOUSANDTHS (1 -> 10000), letting the search go deeper/faster by pruning more leftover moves per the dynamic FH% evidence tables rather than the old hand-picked caps. Partial self-play testing (two ~1k-game SPRT runs, LLR trending up both times, stopped early) supports this but hasn't yet crossed +5 Elo. Also: drop movesup.c's already-commented-out FLEE_BONUS selection-time nudge (superseded by the evidence-gated countermove work), remove search.c's now-stale dated commentary and two unused EFP flags (fTryEFPHere/fThisMoveEFPPruned), and fix searchsup.c's beta-table sentinel to use the named JOINT_TABLE_SENTINEL instead of a bare 0xFFFF. Co-Authored-By: Claude Sonnet 5

* Grab bag of eval.c changes.

2026-10-09T21:18:52Z

* Don't bother not hashing when moves are pruned. * Prune moves move librerally with LMP. Inexplicably, this is slightly shallower than HEAD (despite pruning MORE). It may be because it is slightly slower. It consistently beats HEAD in match_play.py though (3x now).

Continue dynamic move-ordering/LMR session: SPRT-validated at 1770+ games

2026-10-05T16:01:25Z

Gate match vs head (st4.0) shows LLR=+2.89 approaching the H1=+2.94 bound, candidate score 0.5324 (LOWER95=0.5092, ELO=+22.6+/-16.2), candidate depth/ebf both improved (13.98 vs 13.61 avg depth, ebf 3.324 vs 3.464) with comparable nps -- consistent with real search-efficiency gains rather than noise. match_play.py gets additional instrumentation to report these stats; search.c/searchsup.c/generate.c/chess.h/eval.c carry the underlying move-ordering/LMR changes from this session's line of work. Co-Authored-By: Claude Sonnet 5

Dynamic move-ordering/LMR session: honest FH tracking, decoupled admission/magnitude, bugfixes, tooling

2026-10-04T15:17:52Z

Lands the 2026-09-28..10-03 move-ordering investigation (full history in lmr_testing/*.md). Summary of what ships: LMR/move-ordering redesign (searchsup.c, dynamic.c, data.c, chess.h): - GetLMRReduction's admission decision (do we reduce at all) is now decoupled from magnitude (how much): LMR_STATIC_ADMIT_THOUSANDTHS=500 gates admission via the 4D joint fail-high table (data.c) plus a countermove-match veto; magnitude is a flat Crafty/Ethereal-shaped log(depth)*log(moves) table (g_iLMRQuietReduction), which measured as good or better than the old per-tier joint-table payout while being structurally immune to the per-bucket miscalibration that sank the tiered approach. g_FailHighs honesty fix (real 0/0 init, not a fake prior) underlies the dynamic-danger signal. - Retired NumLeftoverMovesToSelect's depth-indexed selection budget -- evidence showed the leftover pool has real findable signal a bailout was discarding. - ShouldLatePrune (overt late-move pruning) implemented but disabled at its call site -- measured net-negative at the tested threshold; kept for reference and as a base for a future hang-bucket-9-targeted retry. - PRUNE_VERIFY, PRUNE_LIVE, DISABLE_LMR: opt-in diagnostic build flags (GNUmakefile), off in the standard profile. Bug fixes found and fixed while exercising this work: - JOINT_EXTRA_TIER_COUNT 4->6 (chess.h): fixed an out-of-bounds counter write reachable at remaining depth >= 14 (unreachable at sd10, routine in real search/self-play). - Node-type instrumentation (search.c, root.c, searchsup.c): fixed four compounding defects that produced impossible percentages (e.g. 106.2% of PV nodes) -- wrong denominator, qsearch misclassified as main search, untyped early-exit nodes, and fail-low tests using the in-place-raised alpha instead of the initial one. - FH-rate unit unification (search.c, searchsup.c, dynamic.c, chess.h): GetMoveFailHigh* returned per-mille while g_uJointFHThousandths is thousandths-of-a-percent, two different units sharing a "_THOUSANDTHS" suffix. Unified to one scale, renamed to "_PCT_THOUSANDTHS". RateBucket's band edges fixed to match. - QSearch option B (search.c): waives the opponent-stand-pat requirement at NODE_PV only (sound there since the exact score propagates; not sound at null-window nodes). iExtensionAmount unit fix (was 0/2 into a ONE_PLY-scaled field). pi->iEval now published when Eval() runs. - eval_tune/match_play.py local_ebf: was reporting 0.000 in every match since 2026-09-18 (regex only matched the trailing-stats parser, not the live move-scan loop; engines were also spawned with "nopost" so root.c never emitted the lines at all). True per-ply EBF is ~2.0, not the ~3.9 every prior match report implied. - eval_tune/test_vs_head.sh: positional-arg shift ordering bug caused every no-extra-args run to silently rebuild ./typhoon with a non-standard profile and, separately, could feed a flag's value in as the candidate binary path. - PERF_COUNTERS cleanup (eval.c, search.c): moved 3 leaked rdtsc sites to the existing EVAL_TIME guard, deleted two concluded one-off diagnostics (u64HeadStyle*, u64FHJointHist/RateBucket/AttemptsBucket) and their report plumbing, and re-gated a per-move hang-material estimate behind FH_TABLE_COLLECTION instead of running it unconditionally. Infrastructure: - match_play.py time-odds support (--odds/--st-head/--st-candidate), validated against the engine seat rather than color. - Wilson-upper-bound re-bake of the joint table (data.c): cells store a confidence-aware bound instead of a raw point estimate, so a consumer can threshold on the value alone; monotone, so every consumer became strictly more conservative. Verified behavior-neutral for LMR. Known-open items (not resolved here, see lmr_testing/GET_LMR_REDUCTION.md section 10 and 2026-10-03_UNCOMMITTED_TRIAGE.md): the self-play gate on this candidate is underpowered (+5 Elo, 95% CI roughly [-6, +17]); the curated-suite solve-count deltas used earlier in the investigation turned out to be noise-dominated cancellation when diffed per position, not the clean signal they first appeared to be. Co-Authored-By: Claude Sonnet 5

Makes recogn.c use g_iDrawScore. Some refactoring in the pawn

2026-09-20T15:13:57Z

hash / _EvalPawn including some scoring changes. Self-play positive.

Redesign lazy/super-lazy eval margins with position-specific percentile

2026-09-18T16:43:17Z

tables, and fix four real search/recognizer bugs found while validating it Main theme: replace the flat, heavily-padded LAZY_EVAL_BASE_MARGIN / SUPER_LAZY_MARGIN_BY_ARMY constants with per-material-bucket percentile tables (king safety by attacker army, residual by own army, separate tables for the lazy vs super-lazy baselines) derived from a 30k-position calibration pool, targeting a measured ~1-1.5% exceeded-margin rate instead of an unmeasured "add lots of headroom" floor. QSearch's FUTILITY_BASE_MARGIN_LAZY/SUPERLAZY were re-derived to match the new, now-meaningful (p50-based) iPositional this produces. Self-play confirms the working tree beats both HEAD, a more liberal (~5% miss target) variant, and a more conservative (near-zero miss) variant of the same margin mechanism. Along the way, an initial self-play regression (this margin work bundled with several changes in one pass, one of which broke things) led to a day of ZFS-snapshot bisection and then targeted DEBUG-build fuzzing against a stratified 200k-position corpus (tests/cross_sectional_200k.ep_, new), which is what actually found and fixed four distinct, real, pre-existing engine bugs -- none of them related to the margin redesign itself: - searchsup.c (CommonSearchInit): three early-return paths (timer abort, draw, too-deep bailout) left a node's own PV[] row stale instead of clearing it, letting an ancestor's UpdatePV splice a leftover move from a completely unrelated node into the displayed PV -- root cause of a WalkPV/MakeMove assert ("moving a piece to capture, but nothing was on the target square"). - dynamic.c (RecordEnprisePieceAtPly): the two-slot en-prise dedup compared (square, piece) instead of square alone, so a same-square re-record with a different piece (legitimate -- a fail-high capture's victim is restored by UnmakeMove and can differ across calls at the same ply) shifted a duplicate square into both slots, tripping ValueOfMaterialInTroubleDespiteMove's two-distinct-squares assert. - recogn.c (_RecognizeKBKP): the wrong-colored-bishop corner-fortress draw shortcut fired even when the weak side has its own pawn(s), which can hand the strong side real zugzwang resources the fortress logic doesn't account for -- produced a false hard draw bound where EGTB shows a near-forced loss. - eval.c (_EvalPasserRacesAgainstLoneKings): a rook pawn's race was credited as winning even when the strong side's own king sits exactly on its own pawn's queening square (distance 0 specifically -- adjacent is fine, the square is empty) -- a classic stalemate resource the race heuristic can't see. Only reachable through _RecognizeKPK's pawn-hash-cache-hit path, which is why testrecogn.c's TestRecognExhaustiveKPK reported "0 wrong" the whole time; extended it to cover both color assignments and the full board (no mirror-shortcut) and documented the remaining cache-hit coverage gap for a future session to close. Also: match_play.py gains a local_ebf metric (geometric-mean of depth-normalized node-growth ratios between consecutive completed iterations) alongside the existing cumulative nodes**(1/depth) ebf, which turned out to be misleading when comparing engines with different lazy-exit aggressiveness -- and probe.c/hash.c pick up two small, independently-found correctness fixes (TB root-probe promotion piece missing its color bit; ASSERT(g_pHashTable) crashing on the legitimate "--hash none" config) noticed along the way. Co-Authored-By: Claude Sonnet 5