Add 5D FH% joint table (improve axis) + beta-side static-null-move

table, wire up static null-move pruning; misc searchsup.c tuning;
--cpus=4 fix; multi-pool match_play.py openings.

data.c: g_uJointFHThousandths grows a 5th "improve" axis
([12][11][11][15] -> [12][11][11][16][11]); new
g_uBetaFHThousandths[12][21][11][11] table added. chess.h/searchsup.c/
search.c/split.c: minimal consumer wiring for the improve axis
(ImproveBucket(), extended _JointTablePredictedFHThousandths
signature/callers).

search.c: new static (score-only, no verification search) null-move
cutoff ahead of the existing real null-move check -- when iEval > iBeta
by enough that the new beta-side table predicts a real null-move
search would fail high with >95% confidence, take the cutoff on the
table's say-so instead of paying for the verification search.

searchsup.c: EstimateHangingMaterialCp now gives half credit when the
mover has an unrelated piece already flagged trapped at node entry;
bumped the SEE>=0 check extension from 3/4 ply to a full ply when the
opponent has significant material left; removed the "entering the
endgame extension" bonus (recapture extension and its window check
untouched).

main.c: accept "--opt=value" in addition to "--opt value" on the
command line (a bare "--cpus=4" used to silently fail to match and
leave uNumProcessors at its default of 1); also fixes a latent
off-by-one that could read one past the end of argv for a value-taking
flag given as the last command-line argument.

unix.c: brace-scoping cleanup around signal handler setup, no behavior
change.

eval_tune/match_play.py, test_vs_head.sh: sample opening positions from
multiple PGN pools (twic_filtered.pgn + lichess*.pgn) instead of one,
for more opening diversity in match-play testing.

Brought out of a larger stashed changeset one piece at a time, each
verified independently against a --cpus 1 / --cpus 4 ratio benchmark
to rule out the MP scaling regression that changeset originally
triggered (see SCOTT_IS_A_MORON.md) -- no regression reproduced in any
of the pieces here. Playing-strength impact measured as neutral via
self-play (170 games, llr=-0.04, score=0.500) despite ~1 ply deeper
average search. The stashed changeset's EFP margin rewrite, LMP
threshold rewrite, and a disabled null-move-skip experiment are
deliberately excluded -- those need their own careful, separate
commits and testing. The stash's eval.c retuning is abandoned (measured
~even/worse over a ~700-game overnight self-play run).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

2 files changed, 130 insertions, 55 deletions

diff --git a/src/eval_tune/match_play.py b/src/eval_tune/match_play.py
index 4eecfd6..5033cad 100755
--- a/src/eval_tune/match_play.py
+++ b/src/eval_tune/match_play.py
@@ -64,12 +64,19 @@ _MOVE_RE = re.compile(r"^move (\S+)")
 # and against real engine output. A per-move stats block trails every
 # move, not just script-run suite summaries -- see EBF/first-move-beta
 # discussion in session history.
+# Matches both the old report format ("Searched for 0.3 seconds, saw 123
+# nodes (45 qnodes) ( 969 nps).") and the new, more compact one ("Searched
+# for   0.3s, saw 123 nodes (45 qnodes) (969knps)."): "seconds"/"s", and an
+# optional "k" multiplier before "nps" (new format reports already-scaled
+# thousands-of-nodes-per-second).
 _SEARCHED_RE = re.compile(
-    r"^Searched for\s+([\d.]+) seconds, saw (\d+) nodes "
-    r"\((\d+) qnodes\) \(\s*([\d.]+) nps\)\.")
+    r"^Searched for\s+([\d.]+)\s*(?:seconds|s), saw (\d+) nodes "
+    r"\((\d+) qnodes\) \(\s*([\d.]+)\s*(k?)nps\)\.")
 _DEPTH_RE = re.compile(r"^tellothers d(\d+),")
+# Old: "First move beta cutoff rate was 93.681 percent."
+# New: "First move beta cutoff rate was 93.681%."
 _FIRST_MOVE_BETA_RE = re.compile(
-    r"^First move beta cutoff rate was\s+([\d.]+) percent\.")
+    r"^First move beta cutoff rate was\s+([\d.]+)\s*(?:percent|%)\.")
 # 2026-09-18: per-iteration line UtilPrintPV (util.c) prints once per
 # completed depth during a move's search (both under-xboard "%2u %6d
 # %5u %-12llu %s" and human "%2u[+-]? %5s %-9s %-11llu %s" formats --
@@ -150,23 +157,77 @@ def _dbg(tag, msg):
         print(f"[{tag}] {msg}", file=sys.stderr, flush=True)
 
 
-def sample_openings(pgn_path, n, min_ply=8, max_ply=20, seed=0,
+def _sample_one_opening(fb, file_size, rng, min_ply, max_ply):
+    """One attempt at pulling a random opening position out of an
+    already-open pgn pool file: seek to a random byte offset, find the
+    next game's header block, parse just that one game, and walk a
+    random number of its plies. Returns a FEN, or None if this attempt
+    didn't land on a usable game (truncated tail, too-short game, etc.)
+    -- callers retry elsewhere on None rather than treating it as fatal."""
+    offset = rng.randint(0, max(file_size - 1, 0))
+    fb.seek(offset)
+    fb.readline()  # discard partial line at the seek point
+    # scan forward to the start of the next game's header block
+    start = fb.tell()
+    line = fb.readline()
+    while line and not line.startswith(b"[Event "):
+        start = fb.tell()
+        line = fb.readline()
+    if not line:
+        return None  # landed past the last game in the file; retry elsewhere
+    # read forward to the START of the game AFTER this one (or EOF)
+    fb.seek(start)
+    fb.readline()  # consume this game's own "[Event " line
+    end = fb.tell()
+    line = fb.readline()
+    while line and not line.startswith(b"[Event "):
+        end = fb.tell()
+        line = fb.readline()
+    fb.seek(start)
+    raw = fb.read(end - start).decode("utf-8", errors="replace")
+    game = chess.pgn.read_game(io.StringIO(raw))
+    if game is None:
+        return None
+    moves = list(game.mainline_moves())
+    if len(moves) < min_ply:
+        return None
+    board = game.board()
+    ply = rng.randint(min_ply, min(max_ply, len(moves)))
+    for mv in moves[:ply]:
+        board.push(mv)
+    return board.fen()
+
+
+def sample_openings(pgn_paths, n, min_ply=8, max_ply=20, seed=0,
                      cache_dir="opening_cache"):
-    """Pick n random (roughly) opening positions out of a pgn pool by
-    seeking to random byte offsets and parsing just the one game found
-    there, rather than sequentially chess.pgn.read_game()-ing from the
-    start -- on a several-hundred-MB pool, replaying every game up to
+    """Pick n random (roughly) opening positions out of one or more pgn
+    pools by seeking to random byte offsets and parsing just the one game
+    found there, rather than sequentially chess.pgn.read_game()-ing from
+    the start -- on a several-hundred-MB pool, replaying every game up to
     the sample point (even while mostly skipping them) means real I/O
     and SAN-parsing cost that scales with how far into the file the
     first few accepted samples happen to land, which was observed
     taking minutes for as few as 2 requested openings. Cached like
     tune_eval_dna.py's position_cache, keyed on pgn identity + params,
-    since repeated cycle.sh runs against the same pool shouldn't pay
-    this more than once."""
-    st = Path(pgn_path).stat()
+    since repeated cycle.sh runs against the same pool(s) shouldn't pay
+    this more than once.
+
+    `pgn_paths` may name several pools at once (e.g. the TWIC game
+    archive alongside a curated opening-book PGN) to widen the set of
+    starting positions beyond whatever one file happens to contain --
+    every match otherwise draws its "variety" from a single source,
+    which just means every match looks like that source's opening
+    repertoire. Each attempt picks a source file weighted by byte size
+    (a bigger pool has proportionally more distinct games to offer), so
+    pools aren't implicitly up- or down-weighted just by being listed
+    first or being much smaller than the others."""
+    if isinstance(pgn_paths, (str, Path)):
+        pgn_paths = [pgn_paths]
+    stats = [Path(p).stat() for p in pgn_paths]
+    key_parts = [f"{p}|{st.st_mtime}|{st.st_size}"
+                 for p, st in zip(pgn_paths, stats)]
     key = hashlib.sha256(
-        f"{pgn_path}|{st.st_mtime}|{st.st_size}|{n}|{min_ply}|{max_ply}|{seed}"
-        .encode()
+        ("||".join(key_parts) + f"||{n}|{min_ply}|{max_ply}|{seed}").encode()
     ).hexdigest()[:16]
     cache_path = Path(cache_dir) / f"openings_{key}.txt"
     if cache_path.exists():
@@ -176,49 +237,36 @@ def sample_openings(pgn_path, n, min_ply=8, max_ply=20, seed=0,
         return openings
 
     rng = random.Random(seed)
-    file_size = st.st_size
+    file_sizes = [st.st_size for st in stats]
+    total_size = sum(file_sizes) or 1
+    # Precompute cumulative weights once for weighted source selection
+    # per attempt, rather than re-summing on every pick.
+    cum_weights = []
+    running = 0
+    for sz in file_sizes:
+        running += sz
+        cum_weights.append(running)
+
     openings = []
     attempts = 0
     max_attempts = n * 50 + 200  # generous; each attempt is one seek+one game parse
-    with open(pgn_path, "rb") as fb:
+    file_handles = [open(p, "rb") for p in pgn_paths]
+    try:
         while len(openings) < n and attempts < max_attempts:
             attempts += 1
-            offset = rng.randint(0, max(file_size - 1, 0))
-            fb.seek(offset)
-            fb.readline()  # discard partial line at the seek point
-            # scan forward to the start of the next game's header block
-            start = fb.tell()
-            line = fb.readline()
-            while line and not line.startswith(b"[Event "):
-                start = fb.tell()
-                line = fb.readline()
-            if not line:
-                continue  # landed past the last game in the file; retry elsewhere
-            # read forward to the START of the game AFTER this one (or EOF)
-            fb.seek(start)
-            fb.readline()  # consume this game's own "[Event " line
-            end = fb.tell()
-            line = fb.readline()
-            while line and not line.startswith(b"[Event "):
-                end = fb.tell()
-                line = fb.readline()
-            fb.seek(start)
-            raw = fb.read(end - start).decode("utf-8", errors="replace")
-            game = chess.pgn.read_game(io.StringIO(raw))
-            if game is None:
-                continue
-            moves = list(game.mainline_moves())
-            if len(moves) < min_ply:
-                continue
-            board = game.board()
-            ply = rng.randint(min_ply, min(max_ply, len(moves)))
-            for mv in moves[:ply]:
-                board.push(mv)
-            openings.append(board.fen())
+            pick = rng.uniform(0, total_size)
+            idx = next(i for i, cw in enumerate(cum_weights) if pick <= cw)
+            fen = _sample_one_opening(file_handles[idx], file_sizes[idx],
+                                       rng, min_ply, max_ply)
+            if fen is not None:
+                openings.append(fen)
+    finally:
+        for fb in file_handles:
+            fb.close()
 
     if len(openings) < n:
         print(f"warning: only found {len(openings)} openings "
-              f"(wanted {n}) in {pgn_path}", file=sys.stderr)
+              f"(wanted {n}) across {pgn_paths}", file=sys.stderr)
 
     cache_path.parent.mkdir(parents=True, exist_ok=True)
     cache_path.write_text("\n".join(openings) + "\n" if openings else "")
@@ -279,7 +327,7 @@ def play_one_game(white_engine_path, black_engine_path, scratch_dir, fen,
     engine_args = ["--hash", "256m", "--cpus", "1", "--logfile", "-"]
     with Engine(white_engine_path, f"{scratch_dir}/w", engine_args) as we, \
          Engine(black_engine_path, f"{scratch_dir}/b", engine_args) as be:
-        # Disable book and pondering.  Set time control.  Resign lost positions.
+        # Disable book and pondering.  Set time control.
         # 2026-10-03: Engine.__init__ (tune_eval_dna.py:120) sends
         # "nopost" unconditionally, and root.c:248 gates the
         # per-iteration PV lines on g_Options.fShouldPost -- so those
@@ -294,8 +342,7 @@ def play_one_game(white_engine_path, black_engine_path, scratch_dir, fen,
         # and the default behavior of a harness in active use should not
         # change silently underneath a running experiment.
         startup = ["book name /none",
-                   "set ThinkOnOpponentsTime false",
-                   "set ResignThreshold -1000"]
+                   "set ThinkOnOpponentsTime false"]
         if post:
             startup.append("post")
         for command in startup:
@@ -763,7 +810,16 @@ def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("head_engine", help="path to the baseline/head typhoon binary")
     ap.add_argument("candidate_engine", help="path to the candidate typhoon binary")
-    ap.add_argument("--pgn", required=True)
+    ap.add_argument("--pgn", required=True, nargs="+",
+                     help="one or more pgn pools to sample starting "
+                          "positions from (e.g. --pgn ../../pgn/"
+                          "twic_filtered.pgn ../../pgn/self_play_games.pgn). "
+                          "Each candidate opening's source file is picked "
+                          "weighted by that file's byte size, so listing "
+                          "several pools widens the set of starting "
+                          "positions instead of implicitly favoring "
+                          "whichever is listed first or is largest/"
+                          "smallest.")
     ap.add_argument("--games", type=int, default=500,
                      help="total games; rounded down to an even number "
                           "since each opening is played as a pair "
@@ -837,7 +893,27 @@ def main():
     ap.add_argument("--log", default="match_history.log")
     ap.add_argument("--pgn-out", default="../self_play_games.pgn",
                      help="every played game is appended here as PGN")
+    ap.add_argument("--opening-seed", type=int, default=None, metavar="N",
+                     help="seed for opening-position sampling. Default: "
+                          "a fresh random seed every run, so successive "
+                          "invocations draw a DIFFERENT set of starting "
+                          "positions from --pgn rather than silently "
+                          "replaying whatever sample a previous run "
+                          "already cached under opening_cache/ for these "
+                          "exact (pgn, n, min_ply, max_ply) params -- that "
+                          "caching exists to make a single run's sampling "
+                          "cheap to redo (e.g. a crashed/resumed run), not "
+                          "to pin one fixed opening set across unrelated "
+                          "runs. Pass an explicit value to reproduce a "
+                          "specific run's exact opening set.")
     args = ap.parse_args()
+    if args.opening_seed is None:
+        # SystemRandom, not random.Random/time-based: back-to-back runs
+        # (e.g. a shell loop launching several matches within the same
+        # second) must not collide on the same seed and silently resample
+        # the same "random" set.
+        args.opening_seed = random.SystemRandom().randrange(2**31)
+    print(f"OPENING_SEED={args.opening_seed}", file=sys.stderr)
 
     # --- resolve the per-side time controls once, loudly ---------------
     # Deliberately strict: a time-odds match that silently degrades into
@@ -893,7 +969,7 @@ def main():
     candidate_engine = str(Path(args.candidate_engine).resolve())
 
     n_pairs = args.games // 2
-    openings = sample_openings(args.pgn, n_pairs)
+    openings = sample_openings(args.pgn, n_pairs, seed=args.opening_seed)
     n_games = len(openings) * 2
 
     jobs = []
diff --git a/src/eval_tune/test_vs_head.sh b/src/eval_tune/test_vs_head.sh
index 6f37039..dbddf10 100755
--- a/src/eval_tune/test_vs_head.sh
+++ b/src/eval_tune/test_vs_head.sh
@@ -105,7 +105,7 @@ fi
 python3 ./match_play.py \
         ${HEAD} \
         ${CANDIDATE} \
-        --pgn ../../pgn/twic_filtered.pgn \
+        --pgn ../../pgn/twic_filtered.pgn ../../pgn/lichess*.pgn \
         --games ${GAMES} \
         --workers 18 \
         --post \
@@ -113,5 +113,4 @@ python3 ./match_play.py \
         ${SPRT_ARGS} \
         --scratch /usr/local/tmp/typhoon_match_${LABEL} \
         --log ${LABEL}.log \
-        --pgn-out ../${LABEL}_games.pgn \
         "$@"