path: root/src/eval_tune/compare_vs_crafty.py

#!/usr/bin/env python3
"""
Cross-engine static-eval audit: compare typhoon's `eval` command output
against a *crowd* of other hand-crafted-eval engines (Crafty, Kiwi, Fruit,
GNU Chess -- all pre-NNUE, additive-term evals, so genuinely comparable to
typhoon's own, unlike Stockfish's NNUE), rank the positions where typhoon
is the outlier against the crowd's median, and attribute the gap back to
typhoon's own named EVAL_TERM lines (from a build with EVAL_DUMP=1).

Methodology, and the gotchas that cost real time to find (2026-09-2x
eval audit session):

  - Build typhoon with EVAL_DUMP=1 (see CLAUDE.md's build instructions)
    to get the per-term breakdown table, not just the "Static eval:"
    total. The breakdown itself is a SEPARATE command, `dump eval`
    (EvalTraceReport() in testeval.c) -- it reports on the terms
    recorded by the *last* Eval() call, so the script sends `eval` then
    `dump eval` back to back for every position, not `eval` alone.

  - Run typhoon with --hash none. Not an optimization -- a correctness-
    adjacent perf fix. SetBoardCommand -> PreGameReset(FALSE) ->
    ClearHashTable() unconditionally, so every single `setboard` with a
    real --hash size does a full memset of the whole hash table. Over
    2000 positions this was the entire difference between ~46s and
    ~2.2s (21x) for byte-identical output. Always pass --hash none for
    eval-only batch runs; there is no search happening, so there is
    nothing to hash.

  - Feed positions via `script <file>` (one `setboard <fen>` / `eval`
    pair per line), not a single giant --command string. A few thousand
    positions' worth of inline command text can exceed the shell's/
    kernel's argv length limits; `script` reads from a file instead and
    has no such ceiling.

  - Only ONE process per engine for the whole batch -- every engine here
    correctly returns different, real scores across repeated
    setboard/eval calls in a single session; none needs restarting per
    position. (Verified directly this session by feeding
    /usr/local/bin/crafty three visibly different positions in one
    stdin session and getting three different totals.)

  - Use /usr/local/bin/crafty (whichever `crafty` resolves to on PATH),
    NOT ~/programming/chess/crafty-25.2/crafty (formerly ~/crafty/crafty
    -- that whole directory was moved to this path). The latter is a
    different, older build (confirmed via md5) that a sibling script
    here (compare_king_safety_crafty.py) found DOES silently reuse the
    first position's cached score on a second setboard+score in one
    session -- that bug is specific to that binary, not a general
    Crafty limitation, and does not reproduce against
    /usr/local/bin/crafty. If you ever see suspiciously identical
    Crafty totals across different positions, check which crafty
    binary you're actually calling before assuming a new engine bug.

  - Crowd engines each have their own sign convention for the "eval"
    command's printed number, confirmed empirically (setboard a position
    with white down a whole knight and check the sign/magnitude), not
    assumed from reading their source alone:
      * crafty `score`, kiwi `eval` -- print the score from WHITE's
        perspective already, no flip needed.
      * fruit `eval` -- prints from the SIDE-TO-MOVE's perspective
        (negamax convention: eval.cpp's `eval()` explicitly negates for
        Black at the very end). Must flip sign when it's Black to move
        to get a White-perspective number comparable to the others.
      * gnuchess (gnuchessr) `score` -- ALWAYS prints "(white to
        move)" and an always-White-relative number, regardless of the
        loaded position's actual active color. Confirmed empirically:
        loading the identical position via two xpos files differing
        only in trailing "w"/"b" produced byte-identical output both
        times. Root cause (read, not guessed): `xget` -> GetXGame()
        (dspcom.c) sets the `computer` global from the file's color
        field but never touches the separate `player` global that
        ScorePosition()'s perspective and the printed "(%s to move)"
        label actually key off -- `player` is a zero-initialized global
        that only search.c's search loop ever updates, so outside of an
        actual search it just stays at its BSS-zero value (White). Net
        effect: harmless for this script (means no stm-relative flip is
        ever needed for gnuchess) but it DOES mean gnuchess's score
        never reflects tempo/mobility-from-whoever's-actually-on-move
        nuance -- a known accuracy ceiling on this crowd member, not a
        bug in this script.
    Getting a sign convention wrong silently halves the apparent
    disagreement rate (a sign-flipped engine agrees with the crowd
    exactly when it should maximally disagree, and vice versa) rather
    than crashing, so it does not announce itself -- always spot check
    a lopsided material position per engine before trusting a new one's
    numbers.

  - Kiwi's own `kiwi` binary in ~/programming/chess/kiwi can be stale
    relative to its own source (confirmed: mtimes showed the checked-in
    binary predated winboard.cxx/engine.cxx, and it silently mis-parses
    `eval` as an illegal `usermove` rather than erroring, because
    command-table dispatch on prefix match falls through to
    cmd_OpponentMoves on a lookup miss) -- there was already a correctly
    rebuilt `kiwi.new` sitting alongside it in that same session, which
    is what this script actually points at
    (KIWI_BIN, below). If kiwi's numbers ever look wrong, rebuild with
    `env OBJDIR=. LD=clang++ CC=clang++ gmake` from ~/programming/chess/kiwi
    (its Makefile uses GNU-make-only `$(addprefix ...)`, so it must be
    `gmake`, not BSD `make`; g++ doesn't exist on this box, hence the
    clang++ overrides) and re-point KIWI_BIN at the fresh binary.

  - gnuchess (gnuchessr, ~/programming/chess/gnuchess/src) has no
    `setboard`/`position fen` command at all -- the only way to load an
    arbitrary position is `xget`, an interactive prompt that reads a
    FILE in gnuchess's old xboard.position.read format (just FEN's
    piece-placement and active-color fields, one line, no
    castling/ep/clock -- castling rights are simply lost, an inherent
    approximation for this crowd member). `_build_gnuchess_script`
    writes one such tiny file per position under
    /tmp/typhoon/eval_audit/gnuchess/ rather than trying to reuse one
    filename, since the position must be fully written before the
    single stdin-scripted process reaches that `xget` line. Also: the
    process needs one leading blank line to satisfy its startup
    "Enter [moves] minutes[:sec]..." time-control prompt before the
    "Your move is?" loop (where `xget`/`score`/`quit` are valid
    commands) even begins reading commands -- a script missing that
    blank line gets its first real command silently swallowed by the
    time-control prompt instead. Also needs its OWN directory as cwd
    (same gotcha class as Glaurung/kpk.bin in an earlier version of
    this crowd): run from elsewhere it prints "NO LANGFILE (file
    gnuchess.lang not found)" and exits immediately instead of falling
    back gracefully.

  - Compare typhoon against the crowd's MEDIAN, not vs. any single
    other engine and not vs. the mean. Median is robust to any one
    crowd engine being the actual outlier on a given position (e.g. a
    known-narrow specialization one of them has and the others don't);
    a mean lets one bad engine drag the "consensus" toward itself.
    Historically (see the old Crafty-only version of this script)
    averaging in an NNUE engine (Stockfish) made typhoon look like the
    outlier for tracking a hand-crafted eval correctly (9 of 10 checked
    outliers were actually typhoon+Crafty agreeing within ~0.3 pawns
    with Stockfish alone ~12-15 pawns off) -- same failure mode extends
    to any structurally-different evaluator diluting the crowd, so keep
    the crowd restricted to same-family (additive, hand-tuned) evals.

  - This box's kern.ipc.semmni (max SysV semaphore sets) is only 50.
    Every typhoon process leaks its semaphore set on `kill -9` (a clean
    `quit`/EOF releases it; a hard kill does not, regardless of the
    unix.c pthread-mutex/condvar rewrite -- that fixed a different
    leak). Repeatedly killing hung/timed-out typhoon processes during
    debugging can silently exhaust the limit; once exhausted, EVERY new
    typhoon process falls back to a ~1000x-slower 100ms polling loop
    for input, which looks exactly like a hang or a new performance
    regression. Check with `ipcs -s | grep $(whoami) | wc -l` before
    concluding a slowdown is a code bug; clean up with `ipcrm -s <id>`
    for each leaked entry (or `for id in $(ipcs -s | grep $(whoami) |
    awk '{print $2}'); do ipcrm -s "$id"; done`).

Usage:

    python3 compare_vs_crafty.py <fen_source> [--limit N] [--top K]
        [--engines crafty,kiwi,fruit,gnuchess]

  <fen_source> is either a path to an .ep_-style file (lines of
  "setboard <fen>" / "go", as in ../tests/*.ep_) or a plain file of one
  FEN per line. Requires a typhoon binary built with EVAL_DUMP=1 at
  ../typhoon (build it first: `gmake clean && gmake -j5 GENETIC=1
  PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 EVAL_DUMP=1` from src/), plus
  whichever crowd engine binaries are selected (all on by default;
  see ENGINE_SPECS for their expected paths).

Output: prints the top-K disagreements (|typhoon(white) -
crowd_median(white)|), each crowd engine's own white-perspective score
for context, and typhoon's largest contributing EVAL_TERM line(s) from
its own perspective -- a quick first read on which term is driving the
gap, before digging in by hand.
"""
import argparse
import re
import statistics
import subprocess
import sys
from pathlib import Path

SRC_DIR = Path(__file__).resolve().parent.parent
TYPHOON_BIN = SRC_DIR / "typhoon"
SCRATCH = Path("/tmp/typhoon/eval_audit")

CHESS_DIR = Path.home() / "programming" / "chess"

# Every crowd engine speaks a different protocol/command for "set this
# position, print static eval." `kind` selects how the position+eval
# commands are framed; `parse_re` must capture exactly the numeric score
# (possibly negative) as group 1; `scale` converts its native units to
# pawns; `stm_relative` marks engines whose printed number is from the
# SIDE TO MOVE's perspective (needs a sign flip on Black-to-move
# positions) rather than already being White-relative.
ENGINE_SPECS = {
    "crafty": dict(
        binary="crafty", cwd=None, kind="xboard",
        startup="ponder off\n", pos_fmt="setboard {fen}\n", eval_cmd="score",
        parse_re=r'^total\.+\s+([+-]?\d+\.\d+)', scale=1.0, stm_relative=False,
    ),
    "kiwi": dict(
        binary=str(CHESS_DIR / "kiwi" / "kiwi.new"), cwd=None, kind="xboard",
        startup="", pos_fmt="setboard {fen}\n", eval_cmd="eval",
        parse_re=r'^Static eval\s*=\s*([+-]?\d+)', scale=0.01, stm_relative=False,
    ),
    "fruit": dict(
        binary=str(CHESS_DIR / "fruit" / "fruit"), cwd=None, kind="uci",
        startup="", pos_fmt="position fen {fen}\n", eval_cmd="eval",
        parse_re=r'info string eval\s+([+-]?\d+)', scale=0.01, stm_relative=True,
    ),
    "gnuchess": dict(
        binary=str(CHESS_DIR / "gnuchess" / "src" / "gnuchessr"),
        cwd=str(CHESS_DIR / "gnuchess" / "src"),
        kind="gnuchess", scale=0.01, stm_relative=False,
        # Not anchored at line-start: gnuchess's "Your move is? " prompt
        # precedes "Static score..." on the SAME line, unlike every
        # other engine here which prints its own line cleanly.
        parse_re=r'Static score \(\w+ to move\):\s*([+-]?\d+)',
    ),
}


def extract_fens(path: Path) -> list[str]:
    fens = []
    for line in path.read_text().splitlines():
        line = line.strip()
        if line.startswith("setboard "):
            fens.append(line[len("setboard "):].strip())
        elif line and not line.startswith(("go", "#")) and " " in line and "/" in line:
            # plain FEN file
            fens.append(line)
    return fens


def run_typhoon_eval_dump(fens: list[str]) -> str:
    SCRATCH.mkdir(parents=True, exist_ok=True)
    script_path = SCRATCH / "typhoon_script.txt"
    out_path = SCRATCH / "typhoon_out.txt"
    with open(script_path, "w") as f:
        for fen in fens:
            # `eval` alone only prints the "Static eval:" total; the
            # per-term "--- WHITE ... BLACK" breakdown table comes from
            # the separate `dump eval` command (EvalTraceReport(),
            # testeval.c), which reports on the terms from the *last*
            # Eval() call -- so it must follow `eval` for the same
            # position, not stand alone.
            f.write(f"setboard {fen}\neval\ndump eval\n")
    cmd = [str(TYPHOON_BIN), "--cpus", "1", "--hash", "none",
           "--logfile", "-", "--batch", "--command",
           f"force; book name /nonexistent.book.bin; script {script_path}"]
    with open(out_path, "w") as out:
        subprocess.run(cmd, stdout=out, stderr=subprocess.STDOUT, cwd=SRC_DIR,
                        timeout=max(60, len(fens) // 20))
    return out_path.read_text()


def run_crowd_engine(name: str, fens: list[str]) -> str:
    spec = ENGINE_SPECS[name]
    if spec["kind"] == "gnuchess":
        script = _build_gnuchess_script(fens)
    else:
        script = spec["startup"]
        for fen in fens:
            script += spec["pos_fmt"].format(fen=fen) + spec["eval_cmd"] + "\n"
        script += "quit\n"
    p = subprocess.run([spec["binary"]], input=script, capture_output=True,
                        text=True, cwd=spec["cwd"], timeout=max(60, len(fens) // 20))
    return p.stdout


def _build_gnuchess_script(fens: list[str]) -> str:
    """gnuchess (gnuchessr) has no `setboard`/`position fen` -- the only
    way to load an arbitrary position is `xget`, which prompts for a
    filename and reads a file in its own old xboard.position.read
    format: just FEN's first two fields (piece placement + active
    color), one per file, no castling/ep/clock info. Also: a leading
    blank line is required first to accept the default time control
    prompt before the "Your move is?" command loop (where `xget` is a
    valid command) even starts accepting input."""
    gc_dir = SCRATCH / "gnuchess"
    gc_dir.mkdir(parents=True, exist_ok=True)
    script = "\n"
    for i, fen in enumerate(fens):
        parts = fen.split()
        pos_path = gc_dir / f"pos_{i}.xpos"
        pos_path.write_text(f"{parts[0]} {parts[1]}\n")
        script += f"xget\n{pos_path}\nscore\n"
    script += "quit\n"
    return script


def parse_typhoon(text: str, n: int):
    evals = [float(x) for x in
             re.findall(r'^Static eval:\s*([+-]?\d+\.\d+)', text, re.M)]
    assert len(evals) == n, f"expected {n} typhoon evals, got {len(evals)}"
    return evals


def parse_crowd_engine(name: str, text: str, stm: list[str], n: int):
    """Returns n scores in pawns, from WHITE's perspective."""
    spec = ENGINE_SPECS[name]
    raw = [float(x) for x in re.findall(spec["parse_re"], text, re.M)]
    assert len(raw) == n, (
        f"expected {n} {name} evals, got {len(raw)} -- protocol/parse_re "
        f"mismatch, or {name}'s binary crashed/changed output format")
    white = []
    for val, side in zip(raw, stm):
        v = val * spec["scale"]
        if spec["stm_relative"] and side == 'b':
            v = -v
        white.append(v)
    return white


LABEL_RE_CACHE = {}


def term_labels_from_eval_c() -> list[str]:
    """Pull every EVAL_TERM(...) call's label string directly out of
    eval.c, so this stays in sync with the source without needing a
    separately-maintained list."""
    src = (SRC_DIR / "eval.c").read_text()
    labels = set()
    for m in re.finditer(r'EVAL_TERM\(', src):
        start = m.end()
        depth = 1
        i = start
        while depth > 0 and i < len(src):
            if src[i] == '(':
                depth += 1
            elif src[i] == ')':
                depth -= 1
            i += 1
        # Allow escaped quotes/backslashes inside the string literal
        # (e.g. eval.c's own "in/out of \"action\"" label) -- a naive
        # [^"]* stops at the first escaped quote and can pick up a
        # spurious empty trailing match instead of the real label.
        strs = re.findall(r'"((?:[^"\\]|\\.)*)"', src[start:i - 1])
        if strs:
            labels.add(strs[-1].replace('\\"', '"'))
    return sorted(labels, key=len, reverse=True)


def get_label_re():
    if 're' not in LABEL_RE_CACHE:
        labels = term_labels_from_eval_c()
        LABEL_RE_CACHE['re'] = re.compile(
            '(' + '|'.join(re.escape(l) for l in labels) + r')\s*(-|[a-h][1-8])?\s+(-?\d+)')
    return LABEL_RE_CACHE['re']


def parse_terms_for_block(block: str):
    """block is the text between '--- WHITE' and 'Static eval:' for one
    position. Returns {label: contribution_to_WHITE} (white/black columns
    split at column 40, matching the dump's fixed layout). Always
    White-relative regardless of side to move -- each column is that
    color's own raw contribution, so White's column adds and Black's
    subtracts from the White-relative total, independent of stm. (Do not
    key the sign off stm: that gives the contribution to the side TO
    MOVE instead, which flips sign on every Black-to-move position and
    silently destroys any correlation against a White-relative residual.)
    """
    label_re = get_label_re()
    contrib = {}
    for line in block.splitlines():
        for half, color in ((line[:40], 'w'), (line[40:], 'b')):
            for m in label_re.finditer(half):
                label, val = m.group(1), int(m.group(3))
                sign = 1 if color == 'w' else -1
                contrib[label] = contrib.get(label, 0) + sign * val
    return contrib


def split_typhoon_blocks(text: str) -> list[str]:
    # Each block runs from its own "--- WHITE" header up to (but not
    # including) the NEXT position's leading board-dump text -- trim at
    # this block's own "Static eval:" line, or the raw split would
    # include the following position's FEN/hash/board-art/"Material:"
    # preamble as trailing garbage, which can spuriously match the term
    # regex (large hex-signature/material digit runs sitting near label
    # text by coincidence).
    parts = re.split(r'(?=^--- WHITE)', text, flags=re.M)
    blocks = []
    for p in parts:
        if not p.startswith('--- WHITE'):
            continue
        m = re.search(r'^Static eval:.*$', p, flags=re.M)
        blocks.append(p[:m.end()] if m else p)
    return blocks


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("fen_source")
    ap.add_argument("--limit", type=int, default=2000,
                     help="max positions to sample (default 2000)")
    ap.add_argument("--top", type=int, default=20,
                     help="how many biggest disagreements to print")
    ap.add_argument("--engines", default=",".join(ENGINE_SPECS),
                     help=f"comma-separated crowd engines to use, from "
                          f"{{{','.join(ENGINE_SPECS)}}} (default: all)")
    ap.add_argument("--sign-deadband", type=float, default=0.5,
                     help="pawns; both |typhoon| and |crowd median| (or "
                          "unanimous crowd value) must clear this to count "
                          "as a real sign disagreement, not ~0 noise "
                          "(default 0.5)")
    ap.add_argument("--spread-floor", type=float, default=0.25,
                     help="pawns; floor on crowd stdev used when ranking "
                          "gap/spread so a near-unanimous crowd doesn't "
                          "produce a division-by-near-zero blowup "
                          "(default 0.25)")
    args = ap.parse_args()

    engines = [e.strip() for e in args.engines.split(",") if e.strip()]
    for e in engines:
        if e not in ENGINE_SPECS:
            sys.exit(f"unknown engine {e!r} -- choose from {list(ENGINE_SPECS)}")
        if not Path(ENGINE_SPECS[e]["binary"]).is_absolute() and \
                ENGINE_SPECS[e]["cwd"] is None:
            continue  # resolved via PATH (e.g. crafty)
    if not TYPHOON_BIN.exists():
        sys.exit(f"missing {TYPHOON_BIN} -- build with EVAL_DUMP=1 first "
                  "(see this file's docstring)")

    fens = extract_fens(Path(args.fen_source))[:args.limit]
    stm = [f.split()[1] for f in fens]
    print(f"{len(fens)} positions, crowd = {engines}", file=sys.stderr)

    ty_text = run_typhoon_eval_dump(fens)
    ty_evals = parse_typhoon(ty_text, len(fens))
    ty_white = [e if s == 'w' else -e for e, s in zip(ty_evals, stm)]

    crowd_white = {}
    for name in engines:
        text = run_crowd_engine(name, fens)
        crowd_white[name] = parse_crowd_engine(name, text, stm, len(fens))

    blocks = split_typhoon_blocks(ty_text)
    assert len(blocks) == len(fens), (
        f"{len(blocks)} eval-dump blocks vs {len(fens)} positions -- "
        "did the typhoon binary actually have EVAL_DUMP=1?")
    contribs = [parse_terms_for_block(b) for b in blocks]

    n = len(fens)
    per_pos = []  # (median, stdev, residual, is_extreme, extreme_dir)
    for i in range(n):
        crowd_vals = [crowd_white[name][i] for name in engines]
        median = statistics.median(crowd_vals)
        stdev = statistics.pstdev(crowd_vals) if len(crowd_vals) > 1 else 0.0
        residual = ty_white[i] - median
        all_vals = crowd_vals + [ty_white[i]]
        is_max = ty_white[i] == max(all_vals) and all_vals.count(ty_white[i]) == 1
        is_min = ty_white[i] == min(all_vals) and all_vals.count(ty_white[i]) == 1
        per_pos.append((median, stdev, residual, is_max, is_min))

    def print_position(i, extra=""):
        median = per_pos[i][0]
        crowd_str = "  ".join(f"{name}={crowd_white[name][i]:+6.2f}"
                               for name in engines)
        print(f"{extra}ty(w)={ty_white[i]:+6.2f}  median(w)={median:+6.2f}  "
              f"{crowd_str}  {fens[i]}")
        top_terms = sorted(contribs[i].items(), key=lambda kv: -abs(kv[1]))[:4]
        for label, val in top_terms:
            print(f"      {label:38s} {val:+5d}")

    # --- 1. Raw magnitude disagreements (original behavior) -------------
    diffs = sorted(range(n), key=lambda i: -abs(per_pos[i][2]))
    print(f"\n=== Top {args.top} disagreements "
          f"(|typhoon(white) - crowd_median(white)|) ===\n")
    for i in diffs[:args.top]:
        print_position(i, extra=f"diff={per_pos[i][2]:+7.2f}  ")

    # --- 2. Gap normalized by crowd spread -------------------------------
    # A raw pawn gap conflates "typhoon is wrong" with "this position is
    # just wild and the whole crowd disagrees with itself." Dividing by
    # the crowd's own stdev (floored, so a near-unanimous crowd doesn't
    # blow up the ratio) surfaces cases where typhoon breaks from an
    # otherwise-tight consensus, even by a modest raw amount.
    spread_ranked = sorted(
        range(n), key=lambda i: -abs(per_pos[i][2]) / max(per_pos[i][1], args.spread_floor))
    print(f"\n=== Top {args.top} disagreements normalized by crowd spread "
          f"(|residual| / max(crowd_stdev, {args.spread_floor})) ===\n")
    for i in spread_ranked[:args.top]:
        ratio = abs(per_pos[i][2]) / max(per_pos[i][1], args.spread_floor)
        print_position(i, extra=f"ratio={ratio:6.2f}  stdev={per_pos[i][1]:5.2f}  ")

    # --- 3. Typhoon as the lone extreme value ----------------------------
    extremes = [i for i in range(n) if per_pos[i][3] or per_pos[i][4]]
    extremes.sort(key=lambda i: -abs(per_pos[i][2]))
    print(f"\n=== Positions where typhoon is the strict min or max of "
          f"typhoon+crowd ({len(extremes)} of {n}) ===\n")
    for i in extremes[:args.top]:
        direction = "HIGHEST" if per_pos[i][3] else "LOWEST"
        print_position(i, extra=f"typhoon is {direction}  ")

    # --- 4. Sign disagreements -------------------------------------------
    db = args.sign_deadband
    vs_median, vs_unanimous = [], []
    for i in range(n):
        ty, median = ty_white[i], per_pos[i][0]
        if abs(ty) >= db and abs(median) >= db and (ty > 0) != (median > 0):
            vs_median.append(i)
        crowd_vals = [crowd_white[name][i] for name in engines]
        if all(abs(v) >= db for v in crowd_vals) and \
                len({v > 0 for v in crowd_vals}) == 1:
            crowd_sign_positive = crowd_vals[0] > 0
            if abs(ty) >= db and (ty > 0) != crowd_sign_positive:
                vs_unanimous.append(i)
    print(f"\n=== Typhoon disagrees with crowd MEDIAN's sign "
          f"(both sides >= {db} pawns) -- {len(vs_median)} positions ===\n")
    for i in vs_median[:args.top]:
        print_position(i)
    print(f"\n=== Typhoon disagrees with a UNANIMOUS crowd's sign "
          f"(every engine agrees, typhoon alone doesn't; >= {db} pawns) "
          f"-- {len(vs_unanimous)} positions ===\n")
    for i in vs_unanimous[:args.top]:
        print_position(i)
    if "gnuchess" in engines and len(engines) > 1:
        others = [e for e in engines if e != "gnuchess"]
        vs_unanimous_no_gc = []
        for i in range(n):
            vals = [crowd_white[name][i] for name in others]
            if all(abs(v) >= db for v in vals) and len({v > 0 for v in vals}) == 1:
                if abs(ty_white[i]) >= db and (ty_white[i] > 0) != (vals[0] > 0):
                    vs_unanimous_no_gc.append(i)
        print(f"\n    (gnuchess drops castling rights and ignores stm; "
              f"same check EXCLUDING gnuchess: "
              f"{len(vs_unanimous_no_gc)} positions)")

    # --- 5. Per-term correlation with the White-relative residual --------
    # For each EVAL_TERM label, correlate its own signed White-relative
    # contribution against ty_white - crowd_median across the whole
    # sample. A term whose contribution consistently tracks the residual
    # is a candidate for being over- or under-weighted relative to the
    # crowd's implicit consensus; a term with no correlation isn't
    # driving the disagreement even if it's large in any one position.
    residuals = [p[2] for p in per_pos]
    all_labels = set()
    for c in contribs:
        all_labels.update(c.keys())
    term_stats = []
    for label in all_labels:
        vals = [contribs[i].get(label, 0) for i in range(n)]
        if len(set(vals)) < 2:
            continue  # constant (e.g. always zero) -- correlation undefined
        try:
            corr = statistics.correlation(vals, residuals)
        except statistics.StatisticsError:
            continue
        mean_abs = sum(abs(v) for v in vals) / n
        term_stats.append((label, corr, mean_abs, corr * mean_abs))
    term_stats.sort(key=lambda t: -abs(t[3]))
    print(f"\n=== EVAL_TERM correlation with (typhoon - crowd_median), "
          f"whole sample (n={n}) ===")
    print("(positive corr = term grows typhoon's score in the same "
          "direction as its disagreement with the crowd -- a candidate "
          "for being over-weighted; negative = under-weighted)\n")
    for label, corr, mean_abs, score in term_stats[:20]:
        print(f"      {label:38s} corr={corr:+.3f}  mean|contrib|={mean_abs:6.2f}  "
              f"score={score:+.3f}")

    # --- 6. Global directional bias ---------------------------------------
    mean_r = statistics.mean(residuals)
    median_r = statistics.median(residuals)
    w_res = [residuals[i] for i in range(n) if stm[i] == 'w']
    b_res = [residuals[i] for i in range(n) if stm[i] == 'b']
    print(f"\n=== Global bias: typhoon(white) - crowd_median(white) ===\n")
    print(f"      overall:          mean={mean_r:+.3f}  median={median_r:+.3f}  n={n}")
    if w_res:
        print(f"      white to move:    mean={statistics.mean(w_res):+.3f}  "
              f"median={statistics.median(w_res):+.3f}  n={len(w_res)}")
    if b_res:
        print(f"      black to move:    mean={statistics.mean(b_res):+.3f}  "
              f"median={statistics.median(b_res):+.3f}  n={len(b_res)}")
    print("      (a nonzero overall mean/median with the same sign on "
          "both stm splits points at a constant-ish term -- tempo, draw "
          "score, a global scale factor -- rather than a positional one)")

    # --- 7. Residual by material phase ------------------------------------
    # Cheap, self-contained phase proxy from the FEN alone (not typhoon's
    # own internal phase bucketing) so this works even without EVAL_DUMP
    # internals: sum of N=3/B=3/R=5/Q=9 for both sides, pawns/kings
    # excluded. Eval bugs are usually phase-local, and a flat top-K list
    # hides that.
    piece_val = {'n': 3, 'b': 3, 'r': 5, 'q': 9}
    phases = []
    for fen in fens:
        board = fen.split()[0]
        phases.append(sum(piece_val.get(ch.lower(), 0) for ch in board if ch.isalpha()))
    buckets = [(0, 12, "endgame (<=12)"), (13, 26, "middlegame (13-26)"),
               (27, 999, "opening/near-full (>=27)")]
    print(f"\n=== Residual by material phase (non-pawn point count, both sides) ===\n")
    for lo, hi, name in buckets:
        idxs = [i for i in range(n) if lo <= phases[i] <= hi]
        if not idxs:
            continue
        r = [residuals[i] for i in idxs]
        print(f"      {name:28s} mean={statistics.mean(r):+.3f}  "
              f"median={statistics.median(r):+.3f}  n={len(idxs)}")


if __name__ == "__main__":
    main()