Phase A NNUE: feature-indexing spec, annotation/training pipeline, dump nnuefeatures

Builds out the NNUE Phase A prototype per docs/NNUE.md:

- src/nnuefeatures.c: standalone HalfKAv2_hm feature-index reference
  implementation (new `dump nnuefeatures` command), sourced from
  Stockfish's half_ka_v2_hm.{h,cpp} (sf_16 tag), including the 0x88->64
  square conversion typhoon's own board representation needs.
- annotate_pgns/: sharded TWIC/self-play annotation harness
  (annotate.py), position extraction, quiet-position filtering (flat
  and game-shaped, plus an upstream --prefilter-quiet option),
  train/holdout game-level stratified split, WDL-sigmoid K fitting,
  the Python HalfKAv2_hm reference encoder (nnue_feature_spec.py) and
  its bit-for-bit cross-checker against the C side
  (cross_check_features.py), and a first PyTorch training pipeline
  (nnue_model.py, nnue_train.py, build_training_tensors.py) -- a
  256x2-32-32-1-ish net trained locally on CPU against ~1.85M quiet
  TWIC positions as a pipeline/framework validation pass.
- Finish an in-progress InitMagic/InitLMRTable/InitEval ->
  InitializeBitboardMagic/InitializeLMRTable/InitializeEval rename:
  chess.h's prototypes hadn't been updated to match the already-renamed
  definitions/call sites, leaving 4 implicit-function-declaration
  warnings.
- tests/arasan.epd: new curated EPD suite.
- docs/: add the project's design docs (previously untracked).

Co-Authored-By: Claude Sonnet 5 <[email protected]>

1 files changed, 222 insertions, 0 deletions

diff --git a/src/docs/PROFILE.md b/src/docs/PROFILE.md
new file mode 100644
index 0000000..67106e0
--- /dev/null
+++ b/src/docs/PROFILE.md
@@ -0,0 +1,222 @@
+# Profiling ./typhoon
+
+Two environments, two different root situations. Short version: **on
+Linux, use `gmake GPROF=1` + gprof(1) -- fully root-free.** On FreeBSD,
+`pmcstat`'s hardware-counter sampling is also root-free *in practice* once
+`hwpmc.ko` is loaded, which is the one step that does need root (and only
+once per boot).
+
+## Linux (this box): gprof, no root required at all
+
+`perf` is blocked here -- `kernel.perf_event_paranoid` is `4` (stricter
+than the usual 0-3 range; Ubuntu-specific), so `perf record`/`perf stat`
+refuse to run for a non-root user with no CAP_PERFMON, and there's no
+non-root workaround short of asking someone to lower that sysctl or grant
+the capability. `valgrind`/`callgrind` and `uftrace` aren't installed and
+installing them needs `apt` (root). `gprof` (part of binutils) is already
+installed and needs no privilege of any kind -- it works by compiling the
+binary with `-pg`, which makes every function call `mcount()` to accumulate
+call-graph edge counts, plus a `SIGPROF`-interval sampler for self-time.
+Nothing about that touches hardware counters or kernel permissions.
+
+A `GPROF=1` gmake flag has been added to `GNUmakefile` for this (folds
+`-pg` into `$(PROFILE)`, so it lands in both the compile and link steps
+automatically):
+
+```sh
+gmake clean   # -pg changes codegen for every TU; must rebuild from scratch
+gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 GPROF=1
+```
+
+Run the resulting `./typhoon` from whatever directory you want `gmon.out`
+written into (it's always named exactly that, in cwd, overwritten on each
+run) -- **use `--cpus 1`**, not the usual `MP=1` multi-threaded search, for
+this run specifically. See the threading caveat below for why.
+
+```sh
+mkdir -p /tmp/typhoon/profile && cd /tmp/typhoon/profile
+/home/scott/typhoon/src/typhoon --cpus 1 --hash 256m --batch --command \
+  "force; sd 12; script ../tests/ecm_confident_quick.ep_"
+```
+
+Then read it with gprof, pointing at the *same* binary that produced it
+(symbol addresses are binary-specific):
+
+```sh
+gprof -b /home/scott/typhoon/src/typhoon gmon.out > report.txt
+less report.txt
+```
+
+`report.txt` has two sections: a **flat profile** (self time, sorted
+descending -- "what's hot") first, then a **call graph** (who calls whom,
+cumulative time) after. The flat profile is usually the right place to
+start. Confirmed working end-to-end on this box: a `sd 14` search from the
+startpos produced a flat profile correctly topped by `Eval` (16.1% self
+time, 74M calls), then `Search`, `SelectBestWithHistory`,
+`_GenerateAllMovesBB`, `GenerateMoves`, etc. -- a sane-looking distribution.
+
+**Don't forget to rebuild the normal release binary afterward** (`gmake
+clean` + the standard `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` profile,
+no `GPROF=1`), same reasoning as `precommit_check.sh`: a `-pg`-instrumented
+`typhoon` sitting in the working directory as a side effect of profiling is
+easy to mistake for the real release build later.
+
+### Threading caveat: use `--cpus 1` for gprof runs
+
+glibc's gprof support predates pervasive multithreading and only reliably
+attributes `SIGPROF`-based self-time samples to the thread that set up
+profiling (typically the one running `main()`) -- worker threads spawned
+under `MP=1`'s multithreaded search can have their CPU time undercounted
+or misattributed in the flat profile. Call-graph edge counts (`mcount()`
+increments into global memory, not timer-based) aren't subject to this,
+but to keep the whole report trustworthy, profile with `--cpus 1` and
+treat multi-threaded gprof numbers as unreliable if you ever need to look
+at search scaling specifically (use the FreeBSD `pmcstat` path below for
+that, or `perf` if the paranoid sysctl ever gets relaxed on this box).
+
+### Other caveat: inlining
+
+`-pg` only instruments actual (non-inlined) function calls -- a function
+the optimizer inlined into its caller never gets its own `mcount()` call
+and shows up folded into the caller's self time, not as its own line. The
+flat profile above is still real, just coarser for small/hot helper
+functions than it would be with a sampling-based (hardware PMC) profiler.
+
+## FreeBSD (`wannabe.house`, this engine's native/original platform): pmcstat + gprof
+
+Verified against the live box 2026-10-05 (session had passwordless ssh to
+`[email protected]`, `~/typhoon` there, a separate enlistment from this
+one). **The commonly-assumed "needs root" turned out to be only partially
+true**: `hwpmc.ko` must be loaded (`kldload hwpmc`, root-only, but
+typically already loaded by someone else on a shared box -- check with
+`kldstat | grep hwpmc` before assuming you need to ask for it) but once
+it's loaded, **process-scoped sampling with `pmcstat -P` and the whole
+gprof-conversion pipeline below ran successfully as a plain, non-root user
+in this session** -- no `sudo` anywhere. The earlier assumption of
+needing root for every run likely came from either testing with `-S`
+(system-wide mode, which genuinely does require `PRIV_PMC_SYSTEM` /
+root) or from the `hwpmc.ko` load itself.
+
+This box is AuthenticAMD (`kern.hwpmc.cpuid: AuthenticAMD-23-71-0`), so
+Intel-style event names like `CPU_CLK_UNHALTED.THREAD` don't exist. List
+available raw events with `pmcstat -L`; the AMD equivalent of "unhalted
+cycles" is `ls_not_halted_cyc`. The portable alias `instructions` also
+works; the portable alias `cycles` does **not** work for sampling (`-P`/
+`-S`) because it's backed by the TSC, which can't deliver the
+overflow interrupt sampling needs -- use `ls_not_halted_cyc` (or another
+real hardware event) instead, not the `cycles` alias.
+
+### 0. Confirm hwpmc is loaded (one-time, needs root if not already)
+
+```sh
+kldstat | grep hwpmc            # if this prints a line, skip straight to step 1
+kldload hwpmc                   # root-only, one-time (until next reboot)
+```
+
+### 1. Collect samples (process-scoped, not system-wide)
+
+Use capital `-P` (process-mode sampling, follows only the spawned child)
+-- NOT lowercase `-p` (counting mode only, incompatible with `-O`) and NOT
+`-S` (system-wide: captures every process on the box, requires root, and
+makes symbol resolution in step 2 slow/noisy).
+
+Append `,<count>` to the event name to set the sampling interval in
+cycles -- the default reload count samples far too often and produces
+multi-GB output files that take forever to convert. `4000000` (~every 4M
+cycles) is a reasonable starting point; increase for a coarser/faster
+profile, decrease for finer resolution on a short run.
+
+```sh
+mkdir -p /tmp/typhoon/profile
+pmcstat -P ls_not_halted_cyc,4000000 -O /tmp/typhoon/profile/pmc.out -- \
+  ./typhoon --cpus 1 --hash 256m --batch --command \
+  "force; sd 12; script ../tests/ecm_confident_quick.ep_"
+```
+
+No `sudo` needed for this step as long as step 0 is satisfied.
+
+### 2. Convert to a gmon.out-style file, per executable
+
+`pmcstat -R <file> -g` writes a `*.gmon` file per executable/library seen
+in the samples, auto-named, into a subdirectory named after the event
+(e.g. `ls_not_halted_cyc/`) under the current directory. (`-O` cannot be
+combined with `-R` -- the input file is the positional arg to `-R`
+itself.) A `kernel.gmon overflowed` warning is expected/harmless in
+process-mode -- ignore it, we only want the typhoon binary's own file.
+
+```sh
+pmcstat -R /tmp/typhoon/profile/pmc.out -g
+# -> creates ./ls_not_halted_cyc/typhoon.gmon (plus one per shared lib)
+```
+
+### 3. View the report with gprof
+
+FreeBSD's base gprof segfaults on the raw dump produced by `-G <file>`
+(a different, two-step raw-dump path) -- skip that entirely. It does
+**not** segfault on the proper gmon.out-format file produced by `-g`
+above:
+
+```sh
+gprof ./typhoon ls_not_halted_cyc/typhoon.gmon > /tmp/typhoon/profile/report.txt
+less /tmp/typhoon/profile/report.txt
+```
+
+Same two-section report as the Linux path: call graph, then flat profile
+(self time, sorted descending) -- the flat profile is usually the more
+useful starting point.
+
+### Gotchas recap
+
+- `-S` = system-wide sampling, requires root, resolves symbols for
+  unrelated processes too -- avoid. `-P` = process-scoped, no root needed
+  once `hwpmc.ko` is loaded -- what we want. `-p` (lowercase) = counting
+  mode, not sampling, incompatible with `-O`.
+- Always set an explicit `,<count>` sampling interval or the output file
+  balloons to multi-GB and conversion takes forever. Confirmed: an
+  ~11-second `sd 14` single-position search at the default interval
+  produced a 300MB+ sample file.
+- `-O` and `-R` are mutually exclusive; `-R` takes the input file
+  directly as its argument.
+- `-G` writes a raw dump gprof segfaults on; `-g` writes a proper
+  gmon.out-format file gprof handles fine. Use `-g`.
+- The workload needs to run long enough for `-P` sampling to actually
+  catch anything -- a single quick `sd 9` search (~1s) produced a fixed
+  ~3.8KB output file (just the exec/mapping record, zero PC samples)
+  regardless of the sampling interval used. A `script <suite>.ep_` run
+  over a whole suite (tens of seconds to minutes, e.g. `ecm_ringers.ep_`
+  at `sd 10`, ~25s) or a single deep `sd 14`+ search (confirmed: ~11s) is
+  long enough to get real samples.
+- **Not every event supports `-P` sampling** (some AMD PMC events are
+  counting-only). Branch-misprediction events (`ex_ret_brn_misp`,
+  `ex_ret_cond_misp`, etc.) work fine in counting mode (`pmcstat -p
+  ex_ret_brn_misp -- <cmd>`, prints a single total at exit) but produce
+  the same fixed ~3.8KB/zero-samples output as the "too-short workload"
+  case even against a long run -- a real per-event limitation, not a
+  workload-duration problem (confirmed by running the known-good
+  `ls_not_halted_cyc` event against an identical command/interval/
+  duration, which produced immediate real samples). Likely cause: not
+  every AMD PMC event supports overflow-interrupt (PMI) delivery, which
+  `-P`/`-S` sampling requires -- some are counting-only, and neither
+  `pmcstat -L` nor `pmccontrol -L` distinguish which is which. The only
+  way to find out is to try `-P` against a known-long-enough workload and
+  check the output file size. If a per-function mispredict breakdown is
+  ever actually needed, look into whether a PEBS-like interrupt-capable
+  event exists for this, before assuming it's simply not possible.
+
+## Which to reach for
+
+- **Linux, day-to-day**: `gmake GPROF=1` + gprof. No privilege
+  dependency, no sysadmin step, works today.
+- **FreeBSD (`wannabe.house`)**: `pmcstat -P` + gprof, after confirming
+  `hwpmc.ko` is loaded. Hardware-counter-based (actual cycles/
+  instructions, not call-count-overhead-inflated like `-pg`), so more
+  trustworthy for small/hot inlined-adjacent functions, and not subject
+  to the multithreading self-time caveat gprof's `SIGPROF` approach has
+  on Linux -- but still process-scoped, so still only reflects the one
+  profiled process, not the whole system.
+- **If Linux `perf` access ever gets relaxed** (someone lowers
+  `kernel.perf_event_paranoid` or grants `CAP_PERFMON`), it would be the
+  best of both: hardware counters *and* correct multithreaded attribution,
+  without `-pg`'s inlining blind spot. Worth revisiting if a future
+  profiling question specifically needs multi-threaded search behavior
+  under `MP=1`.