# Profiling ./typhoon Two environments, two different root situations. Short version: **on Linux, use `gmake GPROF=1` + gprof(1) -- fully root-free.** On FreeBSD, `pmcstat`'s hardware-counter sampling is also root-free *in practice* once `hwpmc.ko` is loaded, which is the one step that does need root (and only once per boot). ## Linux (this box): gprof, no root required at all `perf` is blocked here -- `kernel.perf_event_paranoid` is `4` (stricter than the usual 0-3 range; Ubuntu-specific), so `perf record`/`perf stat` refuse to run for a non-root user with no CAP_PERFMON, and there's no non-root workaround short of asking someone to lower that sysctl or grant the capability. `valgrind`/`callgrind` and `uftrace` aren't installed and installing them needs `apt` (root). `gprof` (part of binutils) is already installed and needs no privilege of any kind -- it works by compiling the binary with `-pg`, which makes every function call `mcount()` to accumulate call-graph edge counts, plus a `SIGPROF`-interval sampler for self-time. Nothing about that touches hardware counters or kernel permissions. A `GPROF=1` gmake flag has been added to `GNUmakefile` for this (folds `-pg` into `$(PROFILE)`, so it lands in both the compile and link steps automatically): ```sh gmake clean # -pg changes codegen for every TU; must rebuild from scratch gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 GPROF=1 ``` Run the resulting `./typhoon` from whatever directory you want `gmon.out` written into (it's always named exactly that, in cwd, overwritten on each run) -- **use `--cpus 1`**, not the usual `MP=1` multi-threaded search, for this run specifically. See the threading caveat below for why. ```sh mkdir -p /tmp/typhoon/profile && cd /tmp/typhoon/profile /home/scott/typhoon/src/typhoon --cpus 1 --hash 256m --batch --command \ "force; sd 12; script ../tests/ecm_confident_quick.ep_" ``` Then read it with gprof, pointing at the *same* binary that produced it (symbol addresses are binary-specific): ```sh gprof -b /home/scott/typhoon/src/typhoon gmon.out > report.txt less report.txt ``` `report.txt` has two sections: a **flat profile** (self time, sorted descending -- "what's hot") first, then a **call graph** (who calls whom, cumulative time) after. The flat profile is usually the right place to start. Confirmed working end-to-end on this box: a `sd 14` search from the startpos produced a flat profile correctly topped by `Eval` (16.1% self time, 74M calls), then `Search`, `SelectBestWithHistory`, `_GenerateAllMovesBB`, `GenerateMoves`, etc. -- a sane-looking distribution. **Don't forget to rebuild the normal release binary afterward** (`gmake clean` + the standard `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` profile, no `GPROF=1`), same reasoning as `precommit_check.sh`: a `-pg`-instrumented `typhoon` sitting in the working directory as a side effect of profiling is easy to mistake for the real release build later. ### Threading caveat: use `--cpus 1` for gprof runs glibc's gprof support predates pervasive multithreading and only reliably attributes `SIGPROF`-based self-time samples to the thread that set up profiling (typically the one running `main()`) -- worker threads spawned under `MP=1`'s multithreaded search can have their CPU time undercounted or misattributed in the flat profile. Call-graph edge counts (`mcount()` increments into global memory, not timer-based) aren't subject to this, but to keep the whole report trustworthy, profile with `--cpus 1` and treat multi-threaded gprof numbers as unreliable if you ever need to look at search scaling specifically (use the FreeBSD `pmcstat` path below for that, or `perf` if the paranoid sysctl ever gets relaxed on this box). ### Other caveat: inlining `-pg` only instruments actual (non-inlined) function calls -- a function the optimizer inlined into its caller never gets its own `mcount()` call and shows up folded into the caller's self time, not as its own line. The flat profile above is still real, just coarser for small/hot helper functions than it would be with a sampling-based (hardware PMC) profiler. ## FreeBSD (`wannabe.house`, this engine's native/original platform): pmcstat + gprof Verified against the live box 2026-10-05 (session had passwordless ssh to `scott@wannabe.house`, `~/typhoon` there, a separate enlistment from this one). **The commonly-assumed "needs root" turned out to be only partially true**: `hwpmc.ko` must be loaded (`kldload hwpmc`, root-only, but typically already loaded by someone else on a shared box -- check with `kldstat | grep hwpmc` before assuming you need to ask for it) but once it's loaded, **process-scoped sampling with `pmcstat -P` and the whole gprof-conversion pipeline below ran successfully as a plain, non-root user in this session** -- no `sudo` anywhere. The earlier assumption of needing root for every run likely came from either testing with `-S` (system-wide mode, which genuinely does require `PRIV_PMC_SYSTEM` / root) or from the `hwpmc.ko` load itself. This box is AuthenticAMD (`kern.hwpmc.cpuid: AuthenticAMD-23-71-0`), so Intel-style event names like `CPU_CLK_UNHALTED.THREAD` don't exist. List available raw events with `pmcstat -L`; the AMD equivalent of "unhalted cycles" is `ls_not_halted_cyc`. The portable alias `instructions` also works; the portable alias `cycles` does **not** work for sampling (`-P`/ `-S`) because it's backed by the TSC, which can't deliver the overflow interrupt sampling needs -- use `ls_not_halted_cyc` (or another real hardware event) instead, not the `cycles` alias. ### 0. Confirm hwpmc is loaded (one-time, needs root if not already) ```sh kldstat | grep hwpmc # if this prints a line, skip straight to step 1 kldload hwpmc # root-only, one-time (until next reboot) ``` ### 1. Collect samples (process-scoped, not system-wide) Use capital `-P` (process-mode sampling, follows only the spawned child) -- NOT lowercase `-p` (counting mode only, incompatible with `-O`) and NOT `-S` (system-wide: captures every process on the box, requires root, and makes symbol resolution in step 2 slow/noisy). Append `,` to the event name to set the sampling interval in cycles -- the default reload count samples far too often and produces multi-GB output files that take forever to convert. `4000000` (~every 4M cycles) is a reasonable starting point; increase for a coarser/faster profile, decrease for finer resolution on a short run. ```sh mkdir -p /tmp/typhoon/profile pmcstat -P ls_not_halted_cyc,4000000 -O /tmp/typhoon/profile/pmc.out -- \ ./typhoon --cpus 1 --hash 256m --batch --command \ "force; sd 12; script ../tests/ecm_confident_quick.ep_" ``` No `sudo` needed for this step as long as step 0 is satisfied. ### 2. Convert to a gmon.out-style file, per executable `pmcstat -R -g` writes a `*.gmon` file per executable/library seen in the samples, auto-named, into a subdirectory named after the event (e.g. `ls_not_halted_cyc/`) under the current directory. (`-O` cannot be combined with `-R` -- the input file is the positional arg to `-R` itself.) A `kernel.gmon overflowed` warning is expected/harmless in process-mode -- ignore it, we only want the typhoon binary's own file. ```sh pmcstat -R /tmp/typhoon/profile/pmc.out -g # -> creates ./ls_not_halted_cyc/typhoon.gmon (plus one per shared lib) ``` ### 3. View the report with gprof FreeBSD's base gprof segfaults on the raw dump produced by `-G ` (a different, two-step raw-dump path) -- skip that entirely. It does **not** segfault on the proper gmon.out-format file produced by `-g` above: ```sh gprof ./typhoon ls_not_halted_cyc/typhoon.gmon > /tmp/typhoon/profile/report.txt less /tmp/typhoon/profile/report.txt ``` Same two-section report as the Linux path: call graph, then flat profile (self time, sorted descending) -- the flat profile is usually the more useful starting point. ### Gotchas recap - `-S` = system-wide sampling, requires root, resolves symbols for unrelated processes too -- avoid. `-P` = process-scoped, no root needed once `hwpmc.ko` is loaded -- what we want. `-p` (lowercase) = counting mode, not sampling, incompatible with `-O`. - Always set an explicit `,` sampling interval or the output file balloons to multi-GB and conversion takes forever. Confirmed: an ~11-second `sd 14` single-position search at the default interval produced a 300MB+ sample file. - `-O` and `-R` are mutually exclusive; `-R` takes the input file directly as its argument. - `-G` writes a raw dump gprof segfaults on; `-g` writes a proper gmon.out-format file gprof handles fine. Use `-g`. - The workload needs to run long enough for `-P` sampling to actually catch anything -- a single quick `sd 9` search (~1s) produced a fixed ~3.8KB output file (just the exec/mapping record, zero PC samples) regardless of the sampling interval used. A `script .ep_` run over a whole suite (tens of seconds to minutes, e.g. `ecm_ringers.ep_` at `sd 10`, ~25s) or a single deep `sd 14`+ search (confirmed: ~11s) is long enough to get real samples. - **Not every event supports `-P` sampling** (some AMD PMC events are counting-only). Branch-misprediction events (`ex_ret_brn_misp`, `ex_ret_cond_misp`, etc.) work fine in counting mode (`pmcstat -p ex_ret_brn_misp -- `, prints a single total at exit) but produce the same fixed ~3.8KB/zero-samples output as the "too-short workload" case even against a long run -- a real per-event limitation, not a workload-duration problem (confirmed by running the known-good `ls_not_halted_cyc` event against an identical command/interval/ duration, which produced immediate real samples). Likely cause: not every AMD PMC event supports overflow-interrupt (PMI) delivery, which `-P`/`-S` sampling requires -- some are counting-only, and neither `pmcstat -L` nor `pmccontrol -L` distinguish which is which. The only way to find out is to try `-P` against a known-long-enough workload and check the output file size. If a per-function mispredict breakdown is ever actually needed, look into whether a PEBS-like interrupt-capable event exists for this, before assuming it's simply not possible. ## Which to reach for - **Linux, day-to-day**: `gmake GPROF=1` + gprof. No privilege dependency, no sysadmin step, works today. - **FreeBSD (`wannabe.house`)**: `pmcstat -P` + gprof, after confirming `hwpmc.ko` is loaded. Hardware-counter-based (actual cycles/ instructions, not call-count-overhead-inflated like `-pg`), so more trustworthy for small/hot inlined-adjacent functions, and not subject to the multithreading self-time caveat gprof's `SIGPROF` approach has on Linux -- but still process-scoped, so still only reflects the one profiled process, not the whole system. - **If Linux `perf` access ever gets relaxed** (someone lowers `kernel.perf_event_paranoid` or grants `CAP_PERFMON`), it would be the best of both: hardware counters *and* correct multithreaded attribution, without `-pg`'s inlining blind spot. Worth revisiting if a future profiling question specifically needs multi-threaded search behavior under `MP=1`.