path: root/src/docs/PROFILE.md

# Profiling ./typhoon

Two environments, two different root situations. Short version: **on
Linux, use `gmake GPROF=1` + gprof(1) -- fully root-free.** On FreeBSD,
`pmcstat`'s hardware-counter sampling is also root-free *in practice* once
`hwpmc.ko` is loaded, which is the one step that does need root (and only
once per boot).

## Linux (this box): gprof, no root required at all

`perf` is blocked here -- `kernel.perf_event_paranoid` is `4` (stricter
than the usual 0-3 range; Ubuntu-specific), so `perf record`/`perf stat`
refuse to run for a non-root user with no CAP_PERFMON, and there's no
non-root workaround short of asking someone to lower that sysctl or grant
the capability. `valgrind`/`callgrind` and `uftrace` aren't installed and
installing them needs `apt` (root). `gprof` (part of binutils) is already
installed and needs no privilege of any kind -- it works by compiling the
binary with `-pg`, which makes every function call `mcount()` to accumulate
call-graph edge counts, plus a `SIGPROF`-interval sampler for self-time.
Nothing about that touches hardware counters or kernel permissions.

A `GPROF=1` gmake flag has been added to `GNUmakefile` for this (folds
`-pg` into `$(PROFILE)`, so it lands in both the compile and link steps
automatically):

```sh
gmake clean   # -pg changes codegen for every TU; must rebuild from scratch
gmake -j5 GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1 GPROF=1
```

Run the resulting `./typhoon` from whatever directory you want `gmon.out`
written into (it's always named exactly that, in cwd, overwritten on each
run) -- **use `--cpus 1`**, not the usual `MP=1` multi-threaded search, for
this run specifically. See the threading caveat below for why.

```sh
mkdir -p /tmp/typhoon/profile && cd /tmp/typhoon/profile
/home/scott/typhoon/src/typhoon --cpus 1 --hash 256m --batch --command \
  "force; sd 12; script ../tests/ecm_confident_quick.ep_"
```

Then read it with gprof, pointing at the *same* binary that produced it
(symbol addresses are binary-specific):

```sh
gprof -b /home/scott/typhoon/src/typhoon gmon.out > report.txt
less report.txt
```

`report.txt` has two sections: a **flat profile** (self time, sorted
descending -- "what's hot") first, then a **call graph** (who calls whom,
cumulative time) after. The flat profile is usually the right place to
start. Confirmed working end-to-end on this box: a `sd 14` search from the
startpos produced a flat profile correctly topped by `Eval` (16.1% self
time, 74M calls), then `Search`, `SelectBestWithHistory`,
`_GenerateAllMovesBB`, `GenerateMoves`, etc. -- a sane-looking distribution.

**Don't forget to rebuild the normal release binary afterward** (`gmake
clean` + the standard `GENETIC=1 PERF_COUNTERS=1 MP=1 SIXTYFOUR=1` profile,
no `GPROF=1`), same reasoning as `precommit_check.sh`: a `-pg`-instrumented
`typhoon` sitting in the working directory as a side effect of profiling is
easy to mistake for the real release build later.

### Threading caveat: use `--cpus 1` for gprof runs

glibc's gprof support predates pervasive multithreading and only reliably
attributes `SIGPROF`-based self-time samples to the thread that set up
profiling (typically the one running `main()`) -- worker threads spawned
under `MP=1`'s multithreaded search can have their CPU time undercounted
or misattributed in the flat profile. Call-graph edge counts (`mcount()`
increments into global memory, not timer-based) aren't subject to this,
but to keep the whole report trustworthy, profile with `--cpus 1` and
treat multi-threaded gprof numbers as unreliable if you ever need to look
at search scaling specifically (use the FreeBSD `pmcstat` path below for
that, or `perf` if the paranoid sysctl ever gets relaxed on this box).

### Other caveat: inlining

`-pg` only instruments actual (non-inlined) function calls -- a function
the optimizer inlined into its caller never gets its own `mcount()` call
and shows up folded into the caller's self time, not as its own line. The
flat profile above is still real, just coarser for small/hot helper
functions than it would be with a sampling-based (hardware PMC) profiler.

## FreeBSD (`wannabe.house`, this engine's native/original platform): pmcstat + gprof

Verified against the live box 2026-10-05 (session had passwordless ssh to
`[email protected]`, `~/typhoon` there, a separate enlistment from this
one). **The commonly-assumed "needs root" turned out to be only partially
true**: `hwpmc.ko` must be loaded (`kldload hwpmc`, root-only, but
typically already loaded by someone else on a shared box -- check with
`kldstat | grep hwpmc` before assuming you need to ask for it) but once
it's loaded, **process-scoped sampling with `pmcstat -P` and the whole
gprof-conversion pipeline below ran successfully as a plain, non-root user
in this session** -- no `sudo` anywhere. The earlier assumption of
needing root for every run likely came from either testing with `-S`
(system-wide mode, which genuinely does require `PRIV_PMC_SYSTEM` /
root) or from the `hwpmc.ko` load itself.

This box is AuthenticAMD (`kern.hwpmc.cpuid: AuthenticAMD-23-71-0`), so
Intel-style event names like `CPU_CLK_UNHALTED.THREAD` don't exist. List
available raw events with `pmcstat -L`; the AMD equivalent of "unhalted
cycles" is `ls_not_halted_cyc`. The portable alias `instructions` also
works; the portable alias `cycles` does **not** work for sampling (`-P`/
`-S`) because it's backed by the TSC, which can't deliver the
overflow interrupt sampling needs -- use `ls_not_halted_cyc` (or another
real hardware event) instead, not the `cycles` alias.

### 0. Confirm hwpmc is loaded (one-time, needs root if not already)

```sh
kldstat | grep hwpmc            # if this prints a line, skip straight to step 1
kldload hwpmc                   # root-only, one-time (until next reboot)
```

### 1. Collect samples (process-scoped, not system-wide)

Use capital `-P` (process-mode sampling, follows only the spawned child)
-- NOT lowercase `-p` (counting mode only, incompatible with `-O`) and NOT
`-S` (system-wide: captures every process on the box, requires root, and
makes symbol resolution in step 2 slow/noisy).

Append `,<count>` to the event name to set the sampling interval in
cycles -- the default reload count samples far too often and produces
multi-GB output files that take forever to convert. `4000000` (~every 4M
cycles) is a reasonable starting point; increase for a coarser/faster
profile, decrease for finer resolution on a short run.

```sh
mkdir -p /tmp/typhoon/profile
pmcstat -P ls_not_halted_cyc,4000000 -O /tmp/typhoon/profile/pmc.out -- \
  ./typhoon --cpus 1 --hash 256m --batch --command \
  "force; sd 12; script ../tests/ecm_confident_quick.ep_"
```

No `sudo` needed for this step as long as step 0 is satisfied.

### 2. Convert to a gmon.out-style file, per executable

`pmcstat -R <file> -g` writes a `*.gmon` file per executable/library seen
in the samples, auto-named, into a subdirectory named after the event
(e.g. `ls_not_halted_cyc/`) under the current directory. (`-O` cannot be
combined with `-R` -- the input file is the positional arg to `-R`
itself.) A `kernel.gmon overflowed` warning is expected/harmless in
process-mode -- ignore it, we only want the typhoon binary's own file.

```sh
pmcstat -R /tmp/typhoon/profile/pmc.out -g
# -> creates ./ls_not_halted_cyc/typhoon.gmon (plus one per shared lib)
```

### 3. View the report with gprof

FreeBSD's base gprof segfaults on the raw dump produced by `-G <file>`
(a different, two-step raw-dump path) -- skip that entirely. It does
**not** segfault on the proper gmon.out-format file produced by `-g`
above:

```sh
gprof ./typhoon ls_not_halted_cyc/typhoon.gmon > /tmp/typhoon/profile/report.txt
less /tmp/typhoon/profile/report.txt
```

Same two-section report as the Linux path: call graph, then flat profile
(self time, sorted descending) -- the flat profile is usually the more
useful starting point.

### Gotchas recap

- `-S` = system-wide sampling, requires root, resolves symbols for
  unrelated processes too -- avoid. `-P` = process-scoped, no root needed
  once `hwpmc.ko` is loaded -- what we want. `-p` (lowercase) = counting
  mode, not sampling, incompatible with `-O`.
- Always set an explicit `,<count>` sampling interval or the output file
  balloons to multi-GB and conversion takes forever. Confirmed: an
  ~11-second `sd 14` single-position search at the default interval
  produced a 300MB+ sample file.
- `-O` and `-R` are mutually exclusive; `-R` takes the input file
  directly as its argument.
- `-G` writes a raw dump gprof segfaults on; `-g` writes a proper
  gmon.out-format file gprof handles fine. Use `-g`.
- The workload needs to run long enough for `-P` sampling to actually
  catch anything -- a single quick `sd 9` search (~1s) produced a fixed
  ~3.8KB output file (just the exec/mapping record, zero PC samples)
  regardless of the sampling interval used. A `script <suite>.ep_` run
  over a whole suite (tens of seconds to minutes, e.g. `ecm_ringers.ep_`
  at `sd 10`, ~25s) or a single deep `sd 14`+ search (confirmed: ~11s) is
  long enough to get real samples.
- **Not every event supports `-P` sampling** (some AMD PMC events are
  counting-only). Branch-misprediction events (`ex_ret_brn_misp`,
  `ex_ret_cond_misp`, etc.) work fine in counting mode (`pmcstat -p
  ex_ret_brn_misp -- <cmd>`, prints a single total at exit) but produce
  the same fixed ~3.8KB/zero-samples output as the "too-short workload"
  case even against a long run -- a real per-event limitation, not a
  workload-duration problem (confirmed by running the known-good
  `ls_not_halted_cyc` event against an identical command/interval/
  duration, which produced immediate real samples). Likely cause: not
  every AMD PMC event supports overflow-interrupt (PMI) delivery, which
  `-P`/`-S` sampling requires -- some are counting-only, and neither
  `pmcstat -L` nor `pmccontrol -L` distinguish which is which. The only
  way to find out is to try `-P` against a known-long-enough workload and
  check the output file size. If a per-function mispredict breakdown is
  ever actually needed, look into whether a PEBS-like interrupt-capable
  event exists for this, before assuming it's simply not possible.

## Which to reach for

- **Linux, day-to-day**: `gmake GPROF=1` + gprof. No privilege
  dependency, no sysadmin step, works today.
- **FreeBSD (`wannabe.house`)**: `pmcstat -P` + gprof, after confirming
  `hwpmc.ko` is loaded. Hardware-counter-based (actual cycles/
  instructions, not call-count-overhead-inflated like `-pg`), so more
  trustworthy for small/hot inlined-adjacent functions, and not subject
  to the multithreading self-time caveat gprof's `SIGPROF` approach has
  on Linux -- but still process-scoped, so still only reflects the one
  profiled process, not the whole system.
- **If Linux `perf` access ever gets relaxed** (someone lowers
  `kernel.perf_event_paranoid` or grants `CAP_PERFMON`), it would be the
  best of both: hardware counters *and* correct multithreaded attribution,
  without `-pg`'s inlining blind spot. Worth revisiting if a future
  profiling question specifically needs multi-threaded search behavior
  under `MP=1`.