loadreport — monitor resource profiler

loadreport answers one question: on an idle box with no site traffic, which recurring script is burning the load average?

BOA's monitor machinery (second.sh, minute.sh, the /var/xdrago/monitor/check/ guards) fans out roughly twenty short-lived helper scripts every ~5 seconds, and each one re-sources /root/.barracuda.cnf and runs pgrep scans. That churn alone can pin a 2 CPU / 4 GB box at a load of 3–4 with zero requests served. loadreport is the read-only /proc profiler that attributes the CPU and RSS of that churn back to the script responsible, so you can decide what to throttle.

It is dependency-free (pure bash + /proc, sort/awk only at print time), writes no configuration, and changes nothing. Linux only. Source: aegir/tools/bin/loadreport.

Where it lives

loadreport is deployed exactly like fpmreport: the real file is /opt/local/bin/loadreport and a symlink at /usr/local/bin/loadreport puts it on PATH (fetched and symlinked by BOA.sh.txt, chmod 700). Invoke it as loadreport from a root shell.

It is not one of the /var/xdrago/ monitors — it does not run itself in a loop and does not enforce anything. It is a diagnostic you run by hand, plus one low-priority cron entry (below) that logs a periodic sample for history.

What problem it solves

The targets are deliberately hostile to ordinary profilers: the monitor children live for a fraction of a second, so a top/ps snapshot almost never catches them, and sysstat/pidstat/process accounting are not installed on a stock BOA box. loadreport sidesteps all of that by sampling /proc repeatedly across a window and reconstructing per-script totals from short-lived processes.

cron fires second.sh / minute.sh once a minute
        │
        │  self-loop with `sleep 5`  →  ~5 s granularity
        ▼
   fan-out: ~20 short-lived monitor scripts every ~5 s
        │      each re-sources /root/.barracuda.cnf, runs pgrep
        ▼
   load average climbs on an otherwise idle box
        │
        ▼
   loadreport  ── samples /proc over a window ──►  who did it?

How it works

The script's header and sampling code spell out the model. The important parts:

• Identity by (pid, starttime). Because the targets are short-lived and PIDs get recycled, every process is keyed by its PID and its start tick. Two different scripts that happen to reuse a PID are never conflated.
• Birth-aware CPU charging. The window start is recorded in boot-relative clock ticks from /proc/uptime. A process whose start tick is at or after the window start was born during the window and is charged its full cumulative CPU (utime+stime) from birth; a pre-existing process is charged only its in-window delta. This is what makes per-run totals for a script that spawns fresh every tick meaningful.
• bash/sh resolve to the script. A process whose comm is bash/sh/dash is relabelled to the basename of the first *.sh/*.pl/*.php argument in its cmdline. So a hundred bash wrappers collapse into minute.sh, scan_nginx.sh, etc., instead of a useless bash bucket.
• Helper forks roll up to the launcher. pgrep, awk, bc, sleep and friends are attributed to the BOA launcher that spawned them by walking the /proc parent chain (depth-bounded to 12). The walk stops at a known launcher or at cron/crond/CRON (anything past that is (non-cron)). The recognised launchers are second.sh, minute.sh, runner.sh, guest-fire.sh, guest-water.sh, owl.sh, clear.sh, ip_access.sh, ai_policy.sh, nginx_deny.sh, migration_proxy_realip.sh, cloudflare_realip.sh, manage_ltd_users.sh, manage_solr_config.sh, purge_binlogs.sh, mysql_cleanup.sh, graceful.sh.
• Systemic fork/ctxt bounds. The window's processes and ctxt deltas from /proc/stat bound the total fork and context-switch churn, including processes too short-lived to ever appear in a sample.
• Self-excluded. loadreport skips its own PID and its own sleep child so the profiler never charges itself.

Two views

The human report prints two tables:

• BY COMMAND — one row per resolved script basename: summed CPU seconds, percent of one core over the span, spawn count, peak RSS for a single process, and concurrent RSS (the worst-case sum of that script's live instances in any one sampling pass).
• BY LAUNCHER (subtree) — the same CPU rolled up to the owning cron launcher, so a launcher's whole cost (itself plus every helper fork it triggers) lands in one number. This is the view that tells you which cron entry is the real load source.

Usage

Numeric inputs are validated defensively and fall back to the default on garbage rather than dividing by zero: a zero interval would collapse the run to a single useless sample, so it is rejected.

Environment tunables

Every default is overridable from the process environment or an equivalent CLI flag. loadreport does not read /root/.barracuda.cnf (unlike the watchdogs), so these are tuned by exporting the variable before the run, not from a control file:

The periodic logger

The root crontab runs the logger every 30 minutes at idle priority:

*/30 * * * * /usr/bin/nice -n10 /usr/bin/ionice -c3 \
  bash /opt/local/bin/loadreport --log >/dev/null 2>&1

The nice -n10 / ionice -c3 (idle I/O class) keeps the logger off the back of real work — it profiles for a minute, so it should never compete. The --log path:

• Skips proxy nodes — if /root/.proxy.cnf exists it exits 0 immediately; a proxy has none of the monitor fan-out worth profiling.
• Runs a normal live profile, then appends the JSON object as one line to ${_LOADPROF_DATA}/YYYY-MM-DD.jsonl (one file per day).
• Prunes *.jsonl older than _LOADPROF_KEEP_DAYS (default 14) via find … -mtime +N -delete, so the history stays bounded.

Read that history back with --data, which walks the retained records, averages CPU seconds per command across them, and tracks the peak concurrent RSS:

loadreport --data /var/log/boa/load-profile --days 7

Reading the output

The example below is illustrative, constructed from the printf formats in the source — it is not captured from a real box. It shows the shape of a quiet 2 CPU box where the monitor fan-out, not site traffic, is the load:

loadreport — BOA recurring-script resource profile
  host  ng019   span 60s   interval 1s   cores 2   HZ 100
  monitor throttle: SLOW   (RAM 3915 MB; cnf: slow.cron)
  load1 3.41 -> 3.18
  forks during span: 2280 (2280/min)   ctxt switches: 41200 (687/s)
  distinct processes sampled: 214

  BY COMMAND                     CPU_s  %1core  spawns  peakRSS  concRSS
  minute.sh                       9.74    16.2      11      6.1     34.8
  second.sh                       7.05    11.8      12      5.9     41.2
  scan_nginx.sh                   3.61     6.0       3      9.4      9.4
  pgrep                           2.88     4.8     180      1.2     12.6
  ip_access.sh                    1.10     1.8       1      4.7      4.7

  BY LAUNCHER (subtree)          CPU_s  %1core   procs
  minute.sh                      14.92    24.8      96
  second.sh                      11.40    19.0     104
  (non-cron)                      1.95     3.2      14

How to read it:

• The monitor throttle: line is the box's live throttle class (CI / SLOW / NORMAL) as classified by the deployed monitors — the canonical way to confirm what class a box is actually running in. It is drift-proof by design: loadreport extracts (sed) and evals the _monitor_box_class function from the deployed /var/xdrago/minute.sh rather than re-implementing the precedence, so it always reflects what the box does, not what the docs say. On a box still running a pre-throttle minute.sh it prints n/a — pre-throttle minute.sh. The parenthesised cnf: list names which class marker files exist (jenkins, slow.cron, fast.cron, force.queue.runner, or none) — diagnostic when a box carries contradictory markers and you need to see which ones the classifier is weighing. --json emits the class as "monitor_class". See Cron cadence & idle-load throttle for what each class changes.
• The summary line — forks during span and ctxt switches — is the systemic churn. 2280/min forks on an idle box is the fan-out, not your sites. This number includes processes too short-lived to be named individually (see the caveat below).
• BY COMMAND names the worst scripts. pgrep with 180 spawns but little CPU each is the per-child cost of the monitor scans, multiplied across the fan-out.
• BY LAUNCHER rolls every one of those pgrep/awk forks back under minute.sh / second.sh. When those two subtrees dominate the launcher table — as here — the fan-out is your idle load.
• concRSS is the worst-case concurrent memory: minute.sh peaked at ~35 MB of simultaneously-live instances even though any single one was ~6 MB.
• scan_nginx.sh shows up as its own BY COMMAND row, but because it is not in the launcher list its cost rolls up under minute.sh (its launcher, via nginx_guard.sh) in the BY LAUNCHER table.

When the fan-out is the problem — throttle it

The diagnosis that points back at this topic: a high forks/min combined with second.sh / minute.sh dominating the BY LAUNCHER table, on a box with little or no real traffic. That is the monitor fan-out spending your CPU on pgrep scans nobody needs at full cadence on a small or idle host.

The same change that shipped loadreport also added a box classifier (_monitor_box_class, CI / SLOW / NORMAL) to second.sh and minute.sh to throttle that fan-out on small, idle, or CI hosts while leaving NORMAL boxes unchanged. The relevant /root/.barracuda.cnf overrides are _MONITOR_FANOUT_ITER and _MONITOR_FANOUT_SLEEP (minute.sh) and _MONITOR_HEAVY_EVERY (second.sh). See Cron cadence & idle-load throttle for the classifier and how to tune it; loadreport is the before/after measurement tool for that change — and its monitor throttle: header line (above) reads the class straight from the deployed minute.sh, so a single run confirms whether a throttle you expect is actually in effect.

The --log history is exactly the artefact you want here: take a baseline over a few days, apply a throttle, and compare the averaged per-command CPU with --data rather than eyeballing one run.

Caveat — sub-tick processes are counted, not named

A process born and reaped entirely between two ticks never appears in a sample, so it is not named in either table. It is still counted in the systemic forks total from /proc/stat, which is why that number can exceed the sum of the named spawns. If the fork total is high but the named tables look light, you are missing short-lived churn: rerun with a finer --interval (e.g. --interval 0.5 or 0.25) to catch more of it. There is a floor — processes shorter than the sample gap are inherently invisible by name — but the /proc/stat counters always bound the true total.

Related

• Cron cadence & idle-load throttle — the _monitor_box_class classifier and the _MONITOR_FANOUT_* / _MONITOR_HEAVY_EVERY knobs that reduce the fan-out loadreport measures.
• Process guards & auth scanners — the heavy fan-out (_proc_control + scanners) that loadreport attributes to second.sh.
• Service auto-healing watchdogs — the minute.sh fan-out that dominates the BY LAUNCHER table on an idle box.
• Abuse Guard — the security-facing member of the same /var/xdrago/monitor/check/ machinery (scan_nginx.sh). It shows up in loadreport as its own BY COMMAND row; because it is not a recognised launcher, its cost rolls up under minute.sh (via nginx_guard.sh) in the BY LAUNCHER table. This page measures it; that topic explains what it does.
• Reference appendix — where the _LOADPROF_* and _MONITOR_* overrides are documented.