Cron cadence & idle-load throttle

BOA's auto-healing watchdogs are not long-running daemons. They are short-lived scripts fired from cron, fanned out many times a minute so that a stuck service or a load spike is caught within seconds rather than at the next minute boundary. That responsiveness has a cost: on a small or idle box the fan-out itself is the dominant load source. This page documents the cadence model, why it shows up as load 3–4 on an otherwise idle 2 CPU / 4 GB box, and the box-class throttle that tames it on small/idle/CI hosts while leaving normal production hosts unchanged.

The security-facing member of this same machinery — the Abuse Guard (scan_nginx.sh) — is documented separately in the Abuse Guard topic; it runs on its own cron tick and is not throttled by the box class. The high-load auto-pause that second.sh performs is covered in Load control and is never throttled.

The cadence model

The root crontab (aegir/tools/system/cron/crontabs/root) launches the two monitor loops once per minute:

* * * * *   bash /var/xdrago/second.sh   >/dev/null 2>&1
* * * * *   bash /var/xdrago/minute.sh   >/dev/null 2>&1

Neither runs once and exits. Each self-loops with sleep between passes so that one cron invocation covers the whole minute at roughly 5-second granularity:

cron fires (once/min)
   │
   ├── second.sh ── 10 passes, sleep 5 between ──┐
   │       each pass: sample load (cheap)        │  ~50 s of wall-clock
   │       every Nth pass: heavy fan-out         │  per invocation
   │                                             │
   └── minute.sh ── N passes, sleep S between ───┘
           each pass: _launch_auto_healing
                      (≈8 watchdog scripts, detached)

minute.sh calls _launch_auto_healing on every pass of its loop, which nohup-spawns the per-service watchdogs: system.sh, unbound.sh, valkey.sh/redis.sh (whichever is installed), mysql.sh, php.sh, fpm_tune.sh (if present), nginx.sh, nginx_guard.sh, java.sh. second.sh samples /proc/loadavg every pass via _load_control → _get_load (cheap) and, on its heavy passes, runs the watchdog fan-out _proc_control plus the hackcheck.sh, hackftp.sh, and escapecheck.sh scanners.

Every one of those spawned children re-sources /root/.barracuda.cnf and runs one or more pgrep scans before deciding it has nothing to do. That re-source + pgrep cost is paid on every spawn whether or not the watched service needs healing.

Why an idle box shows load 3–4

On a 2 CPU / 4 GB box with the Aegir queue and all site cron disabled, the load average still sits around 3–4. That is not a bug in any single watchdog — it is the spawn churn:

• minute.sh at the NORMAL cadence does 9 passes × ≈8 spawns = ~72 watchdog spawns/min, plus their pgrep/awk/source helper forks.
• second.sh adds its 10 load samples plus a heavy fan-out (_proc_control iterates ~12 service guards, each detached) on every pass.
• Net result is on the order of 100–150 short-lived processes per minute, each re-reading .barracuda.cnf and scanning the process table.

The Linux load average counts processes that are runnable or in uninterruptible sleep, sampled continuously. A constant stream of fork → source → pgrep → exit keeps several processes runnable at almost any instant, so the average number of runnable tasks stays near 3–4 even though no single task runs for long and the box does no real work. This is why only disabling cron drops the load to ~0: removing the launchers removes the churn. Tuning an individual watchdog does not help; the cost is in the cadence, not the work.

The gap this fixed

BOA already had two control files meant to signal "this is a small or slow box, go easy": .slow.cron.cnf and .look.like.jenkins.cnf. Historically they were honoured only by the task-queue path:

• runner.sh reads .look.like.jenkins.cnf, .slow.cron.cnf, .fast.cron.cnf, and .force.queue.runner.cnf to decide whether and how aggressively to drain the Aegir queue, and it auto-creates .slow.cron.cnf (with chattr +i) on any box with ≤ 4096 MB RAM.
• usage.sh only checks .look.like.jenkins.cnf to bail out entirely on CI.

Neither second.sh nor minute.sh consulted any of these signals. So the monitor fan-out — the dominant idle-load source — ran at the full NORMAL 5-second cadence on a small or CI box exactly as it did on a 192 GB production host. Throttling the queue while leaving the watchdog fan-out at full speed left most of the idle load in place.

_monitor_box_class: CI / SLOW / NORMAL

The fix adds an identical classifier function, _monitor_box_class, to both loops. It reuses the same signals the queue path already honours, plus a direct RAM check as a fallback, and resolves to one of three classes with this exact precedence:

The order matters: a CI box is CI even if it is also small; an explicit .force.queue.runner.cnf wins over the RAM/slow heuristic so an operator can force full cadence on a small box; otherwise the ≤ 4 GB RAM check (read from free -m) classifies the box SLOW even if runner.sh has not yet written .slow.cron.cnf. Because runner.sh auto-creates that immutable marker on ≤ 4 GB boxes, most small hosts hit the SLOW branch by either path. Note that .fast.cron.cnf is deliberately not honoured by this classifier (unlike runner.sh's own queue-drain cadence): runner.sh ignores .fast.cron.cnf while .slow.cron.cnf is set, so honouring it here would un-throttle exactly the tiny boxes the throttle targets (e.g. a 4 GB box carrying both markers). Only .force.queue.runner.cnf forces NORMAL.

These control files are described in Host control files & INI.

Per-class cadence

The two loops throttle different things. minute.sh reduces both the number of passes and the sleep between them. second.sh keeps its responsive 10×/5 s load-sampling loop intact — load sampling is cheap and auto-pause must stay timely — but gates only the heavy fan-out (the watchdog _proc_control and the hack/escape scanners) to run every Nth pass.

NORMAL          SLOW            CI
──────          ────            ──
minute.sh
 9 passes        3 passes        1 pass
 sleep 5         sleep 18        (single)
 ≈72 spawns/min  ≈24 spawns/min  ≈8 spawns/min

second.sh   (always 10 passes, sleep 5 — load sampling unchanged)
 heavy every     heavy every     heavy every
 1 pass          4 passes        10 passes
 (10×/min)       (≈3×/min)       (1×/min)

The defaults are derived directly from source: minute.sh sets _ITER/_SLEEP per class (NORMAL 9/5, SLOW 3/18, CI 1/5); second.sh sets _HEAVY_EVERY (NORMAL 1, SLOW 4, CI 10). NORMAL values are exactly the historical behaviour, so a normal production host is byte-for-byte unchanged. SLOW and CI trade fan-out granularity for idle quiet — a tiny or CI box does not need a 5-second watchdog heartbeat.

second.sh keeps its full 10-pass / sleep 5 loop in every class because that is where the high-load auto-pause sampling lives. The load detection and service-pause logic in Load control therefore reacts just as fast on a SLOW or CI box as on a NORMAL one; only the expensive watchdog fan-out is spaced out.

Under the hood — the heavy-pass gate. second.sh runs the heavy block on iteration k when (k - 1) % _HEAVY_EVERY == 0. On SLOW (_HEAVY_EVERY=4) that is iterations 1, 5 and 9 — three heavy passes per minute; on CI (10) only iteration 1.

Override variables

Each computed default can be overridden from /root/.barracuda.cnf. The override must be an integer; the value read from the config replaces the per-class default, and each is then floored at 1 so a misconfiguration cannot disable the loop entirely.

The validation is [[ "${_VAR}" =~ ^[0-9]+$ ]] then (( _VAR < 1 )) && _VAR=1, so a non-numeric or empty value is simply ignored and the class default stands.

NORMAL is unchanged. With no override and no slow/CI marker, the defaults reproduce the historical cadence exactly. You only need these variables on a box where the class default is not aggressive enough (or, rarely, where you want to force a NORMAL box quieter).

second.sh, minute.sh and loadreport are serial-gated in BOA.sh.txt, so each box automatically re-fetches the current copy whenever one of these scripts changes — you do not pull updates by hand.

Tuning idle load (operations recipe)

The throttle is measured with loadreport (aegir/tools/bin/loadreport, run from cron as /opt/local/bin/loadreport --log every 30 minutes), a read-only /proc profiler that attributes CPU and RSS to each recurring monitor script. Use it to confirm the win on a specific box. See loadreport for the full tool reference.

Confirm the class first. A live loadreport run prints a monitor throttle: header line with the class the box actually resolves to (CI, SLOW or NORMAL). It does not re-implement the precedence table above — it extracts and runs the deployed minute.sh's own _monitor_box_class, so the line is drift-proof by design: it tracks what the deployed classifier does even across BOA versions. The parenthesised cnf: list shows which of the four marker files (table below) exist, printed as short labels (jenkins, slow.cron, fast.cron, force.queue.runner; none when absent) — the quickest way to see why a box classified the way it did when it carries contradictory markers. On a box still running a pre-throttle minute.sh the line reads n/a — pre-throttle minute.sh. The --data history summary does not repeat this line.

Confirm before/after:

# Baseline (current class behaviour), live 60s profile:
loadreport --top 40

# Summarise the logged history written by the cron --log run:
loadreport --data /var/log/boa/load-profile --days 1

second.sh/minute.sh (and their _launch_auto_healing / _proc_control children) should dominate the CPU columns on an otherwise idle box; that is the churn this throttle targets.

On a tiny CI box where even the SLOW/CI defaults leave more heartbeat than the workload needs, push the knobs further in /root/.barracuda.cnf, e.g.:

# even quieter monitor fan-out on a disposable CI VM
_MONITOR_FANOUT_ITER=1
_MONITOR_HEAVY_EVERY=20

Re-run loadreport after the next couple of minutes to confirm the spawn count and CPU share dropped, and that the per-service watchdogs still fire often enough for your purposes. Do not raise _MONITOR_FANOUT_SLEEP so high that a stuck service goes unnoticed for minutes on a host you actually care about — the trade-off is detection latency for idle quiet, and it is only safe to lean hard on disposable or genuinely idle hosts.

Control files involved

These are catalogued in Host control files & INI; the /root/.barracuda.cnf that holds the override variables is in the Reference appendix.

Related

• Load control & auto-pause — the load sampling that is deliberately exempt from this throttle, and the SPIDER/MAX/CRIT ladder.
• Process guards & auth scanners — _proc_control and the scanners whose cadence _HEAVY_EVERY governs.
• Service auto-healing watchdogs — the minute.sh fan-out whose _ITER/_SLEEP this page throttles.
• loadreport profiler — the before/after measurement tool for this change.
• Host control files & INI — the .slow.cron.cnf / .look.like.jenkins.cnf / .fast.cron.cnf / .force.queue.runner.cnf markers.
• Reference appendix — the _MONITOR_* overrides and their defaults.