Self-healing monitor stack

BOA runs no external monitoring agent and no resident daemon. Every box watches and repairs itself from a single root crontab that fires a handful of short-lived bash scripts every minute. Those scripts sample load, restart dead services, kill runaway processes, scan auth logs for abuse, and drain the Aegir task queue — then exit. There is nothing long-running to crash; the watchdog is the cron tick.

This topic covers the service / load / process side of that machinery: what cron launches, the self-looping fan-out that gives it sub-minute reaction time, the per-service watchdogs, the load auto-pause ladder, the process guards and auth scanners, the box-class throttle that keeps the fan-out from pinning idle load on small hosts, and the loadreport profiler used to measure it.

The security-facing members of the same /var/xdrago/monitor/ family — the nginx log scorer scan_nginx.sh and the CSF ban pipeline — live in the Abuse Guard topic, and the per-host firewall those bans drive is in Security & isolation. This topic points at those rather than repeating them.

The root crontab

The installed root crontab lives at /var/spool/cron/crontabs/root. Its master copy is aegir/tools/system/cron/crontabs/root, deployed to /var/xdrago/cron/crontabs/root and copied into place on every install/upgrade. Do not hand-edit the installed file — its own header says so, and the next upgrade overwrites it. Per-box additions go in /var/xdrago/cron/custom.txt, which BOA appends after the copy.

The per-minute monitors (second.sh, minute.sh, guest-fire.sh, runner.sh) are the live self-healing core. The 2-minute and longer jobs are slower-changing maintenance. The nightly owl.sh run has its own page — Nightly maintenance (owl.sh); the dump/backup jobs are out of scope here — see Backups.

Where the pieces live

The repo source under aegir/tools/system/ is deployed verbatim to /var/xdrago/ on the box (cp -af .../aegir/tools/system/* /var/xdrago/). So a file the source tree calls aegir/tools/system/minute.sh is /var/xdrago/minute.sh at runtime. The single-shot monitors live one level down in /var/xdrago/monitor/check/. loadreport is the exception — it is deployed to /opt/local/bin/loadreport with a /usr/local/bin/loadreport symlink, like fpmreport.

Every per-service watchdog launched by minute.sh re-sources /root/.barracuda.cnf on entry (so every _VAR override below is read fresh each pass) and exits immediately unless /var/log/boa/reset_no_new_password.pid exists — i.e. the auto-healing watchdogs stay dormant until the box is a fully installed BOA system. The second.sh process guards and the launchers themselves do not gate on that marker.

Pages in this topic

• Service auto-healing watchdogs — the minute.sh fan-out and the monitor/check/* single-shot watchdogs that restart dead services (nginx, php-fpm, mysql, unbound, valkey/redis, java, system), the OOM cascade, and the cooldown / double-check / load-gating patterns they share.
• Load control & auto-pause — second.sh's per-CPU load model, the SPIDER / TASK / MAX / CRIT thresholds and their _CPU_*_RATIO defaults, the escalation ladder, the /run/*_load.pid state files, and incident reporting.
• Process guards & auth scanners — the _proc_control service guard set, the hackcheck / hackftp / escapecheck auth and shell-escape scanners, the flood guards, and the runner.sh task-queue runner.
• Cron cadence & idle-load throttle — why an idle box shows load 3–4, the _monitor_box_class CI / SLOW / NORMAL classification, the per-class cadence, and the _MONITOR_FANOUT_ITER / _MONITOR_FANOUT_SLEEP / _MONITOR_HEAVY_EVERY overrides.
• loadreport profiler — the read-only /proc profiler that attributes CPU and RSS to each monitor script, its JSONL history log, and how to read its output to find the costly pass.
• Nightly maintenance (owl.sh) — the 04:15 nightly orchestrator and its /var/xdrago/night/ workers, the frozen run context, per-account logs, and the _NIGHT_PARALLEL / _NIGHT_MAX_PARALLEL fan-out.
• Ghost & empty-artifact cleanup — the opt-in dry-run-by-default reapers for ghost codebases, empty platforms, orphaned vhosts/sites/aliases, and their always-on safeguards.

Cross-cutting

• The security half of this same /var/xdrago/monitor/ machinery — scan_nginx.sh, the request scorer, and the guest-fire → guest-water → nginx_deny ban pipeline — is the Abuse Guard. This topic deliberately does not repeat it.
• The firewall those guards drive — temp bans, the allow list, LFD, SYNPROXY — is Security & isolation. The flood guards in this topic also pkill/purge runaway csf processes.
• Every monitor re-sources its config from /root/.barracuda.cnf on each spawn; the _CPU_*_RATIO load thresholds, the _INCIDENT_REPORT alert level, and the _MONITOR_* throttle overrides are catalogued in the host control-file reference and the Reference appendix.