Service auto-healing watchdogs

BOA keeps the stack alive without operator intervention through a fleet of short-lived watchdog scripts that the minute.sh launcher fans out, on the box, every few seconds. Each watchdog checks one service and — if it finds it down, stuck, or misconfigured — repairs or restarts it in place. There is no daemon: the watchdogs are spawned, run to completion, and exit. Cron re-launches minute.sh every minute; the launcher self-loops to drive a tighter cadence inside the minute.

This page documents the per-service watchdog fan-out (_launch_auto_healing in aegir/tools/system/minute.sh) and the single-shot monitors under monitor/check/. The load-reactive side of the same machinery — the high-load brakes that write /run/max_load.pid and /run/critical_load.pid, which almost every watchdog here honors — is in Load control. The cadence (how many times per minute the fan-out runs, and how that is throttled on small or CI boxes) is in Cron cadence & idle-load throttle. The security-facing member of this same /var/xdrago/monitor/ family, scan_nginx.sh, is reached through this fan-out — it is looped by nginx_guard.sh (one of the watchdogs minute.sh spawns), not run directly — and is documented in the Abuse Guard topic.

Where it runs from

On a box the launchers are deployed under /var/xdrago/; the per-service watchdogs are under /var/xdrago/monitor/check/. In the repository the same files live under aegir/tools/system/ and aegir/tools/system/monitor/check/.

cron (every minute)
   │
   ├── /var/xdrago/second.sh   ── self-loops; load-control + process guards
   │        └── (its own fan-out — see Load control / Process guards)
   │
   └── /var/xdrago/minute.sh   ── self-loops, drives the SERVICE watchdogs
            └── _launch_auto_healing()  ── spawns, per pass:
                   system.sh   unbound.sh   valkey.sh|redis.sh
                   mysql.sh    php.sh       fpm_tune.sh
                   nginx.sh    nginx_guard.sh   java.sh
                                  └── loops scan_nginx.sh (Abuse Guard)

Each watchdog re-sources /root/.barracuda.cnf on entry (so every _VAR override below is read fresh each pass) and exits immediately unless /var/log/boa/reset_no_new_password.pid exists — i.e. nothing runs until the box is a fully installed BOA system.

What minute.sh does before the fan-out: flood guards

minute.sh is not only a launcher. Before it spawns any watchdog it runs two flood guards of its own. These protect the box from a runaway monitor or a CSF reload storm spinning up faster than it drains — exactly the failure mode that would otherwise pin a small box at high load.

Both guards are skipped while a BOA run is in progress (/run/boa_run.pid) so they never fight an upgrade. _csf_flood_guard is additionally gated on the csf binary being present (/usr/sbin/csf) and no /run/water.pid (the guest-water drain) being active, and it re-asserts SYNPROXY (synproxy_reassert -p "443 80" --no-quic) when /etc/csf/csfpost.d/synproxy.sh is present.

The fan-out itself is then run _ITER times with _SLEEP seconds between passes; on a NORMAL box that is 9 passes 5 s apart, giving ~5 s monitoring granularity across the minute. See Cron cadence & idle-load throttle for the box-class logic and the _MONITOR_FANOUT_ITER / _MONITOR_FANOUT_SLEEP overrides.

Re-entrancy: the shared lock and the kill guard

Every watchdog (and the launchers) open with the same _manage_single_lock pattern. It sources lock.inc from /opt/local/bin/ or /opt/local/lib/ and, when that shared single-instance library is present, takes its lock. If the library is absent it falls back to a legacy pgrep -fc guard: if more than 2 copies of the same script are already running, it logs to /var/log/boa/too.many.log and exits 0. So a slow watchdog can never stack up more than briefly — the next launch detects the pile-up and backs off instead of restarting the service underneath a still-running instance.

The per-service watchdogs

_launch_auto_healing spawns each of the following with nohup … &, so they run concurrently and independently. The Valkey/Redis choice is made by an init-script probe: valkey.sh runs if /etc/init.d/valkey-server exists, else redis.sh if /etc/init.d/redis-server exists. fpm_tune.sh is launched only if the file is present (newer nodes).

A few load-bearing details worth calling out:

• The OOM cascade in system.sh is the heaviest action in the set. At ≤5% free RAM (computed from free -mt, not MemAvailable) it kills wkhtmltopdf, PHP CLI, nginx, PHP-FPM, Java, the cache server (wiping /var/lib/valkey or /var/lib/redis) and New Relic, then restarts Percona via move_sql.sh — a deliberate hard reset for a box seconds from a kernel OOM-kill. Between 5% and 10% free it only culls runaway wkhtmltopdf (more than 2 instances). The threshold is low on purpose: the cascade is disruptive and must fire only in a genuine emergency.

• mysql.sh requires both high load and a high thread count before it restarts Percona (_LOAD_THRESHOLD 33.0 and _THREAD_THRESHOLD 99). Either alone is normal under bursty traffic; the conjunction is what distinguishes a wedged server from a busy one. A needless MySQL restart is itself an outage, so the defaults are conservative.

• fpm_tune.sh never tunes anything live. It is purely the data collector for FPM capacity sizing; the JSONL it writes is consumed by fpmreport. See PHP-FPM & performance for how that data drives worker and memory sizing.

• nginx_guard.sh is the bridge to the security layer. scan_nginx.sh is detection-only and post-hoc, and is launched only here (10 spawns, 5 s apart) on this minute.sh fan-out — second.sh does not run it. See Abuse Guard.

Cooldowns, double-checks, and load gating

Three patterns recur across the watchdogs and are worth understanding once:

1. Double-check before acting. Almost every restart path re-reads the symptom after a short sleep (2–5 s) and acts only if it persists. This filters out the transient flap — a socket mid-reload, a log line from a restart that already happened — that would otherwise trigger a needless restart.

2. Per-service cooldown stamp. A restart writes date +%s to a /run/<service>-monitor.cooldown (or per-version /run/phpNN-fpm.cooldown) file. A subsequent failure within the cooldown window (default 30 s) logs a skip instead of restarting again, breaking restart storms when a service is crash-looping for a reason a restart cannot fix. Overridable per service: _UNBOUND_COOLDOWN_SECS, _VALKEY_COOLDOWN_SECS, _REDIS_COOLDOWN_SECS, _FPM_COOLDOWN_SECS, _NGINX_COOLDOWN_SECS, and _CRON_/_POSTFIX_/_LFD_COOLDOWN_SECS (in system.sh).

3. Load and run gating. Most watchdogs stand down while /run/max_load.pid or /run/critical_load.pid exists. Under a load emergency the box is already shedding work via Load control; piling service restarts on top would make recovery slower, not faster. They also stand down during a BOA run (/run/boa_run.pid) and while another auto-heal of the same service is mid-flight (/run/boa_<service>_auto_healing.pid).

Email reporting

When a watchdog takes a corrective action it appends to its own /var/log/boa/<service>.incident.log and may e-mail ${_MY_EMAIL}, gated by _INCIDENT_REPORT (sourced from /root/.barracuda.cnf). The normalization is uniform — NO maps to OFF, anything unrecognized maps to the script's default — but the send threshold differs by watchdog:

The important caveat: in the per-service watchdogs (unbound.sh, valkey.sh/redis.sh, php.sh, nginx.sh, java.sh) the e-mail is sent only when _INCIDENT_REPORT is ALL — their MINI/CRIT settings still log the incident but mail nothing. mysql.sh sends on anything other than OFF (its own default is MINI). Only system.sh implements the full OFF / CRIT (= ALERT-only) / ALL ladder. In every case the corrective action and the .incident.log entry happen regardless of the e-mail setting.

Reporting is additionally suppressed during the post-boot uptime grace period (_check_uptime_grace_period, from the shared lock.inc), so a reboot does not spray "service was down, restarted" mail while the stack is still coming up. See Load control for the full grace-gate conditions.

Both the mail body and the on-disk history are bounded. Every _incident_email_report sends only the last 200 lines of the service's incident log (tail -n 200), applied identically in second.sh and each of the eight mail-sending monitor/check/ watchdogs (system, nginx, php, mysql, java, unbound, valkey, redis) — an alert shows the latest entries, never the whole file history. The logs themselves are rotated by a shipped logrotate policy (aegir/conf/var/logrotate.d.boa.conf, deployed to /etc/logrotate.d/boa on both the upgrade and INIT paths): /var/log/boa/*.incident.log is rotated weekly, keeping 4 rotations, with compress + delaycompress, missingok, notifempty. No postrotate signal or copytruncate is needed — the short-lived monitors reopen the log on each append, so logrotate's default create-after-rotate is safe. Full history stays on disk in the rotated files; only the e-mail body is capped.

Verify

# recent auto-heal activity, per service
tail -n 20 /var/log/boa/*.incident.log 2>/dev/null
ls -1 /var/log/boa/*.kill.log 2>/dev/null

# is a watchdog mid-flight, or is the box load-paused?
ls -1 /run/boa_*_auto_healing.pid /run/*_load.pid 2>/dev/null

# cooldown stamps currently in force
ls -1 /run/*-monitor.cooldown /run/phpNN-fpm.cooldown 2>/dev/null

# incident-log rotation policy deployed on this box?
cat /etc/logrotate.d/boa 2>/dev/null

Related

• Load control & auto-pause — the load-reactive brakes (max_load.pid / critical_load.pid) that gate these watchdogs, and the uptime grace gate shared with this page.
• Cron cadence & idle-load throttle — box-class fan-out cadence and the _MONITOR_FANOUT_* / _MONITOR_HEAVY_EVERY knobs.
• Process guards & auth scanners — the second.sh side: the _proc_control service guards and the auth scanners.
• Abuse Guard — the security scanner (scan_nginx.sh) driven by the same monitor home via nginx_guard.sh on this minute.sh fan-out.
• PHP-FPM & performance — how the fpm_tune.sh JSONL feeds fpmreport and capacity sizing.
• Reference appendix — every _VAR override named above with its default and source.