Monitor deploy surfaces & registration

A BOA monitor is a single-shot bash script under aegir/tools/system/monitor/check/ that is fired from cron many times a minute, does one job, and exits. There is no daemon, no supervisor, and no service unit. Shipping a new monitor or agent means wiring it into every surface that copies it onto a box, fetches it between releases, and actually launches it — miss one and the file either never arrives or arrives and never runs. This page is the registration checklist, each item grounded in the surface that enforces it, plus the box-class throttle that decides how often your monitor's launcher fans out.

Two files decide the layout you follow: an in-tree monitor lives at aegir/tools/system/monitor/check/<name>.sh; on a box it lands at /var/xdrago/monitor/check/<name>.sh. Those two paths are the whole deploy story — everything below is how the code moves the file between them and calls it.

Where a monitor lives — in-tree vs on-box

/var/xdrago/ mirrors the whole aegir/tools/system/ subtree. The /var/xdrago/monitor/check/ directory is created (if absent) by both deploy surfaces before anything is written into it (BOA.sh.txt:1242, system.sh.inc copy path below). A monitor that is not in the in-tree directory is never copied; a monitor that is not in the per-file fetch list is never refreshed between full upgrades — you need both.

The four surfaces a new monitor touches

Shipping newthing.sh means editing, in one change:

1. The in-tree file — aegir/tools/system/monitor/check/newthing.sh. This is the source of truth the wholesale copy and the per-file fetch both pull from.
2. A launch surface — the loop that actually spawns it (minute.sh _launch_auto_healing, or second.sh _proc_control, or the _pncw guard list). A file in monitor/check/ that no loop calls never runs.
3. The _update_agents per-file fetch in BOA.sh.txt — a _fetch_versioned line with an fNN serial, so long-lived boxes pick up the file (and later edits) within one SKYNET tick, not only at the next full barracuda up-*.
4. Base-package deps, if the monitor calls a binary the base install does not guarantee (system.sh.inc:_run_aptitude_deps_install). Ensured outside the FULL-upgrade gate, because monitors deploy at any time.

The rest of the page is each surface in turn.

Surface 1 — the wholesale copy (_xdrago_install_upgrade)

A full barracuda up-* run stages the build tree at _bldPth=/opt/tmp/boa and _xdrago_install_upgrade (lib/functions/system.sh.inc:9152) copies the whole system-tools subtree over the live one:

cp -af ${_bldPth}/aegir/tools/system/* /var/xdrago/ &> /dev/null

Before the copy it snapshots the old tree and the root crontab to ${_vBs}/dragon/x/xdrago-pre-${_xSrl}-${_X_VERSION}-${_NOW} (rollback), and after it re-asserts chmod -R 700 /var/xdrago/monitor/check (:9292). So on a full run your in-tree monitor/check/newthing.sh reaches the box for free — this is the path with no serial. Its "serial" is the release identity (_xSrl / _X_VERSION) itself, exactly like lib/functions/*.sh.inc; the copy is unconditional on any full upgrade of an installed box (gated only on /var/xdrago/conf + the Hostmaster alias existing).

The catch: a full upgrade is the slow path — it happens manually or at the scheduled autoupboa window, not every few minutes. If your monitor (or a fix to one) must reach the fleet fast, the wholesale copy is not enough. That is surface 3.

Surface 2 — the launch surfaces

monitor/check/*.sh files do not run themselves. Three launchers spawn them, and which one you register in sets the cadence and the gating:

The root crontab fires both loops once per minute (aegir/tools/system/cron/crontabs/root:4-5); each self-loops with sleep so one invocation covers the whole minute:

* * * * *   bash /var/xdrago/second.sh   >/dev/null 2>&1
* * * * *   bash /var/xdrago/minute.sh   >/dev/null 2>&1

Pick the launcher by the monitor's job:

• A per-service healer (restart a daemon if down) → add a nohup … & line to _launch_auto_healing in minute.sh. It runs on every fan-out pass, so it gets the tightest cadence.
• A "keep this agent alive" guard modelled on the proc_num_ctrl.pl split-outs → add the basename to the _w list in _proc_control (second.sh:232-234). The loop tests [ -e "${_monPath}/<name>.sh" ] before spawning, so an absent file is skipped — but the name must be in the list.
• A security/forensic scanner → follow hackcheck.sh/escapecheck.sh in the second.sh heavy block.

The _w / _pncw guard list is duplicated in three places — keep them in sync. The same split-out guard list appears in second.sh _proc_control (the for _w in … at :232-234), and twice in lib/functions/system.sh.inc as the _pncw dist-upgrade kick (:3240-3242 and :7806-7808, each right after a nohup /var/xdrago/minute.sh). A new guard that must fire during a distro-migration reboot needs adding to all three lists; a guard that only needs the steady-state cadence needs only the second.sh copy. These lists are intentionally hand-maintained, not derived — do not try to DRY them.

The nginx_guard → scan_nginx relationship

nginx_guard.sh is in _launch_auto_healing, but the Abuse Guard scorer scan_nginx.sh is not in any launcher list — nginx_guard.sh launches it. On the normal path it fires the scorer in its own inner for _iteration in {1..10}; do … sleep 5; done loop (nginx_guard.sh:87-90), so one nginx_guard.sh invocation spawns scan_nginx.sh ten times over ~50 s; on the load-paused branch it runs a single bounded pass (:104-105). So the scorer's cadence is a product of two surfaces — that inner 10× loop, multiplied by how many times minute.sh's fan-out spawns nginx_guard.sh per minute (the box-class _ITER) — never a line of its own. A new detector inside scan_nginx.sh inherits that surface automatically; a genuinely new launcher-level scanner does not. The scorer's own internals are the Abuse Guard code internals leaf.

Re-entrancy: every monitor self-guards

A launcher spawns the same monitor every pass, and cron fires the launcher every minute, so a slow pass can overlap the next. Each monitor guards itself at the top by defining and calling a _manage_single_lock wrapper (system.sh:31-49 is the canonical shape, called at :50): it sources lock.inc from /opt/local/bin/lock.inc or /opt/local/lib/lock.inc, then calls the library's _single_instance_lock only when the lib set _SINGLE_INSTANCE_LIB_VER and _single_instance_lock is defined; otherwise it falls back to a pgrep -fc "$(basename "$0")" count that exit 0s when more than two instances are already running (_CNT > 2). A new monitor should carry the same wrapper — the launchers do no locking for you. lock.inc itself is a serial-fetched tool (_update_agents), so it is present on any current box.

Surface 3 — the _update_agents per-file fetch

The fast transport is per-file, serial-gated. _update_agents in BOA.sh.txt carries one _fetch_versioned line per monitor, all inside a block gated on the box being installed:

if [ -e "/var/xdrago/monitor/check" ] \
  && [ -e "/var/aegir/.drush/hm.alias.drushrc.php" ]; then
  ...
  _fetch_versioned /var/xdrago/monitor/check/scan_nginx.sh \
    "${_urlHmr}/tools/system/monitor/check/scan_nginx.sh" fNN
  _fetch_versioned /var/xdrago/monitor/check/nginx_guard.sh \
    "${_urlHmr}/tools/system/monitor/check/nginx_guard.sh" fNN
  ...
fi

_urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir" — the fetch URL path mirrors the repo's aegir/ subtree exactly, so the URL for monitor/check/newthing.sh is ${_urlHmr}/tools/system/monitor/check/newthing.sh. _update_agents is driven off the SKYNET self-update tick (clear.sh re-streams BOA.sh.txt every 5 minutes), so a serial bump here reaches installed boxes within one tick — the deep stack still waits for a full upgrade, but the monitor file does not. The two transports and how a tagged release propagates are the serial & fetch pipeline leaf; only the monitor-specific mechanics are below.

How serial gating decides re-fetch

_fetch_versioned <dest> <url> <serial> writes a stamp ${_pthLog}/<basename>.ctrl.<serial>.${_tRee}.${_xSrl}.pid on a successful fetch. Next tick it returns early (no download) when that stamp exists and the destination is non-empty and any declared --symlink is intact. So an unchanged monitor is fetched exactly once per serial value and then skipped every tick until the serial changes. Editing the monitor's body without changing its serial means installed boxes keep their stamp and never re-download your edit — the file only converges at the next full wholesale copy.

To force a re-fetch, decrement the fNN serial — this is the whole mechanism that makes an edit land fleet-wide within a tick. The direction is fixed and load-bearing:

• fNN counts DOWN. Bump = decrement (f99 → f98 → … → f01), per the in-code rule stated in the _update_boa_tools header comment ("bump a script's serial (counts DOWN, f99->f98...) to force a refresh after editing it").
• Never increment, never re-use a value. A box that skipped an intermediate serial (offline, mirror lag) may still hold a stamp bearing an older, higher value; reusing that value makes the box match its stale stamp and silently skip the refetch. Monotonic decrement makes stamp collision impossible.
• Do not cite the current numeric value in the doc or in comments — it churns every release; read it from BOA.sh.txt at HEAD. State the rule, not the number.

A new monitor's _fetch_versioned line starts at f99. On top of the fetch line, _update_agents also carries a self-heal stanza just above the fetch block: [ ! -e "/var/xdrago/monitor/check/<name>.sh" ] && rm -f ${_pthLog}/<name>.sh.ctrl.*.pid (BOA.sh.txt:1279-1306), which drops the stamp when the file went missing so the next fetch re-installs it rather than trusting a stamp for an absent file. Add the matching line for a new monitor.

Monitors are not in the shared dual-fetch group. The core CLI tools (barracuda, boa, octopus, xcopy, the synproxy family, …) are fetched by both _update_agents and _update_boa_tools against one shared stamp, so their serials must be bumped in both sites together. The monitor/check/*.sh fetches live only in _update_agents, so a monitor serial is bumped in one place. The pairs that do need both-site bumps (autoupboa/fixmounts, the synproxy family) carry an explicit "bump serials in both functions together" comment at the call site — heed it; a divergent shared serial makes each function wipe the other's stamp and refetch every tick, fleet-wide.

Retiring a monitor

Removing the in-tree file is not enough — long-lived boxes keep the deployed copy and its stamp. _update_agents deletes retired files explicitly by path (e.g. the Perl-generation leftovers scan_nginx.pl, locked_nginx.pl, locked_java.pl at BOA.sh.txt:1319-1321, joining the earlier .pl cleanups), and *.sh.old/*.pl.old backups are swept (:1327). To retire a monitor, add the rm -f /var/xdrago/monitor/check/<name>.sh cleanup line and drop it from the fetch list and every launcher list. The retired Perl monitor generation is recorded in Discontinued features.

Surface 4 — base-package deps

If your monitor calls a binary the base install does not already guarantee, add it to _run_aptitude_deps_install (lib/functions/system.sh.inc) — outside the FULL-mode gate. The pattern is the fpm_tune monitor's cgi-fcgi dependency:

# Required by the fpm_tune monitor (cgi-fcgi). Ensured outside the FULL-mode
# gate below because monitors deploy/update via _update_agents at any time,
# while FAST upgrades skip the big package section -- every barracuda run
# must guarantee this dependency.
if ! _pkg_installed "libfcgi-bin"; then
  _mrun "${_INSTAPP} libfcgi-bin"
fi

The reasoning is the transport split: a monitor can arrive via the _update_agents per-file fetch at any SKYNET tick, independent of any full upgrade, but a FAST barracuda up-* skips the large package section. If the dep were only installed in the FULL block, a box that fetched the new monitor but has not run a FULL upgrade would run it against a missing binary. Ensuring the package on every run — FAST or FULL — closes that window. Guard with _pkg_installed so the check is idempotent and cheap.

The box-class throttle — what gates your launcher's cadence

The launch surfaces in surface 2 do not run at a fixed rate. Both loops classify the box and throttle the fan-out on small, idle, or CI hosts, because on those boxes the fan-out itself — every spawned child re-sources /root/.barracuda.cnf and runs pgrep scans — is the dominant idle-load source. A historical pre-throttle 2 CPU / 4 GB box sat at load 3–4 with no traffic purely from spawn churn: ~100–150 short-lived processes a minute, each fork → source → pgrep → exit keeping several tasks runnable at any instant. The throttle exists to tame that on hosts that do not need a 5-second watchdog heartbeat, while leaving normal production hosts byte-for-byte unchanged.

An identical classifier _monitor_box_class is defined in both loops (minute.sh:122-135, second.sh:445-458). It resolves to one of three classes with this exact precedence — and this is the code, which supersedes any older .fast.cron.cnf narrative:

.fast.cron.cnf plays no role in monitor cadence. It controls queue-drain speed only, and runner.sh itself ignores it while .slow.cron.cnf is set. An earlier version of _monitor_box_class checked .fast.cron.cnf → NORMAL before the slow test, which un-throttled exactly the tiny boxes the throttle targets — a ≤ 4 GB box carrying both markers (runner.sh auto-creates an immutable .slow.cron.cnf on ≤ 4 GB hosts) ran the full fan-out. The classifier now matches runner.sh's precedence: /root/.force.queue.runner.cnf is the single box-level full-cadence escape. Do not reintroduce .fast.cron.cnf into this function.

The two loops throttle different things:

• minute.sh gates the whole fan-out. The class sets _ITER (passes) and _SLEEP (seconds between them) — minute.sh:152-154 — and the main loop runs _launch_auto_healing _ITER times. So the class directly sets how many times a minute your _launch_auto_healing monitor is spawned.
• second.sh keeps its responsive 10×/5 s loop (load sampling and auto-pause must stay timely) but gates only the heavy fan-out — _proc_control and the hack/escape scanners — to run every _HEAVY_EVERYth pass (second.sh:467-469, the gate at :479). So the class sets how often a _proc_control guard runs, not the load-sampling cadence.

NORMAL reproduces the historical cadence exactly, so a normal production host is unchanged. Each computed default is overridable from /root/.barracuda.cnf — the override must match ^[0-9]+$ (a non-numeric value is ignored and the class default stands) and is then floored at 1 so a misconfiguration cannot disable the loop:

Consequence for a new monitor: register it in _launch_auto_healing and its cadence scales with _ITER; register it in _proc_control and it scales with _HEAVY_EVERY. Neither loop lets a monitor run more often than its launcher's class-gated cadence, so do not assume a fixed 5-second heartbeat when writing a monitor whose job is time-sensitive — read the class the box resolves to. The deployed loadreport prints that class in its header, extracted from the box's own minute.sh _monitor_box_class rather than re-implementing the precedence, so it never drifts from what the box actually does.

Shipping checklist — a new monitor

• In-tree file at aegir/tools/system/monitor/check/<name>.sh, with the _manage_single_lock / lock.inc re-entrancy guard block at the top and the standard export HOME/SHELL/PATH + root check header (copy an existing single-shot monitor).
• Launcher wired: a nohup … & line in _launch_auto_healing (minute.sh) for a per-service healer, or the basename added to the _w list in _proc_control (second.sh) for a keep-alive guard — and to the two _pncw lists in system.sh.inc if it must also fire during a distro-migration reboot.
• _update_agents fetch line in BOA.sh.txt, serial f99, URL under ${_urlHmr}/tools/system/monitor/check/, plus the matching [ ! -e … ] && rm -f ${_pthLog}/<name>.sh.ctrl.*.pid self-heal line.
• Base deps added to _run_aptitude_deps_install (outside the FULL gate, guarded by _pkg_installed) if the monitor needs a non-base binary.
• Editing an existing monitor later: decrement its fNN in _update_agents in the same commit, or installed boxes keep the old file until the next full upgrade. Never increment or re-use a serial.
• Retiring a monitor: add the rm -f /var/xdrago/monitor/check/<name>.sh cleanup, drop the fetch line and every launcher-list entry, and note it in Discontinued features.
• Verify fetch URLs against the repo layout and test a mirror URL with curl -A iCab (the fetch identity every BOA downloader sends).

Related

• Abuse Guard code internals — extending scan_nginx.sh / nginx_guard.sh: the detector pattern, cross-run window state, and the function-local IFS gotcha. scan_nginx.sh inherits its cadence from the nginx_guard.sh launcher documented here.
• Nightly worker architecture — the owl.sh + night/ family, which rides the same _update_agents per-file fetch surface as the monitors.
• The serial & fetch pipeline (SKYNET) — the full two-transport model, _fetch_versioned semantics, and how a tagged release propagates fleet-wide.
• Variables reference and Commands reference — the consolidated _MONITOR_FANOUT_* / _MONITOR_HEAVY_EVERY and related _VAR tables.
• Discontinued features — the retired Perl monitor generation and the upgrade-path cleanup that removes its leftovers.