Process guards & auth scanners

second.sh is one of the two per-minute monitor launchers (the other is minute.sh). Cron starts it once a minute, but it self-loops ten times with sleep 5 to give roughly 5-second monitoring granularity across the whole minute. On each loop it samples system load and reacts to it (the load auto-pause logic); on the heavy loops it also fans out a set of short-lived service guards and security scanners. This page documents that heavy fan-out — the service guard set spawned by _proc_control, the three auth / shell-escape scanners, and the separate Aegir task-queue runner (runner.sh).

Source: aegir/tools/system/second.sh, aegir/tools/system/runner.sh, and the individual guards under aegir/tools/system/monitor/check/. On a running box the launchers live at /var/xdrago/second.sh etc., and the guards at /var/xdrago/monitor/check/.

Where this sits

cron (every minute)
   │
   ├─ second.sh ── 10× loop, sleep 5 ──┐
   │   ├─ _load_control   (every loop, cheap — see Load control)
   │   └─ HEAVY pass (every Nth loop):
   │        ├─ _proc_control ── fan out service guards (monitor/check/*.sh)
   │        ├─ hackcheck.sh    ┐
   │        ├─ hackftp.sh      │ auth / shell-escape scanners
   │        └─ escapecheck.sh  ┘
   │
   ├─ minute.sh ── auto-healing watchdogs (system, mysql, php, nginx,
   │                nginx_guard → scan_nginx, …)   [Auto-healing page]
   │
   └─ runner.sh ── Aegir task-queue runner (load-gated, separate)

The nginx Abuse Guard (scan_nginx.sh) is the security-facing member of this same /var/xdrago/monitor/ machinery, but it is not launched from here. minute.sh spawns nginx_guard.sh, which in turn runs scan_nginx.sh; that whole chain is on the minute.sh side. It is documented in depth in the Abuse Guard topic and is not duplicated on this page.

Why this matters for idle load. On a NORMAL box the heavy pass runs every loop, so _proc_control iterates its 12 service guards (plus the three scanners) roughly every 5 s, each spawned child re-sourcing /root/.barracuda.cnf and running pgrep. Combined with the minute.sh auto-healing fan-out, that spawn churn is the dominant idle-load source on small boxes (load 3–4 on a 2 CPU / 4 GB box with no traffic). The box-class throttle exists to tame exactly this; see Cron cadence & idle-load throttle.

The heavy-pass cadence

second.sh classifies the box once at startup (_monitor_box_class) into CI, SLOW, or NORMAL, and from that derives how often the heavy work runs (_HEAVY_EVERY):

.fast.cron.cnf is not consulted by _monitor_box_class (it only affects runner.sh's queue-drain cadence); only .force.queue.runner.cnf over-rides the slow/RAM heuristic to force NORMAL. This matches minute.sh exactly — both loops share an identical classifier.

The cheap part — _load_control, the load sampling and auto-pause — runs on every loop regardless of class, so responsiveness to a load spike is unchanged. Only the expensive fan-out (_proc_control plus the three scanners) is gated by _HEAVY_EVERY. The cadence is overridable in /root/.barracuda.cnf via _MONITOR_HEAVY_EVERY (a non-numeric or empty value is ignored and the class default stands; otherwise the value replaces the default and is floored at 1). On NORMAL this is the historical every-pass behaviour, unchanged.

_proc_control — service guards

_proc_control is the successor to the legacy proc_num_ctrl.pl. Rather than one monolithic Perl service monitor, it has been split into individual single-shot guards under monitor/check/, and _proc_control simply fans them out, each detached via _spawn_detached (nohup/setsid):

for _w in sendmail_guard convert_guard hostname_sync syslog_legacy \
          bind9 proxysql droplet newrelic_daemon newrelic_sysmond \
          collectd xinetd lsyncd; do
  [ -e "${_monPath}/${_w}.sh" ] && _spawn_detached "bash ${_monPath}/${_w}.sh"
done

Each guard is presence-gated: a guard whose .sh file is not present (not fetched on this box, or deliberately removed) is simply skipped. _proc_control itself is skipped on a heavy pass when load limits were already exceeded (_skip_proc_control, set by _load_control on a MAX/CRIT trip), since the auto-pause is already shedding work. Note that the three scanners below are launched outside that gate, so they still run on every heavy pass even when _proc_control is being skipped. After fanning out, _proc_control touches /var/log/boa/proc_num_ctrl.done.pid as a heartbeat.

Every guard follows the same shape: _check_root, then _manage_single_lock (the shared lock.inc single-instance lock with a legacy pgrep -fc … > 2 fallback — see re-entrancy), then one short action, then exit. They are intentionally single-shot: the every-~5 s cadence comes from second.sh, not from any internal loop.

The guards spawned, and what each one watches:

All twelve rows are characterised from the source under monitor/check/. The nine daemon watchdogs (from syslog_legacy down) share a common shape: they act only when the service's init script is present, so on a box where the daemon is not deployed the watchdog is a no-op (and where the guard .sh file itself was never fetched, the _proc_control loop skips it entirely). The eight pure service-recovery guards (bind9 through lsyncd) additionally bail out early while a BOA upgrade/maintenance run is in progress (_run_to_active: any of the /root/.run-to-*.cnf markers or /run/boa_run.pid), so service recovery never fights an in-flight upgrade. (syslog_legacy, hostname_sync, sendmail_guard and convert_guard do not carry that guard.)

Auth and shell-escape scanners

On every heavy pass, second.sh also launches three log-scanning monitors directly, each detached with nohup. Unlike the nginx Abuse Guard these are single-pattern auth and shell-escape scanners — two ban via CSF, one e-mails an alert:

Shared design notes, verified against the three scripts:

• Cron-race window. All three accept both the current and previous minute so an event logged late in one minute is not missed when the scan runs at the top of the next.
• Dual timestamp formats. hackcheck.sh and hackftp.sh parse both classic syslog and ISO 8601 (Debian 12+ rsyslog) timestamps; escapecheck.sh uses lsh's single fixed YYYY-MM-DD HH:MM:SS,mmm format.
• Incremental reads. hackcheck.sh and hackftp.sh track a byte offset (/var/log/scan_hackcheck_lastpos, /var/log/scan_hackftp_lastpos) and read only bytes appended since the previous run, resetting on rotation/truncation — the same approach as scan_nginx.sh.
• noclobber lock. Each takes a PID-checked noclobber lock (/var/run/{hackcheck,hackftp,escapecheck}.lock) so overlapping cron runs cannot stack.
• Allow-list / maintenance safety. hackcheck.sh never bans an IP present in /etc/csf/csf.allow or /etc/csf/csf.ignore, never bans an IP that had an Accepted login in the window, and — like hackftp.sh — suppresses direct CSF bans while the /var/xdrago/guest-fire.sh maintenance flag is present (enforcement is then handled by guest-fire.sh). Bans are recycled after their TTL (_BAN_SECONDS, 900 s SSH / 3600 s FTP) so an expired IP can be re-armed.

runner.sh — the Aegir task-queue runner

runner.sh is a separate cron job (its own per-minute line), not part of the second.sh/minute.sh fan-out. It is what actually drains the Aegir verify/migrate/backup task queue by executing the /var/xdrago/run-* runners. It is heavily gated so it never adds load on a box that should stay quiet:

• Hard stops first. It exits immediately if /root/.proxy.cnf, /etc/boa/.pause_tasks_maint.cnf, or a max_load/critical_load pid is present, and again if too many runner.sh instances are already running, or a SQL backup, the nightly owl.sh maintenance run, a MySQL restart/cluster-backup, or boa_cron_wait.pid is in flight. That second deferral touches a /var/log/boa/wait-runner.pid marker and logs "Another BOA task is running, we will try again later..." before exiting.
• Load-gated per runner. _runner_action runs a /var/xdrago/run-* runner only while the 1-minute per-CPU load is *below `_CPU_TASK_RATIO 100** (default_CPU_TASK_RATIO=3.1` → 310%); above that it waits. This is the same task ratio used by the load-control logic — backend tasks are skipped under load while the web tier stays up.
• CI hosts (.look.like.jenkins.cnf). On a CI instance there is no automatic queue by default. It runs only if the box is a PRO plan (POWER/PHANTOM/CLUSTER/ULTRA/MONSTER in the octopus control file) or /etc/boa/.allow.aegir.queue.cnf is present, and at least one run-aegir-queue.info exists.
• Small boxes auto-throttle. runner.sh itself writes /root/.slow.cron.cnf and pins it immutable with chattr +i when total RAM ≤ 4096 MB. With .slow.cron.cnf present (and no .force.queue.runner.cnf) it allows only one concurrent runner and runs a single throttled pass per minute with sleep 15 pads.
• Fast / forced. With /root/.fast.cron.cnf or /root/.force.queue.runner.cnf it runs the queue 10 times in the minute (sleep 5 between), mirroring the second.sh cadence.

runner.sh
   │  hard-stop flags? (.proxy / pause_tasks / max|critical load / backup running) ── exit
   ▼
classify cadence by control flag
   ├─ .look.like.jenkins.cnf → queue only if PRO plan / .allow.aegir.queue.cnf
   ├─ .slow.cron.cnf         → 1 throttled pass (RAM ≤ 4 GB auto-sets this flag)
   ├─ .fast/.force.queue     → 10 passes, sleep 5
   └─ default                → single pass
   ▼
_runner_action ── for each /var/xdrago/run-* :  load < _CPU_TASK_RATIO*100 ? run : wait

Disabling the queue does not lower idle load. Marking a box CI with .look.like.jenkins.cnf stops runner.sh draining the Aegir queue, but it does not by itself remove the idle CPU cost — that comes from the second.sh/minute.sh monitor fan-out, which is a separate set of cron jobs. .look.like.jenkins.cnf does also push second.sh/minute.sh into the CI box-class (heavy pass once per minute), but the cheap per-loop sampling still runs. To understand and tune the residual idle load, see Cron cadence & idle-load throttle.

Re-entrancy

Both the launcher (second.sh) and every individual guard protect against overlapping runs with the same _manage_single_lock pattern:

source lock.inc  (/opt/local/bin/lock.inc or /opt/local/lib/lock.inc)
   ├─ shared single-instance lock available?  → _single_instance_lock
   └─ else legacy fallback: pgrep -fc <script> > 2  → log to too.many.log, exit

The shared lock.inc lock is preferred; the legacy pgrep -fc fallback (exit if more than two instances are already running, logging to /var/log/boa/too.many.log) is kept for boxes where lock.inc is not yet deployed. The security scanners use their own PID-checked noclobber lock instead, but to the same end.

Profiler

A small /proc profiler, loadreport (aegir/tools/bin/loadreport, deployed to /opt/local/bin/loadreport), measures which of these scripts is actually consuming CPU/IO during an idle-load investigation. Cron runs it every 30 minutes with --log under nice -n10 ionice -c3, without itself adding measurable load. It is read-only; it never kills or bans. See loadreport.

Related

• Load control & auto-pause — the cheap per-loop half of second.sh: load sampling, spider protection, web auto-pause, and the _skip_proc_control gate that governs this page's heavy pass.
• Cron cadence & idle-load throttle — the box-class fan-out throttle, the _MONITOR_* override variables, and the idle-load model.
• Service auto-healing watchdogs — the minute.sh side of the same machinery: the per-service watchdogs and nginx_guard.sh → scan_nginx.sh.
• Abuse Guard — scan_nginx.sh, the security-facing log scorer, launched on the minute.sh side of this machinery (via nginx_guard.sh).
• Reference appendix — _CPU_TASK_RATIO, _MONITOR_HEAVY_EVERY and the other override variables referenced above.