Monitor & abuse-guard internals

The runtime tier of BOA: the scripts that live under /var/xdrago/ on a deployed box and run on cron, not the installer that put them there. Everything in this topic is Barracuda-side bash in boa-private (aegir/tools/system/, mirrored to /var/xdrago/ at build time) plus the one PHP boundary that guards the backend user — the Drush extension deny-filter in the omega8cc Drush 8 fork. It is deliberately not the operator's how-to: those pages read live state and cite defaults; these pages are for the maintainer changing the script. If you touch a monitor/check/*.sh, the ban pipeline, owl.sh / the night/ workers, or boa_extension_filter.inc, this is the execution model your change lands in.

Two clocks and one privilege boundary organise the whole tier:

  • the per-minute clock — second.sh + minute.sh, each a one-line crontab entry, fan a set of monitor/check/*.sh watchdogs out several times a minute;
  • the nightly clock — one crontab entry (owl.sh) drives a per-account / per-site maintenance sweep that drops privilege to each tenant's user;
  • the trust boundary — every one of these runs as root or as aegir/oN, reads tenant-writable paths and tenant-supplied Drush aliases, and must never let tenant input escalate. The abuse-guard, the path validators and the extension filter all sit on that line.

The deployed tier in one pass

Nothing here is loaded from the build tree at runtime. _xdrago_install_upgrade() (lib/functions/system.sh.inc:9152) cp -afs the entire aegir/tools/system/* tree into /var/xdrago/ and installs cron/crontabs/root into /var/spool/cron/crontabs/root (system.sh.inc:9179, :9203, :9240) — so on a box every path below is /var/xdrago/..., and the crontab is the single source of truth for what runs when.

/var/spool/cron/crontabs/root  (from aegir/tools/system/cron/crontabs/root)
   │
   ├─ * * * * *  second.sh ──▶ _proc_control fan-out (12 watchdogs)   [1 deploy surfaces]
   │                         └▶ hackcheck.sh · hackftp.sh · escapecheck.sh (SSH/FTP IDS)
   ├─ * * * * *  minute.sh  ──▶ _launch_auto_healing × _ITER passes:
   │                            system unbound valkey|redis mysql php
   │                            fpm_tune nginx nginx_guard java
   │                                            │
   │                                            └▶ nginx_guard.sh ──▶ scan_nginx.sh   [2 abuse-guard]
   ├─ * * * * *  guest-fire.sh   web.log ─▶ csf -td 900                    [2 ban pipeline]
   ├─ 01 5 …     guest-water.sh  archive ─▶ csf.deny (_NR_TEST ≥12/≥24)    [2 ban pipeline]
   ├─ */2 …      ip_access.sh · user_admin_access.sh · ai_policy.sh ·
   │             nginx_deny.sh · nginx_deny6.sh                            [2 / 4]
   ├─ */3 …      manage_ltd_users.sh   (per-user FPM pool, jailed shells)  [4 security model]
   ├─ * * * * *  runner.sh   (Aegir task queue drain)
   └─ 15 4 …     owl.sh ──▶ night/night.inc.sh + 90-global-post.sh         [3 nightly worker]
                     └▶ per account: night/10-account.sh ──▶ night/20-sites.sh
                          (su -s /bin/bash - <HM_U>  → drush8 @<site>)     [3 / 4]
  • [1] Deploy surfaces. second.sh and minute.sh are the fan-out engines. minute.sh classifies the box (_monitor_box_class → CI/SLOW/NORMAL) and loops _launch_auto_healing _ITER times spaced _SLEEP seconds apart (default NORMAL = 9×5 s, minute.sh); second.sh runs _proc_control, the twelve service watchdogs split out of the legacy proc_num_ctrl.pl, plus the SSH/FTP IDS scanners. There are 27 monitor/check/*.sh on disk; only the dispatched subset actually runs each tick.
  • [2] Abuse-guard. nginx_guard.sh → scan_nginx.sh is the web IDS; the detection→ban→enforce pipeline (scan_nginx → web.log → guest-fire.sh → guest-water.sh → nginx_deny.sh → the $is_banned geo) is the same mechanism the operator guide documents, re-read here at code level: the run model, the detector functions and the escalation arithmetic.
  • [3] Nightly worker. owl.sh is a single-threaded (optionally _NIGHT_PARALLEL) orchestrator that sources night/night.inc.sh (the shared Tier-0 helper library) and night/90-global-post.sh, then invokes night/10-account.sh as a subprocess per /data/disk/* account, which in turn drives night/20-sites.sh per site. The split exists so one copy of the drush8 wrappers and chattr helpers is shared, not duplicated.
  • [4] Security model. The cross-cutting boundary: the anti-lockout allow logic in ip_access.sh, the _validate_safe_dir path gate the night workers run before any tenant-derived chown/chmod, the per-user FPM disable_functions injection in manage_ltd_users.sh, the chattr +i immutability locks, and the one PHP boundary — boa_extension_filter.inc, which stops tenant-dropped *.drush.inc from auto-loading as a privileged backend identity.

Each leaf below zooms into one of these boxes.

Pages in this topic

  • Monitor deploy surfaces (second/minute fan-out) — how the watchdog tier is delivered and driven. The deploy half: _xdrago_install_upgrade() (system.sh.inc:9152) mirroring aegir/tools/system/* → /var/xdrago/ and installing the root crontab (:9179,:9203,:9240), the _spawn_detached helper (system.sh.inc:3191) and the boot-time monitor kick from the setup path (system.sh.inc:3238-3245, :7804-7811). The drive half: minute.sh's _monitor_box_class (CI/SLOW/NORMAL, RAM≤4096 → SLOW unless .force.queue.runner.cnf, .look.like.jenkins.cnf → CI) and its _ITER/_SLEEP fan-out cadence (overridable via _MONITOR_FANOUT_ITER/_MONITOR_FANOUT_SLEEP), the _launch_auto_healing dispatch set (system unbound valkey|redis mysql php fpm_tune nginx nginx_guard java), the re-entrancy guard (_manage_single_lock via lock.inc, with the legacy pgrep -fc >2 fallback), and the _second_flood_guard / _csf_flood_guard self-protection kills. The second.sh half: the _proc_control twelve-watchdog list and the hackcheck/hackftp/escapecheck IDS launch, both gated on the heavy-cadence _skip_proc_control load check. The invariant a change must respect: the build tree is never executed at runtime — a fix to any monitor/check/*.sh only reaches a box through a rebuild that re-runs _xdrago_install_upgrade.

  • Abuse-guard code internals — the web IDS at function level: the maintainer's cut of the same detection→ban→enforce mechanism, read at code level rather than as operator how-to. nginx_guard.sh's load-pause branch (10 passes/min on the normal path vs one bounded scan_nginx pass during a max_load/critical_load pause); scan_nginx.sh's incremental byte-offset window (/var/log/scan_nginx_lastpos, baseline _NGINX_DOS_LINES, default 1999 — the tail-scan window, a distinct variable from the per-IP threshold _NGINX_DOS_LIMIT, default 399; never write them glued as "399/1999"); the real-IP resolution (last realip/XFF token only, IPv4 or IPv6 — a v6 client is only reachable via the trusted realip proxy, scored and banned via the nginx-native path, gated by _NGINX_V6_BAN_DETECT), the laundering-proof _is_ignored_request exemption gate, and the detector functions (_process_ip, _track_ua_ip/_handle_ddos_blocking, _track_path_flood, _handle_http10_auth_flood, _track_ua_burst, plus the Tier-B alert-only _handle_i18n_flood / _check_fpm_saturation). The ban side: _block_ip → web.log/scan_nginx.archive.log, guest-fire.sh applying csf -td 900, and guest-water.sh's archive escalation — the recidivist counter is _NR_TEST over the web archive _WA (guest-water.sh:639): ≥12 re-bans (:652), ≥24 writes the permanent csf -d … do not delete Brute force Web Server … rule (:666-671). (The :596/:609/:623-628 SSH twin over _HA is the identical mechanism for port 22, not the web tier this page documents.) This "24" is a guest-water threshold, not a csf.conf setting (stock LF_REPEATBLOCK='0'). Also the function-IFS gotcha every space-split helper in scan_nginx.sh must handle (global IFS=$'\n\t').

  • Nightly worker architecture (owl + night/) — the maintenance-sweep engine. owl.sh's preconditions and load-gated account loop (/root/.proxy.cnf, .pause_heavy_tasks_maint.cnf, the /run/boa_run.pid wait, _load_control per iteration), the _NIGHT_PARALLEL/_NIGHT_MAX_PARALLEL fan-out that waits for a free slot + load headroom rather than skipping an overloaded account (so parallelism does not raise the silent-skip rate), and the source split: night.inc.sh is a functions-only library (no top-level side effects — callers own their PATH/HOME/_check_root prologue) sourced by both owl.sh and each subprocess worker; 10-account.sh is invoked per-account and sources its own deps; 20-sites.sh does the per-site work. The contracts that carry state: the drush8 wrappers (_run_drush8_cmd/_run_drush8_hmr_cmd) that su -s /bin/bash - ${_HM_U} -c into the tenant user and read _HM_U/_Dom from the caller's scope at call time; the _enable_chattr/_disable_chattr immutability toggles; the _detect_real_docroot structure detection (index.php-keyed, follows the symlink so a web -> /etc decoy can't register); and _normalize_incident_report (OFF/ALL/MINI/CRIT, legacy NO→OFF/YES→MINI). Why the worker family carries its own copies of these helpers instead of sourcing the install-time helper.sh.inc.

  • Security-model internals — the trust boundary the whole tier sits on, at code level. The privilege map (which script runs as root vs aegir/oN, and where each su -s /bin/bash - boundary is); the anti-lockout logic in ip_access.sh (127.0.0.1 + the box's own IPv4 + ::1 + every currently-established SSH peer (IPv4 or IPv6) read from netstat always allowed, per-token IPv4/IPv6/CIDR validation so one bad control-file line can't fail the whole nginx -t, fail-closed skip of malformed records); the _validate_safe_dir gate (night.inc.sh:41, realpath -e + a /data/disk/*|/var/aegir/*|/home/* prefix whitelist) that stops a tenant who rewrites a Drush alias from steering the nightly chown/chmod at /etc (20-sites.sh:1913,1917); the chattr +i locks on platforms/ and .drush/; the per-user FPM disable_functions injection driven by _PHP_FPM_DENY (read only from /root/.barracuda.cnf and the per-Octopus /root/.<user>.octopus.cnf, applied per pool in manage_ltd_users.sh:1951 — there is no per-platform or per-site INI row for it); and the one PHP defence, boa_extension_filter.inc — default-deny *.drush.inc loading from tenant-writable paths, but only for privileged backend identities (boa_drush_extension_backend_identity()), with the realpath-before-match fix, the deny-pattern set and its carve-outs, and the per-Octopus opt-in control files (/data/conf/<oN>_civicrm.txt, <oN>_elysia_cron.txt) plus the global kill switch /data/conf/drush_extension_filter_disabled.txt (boa_drush_extension_filter_disabled()). The filter is loaded from preflight.inc:129 on every Drush run, but its behaviour turned over two Drush 8 bumps: it was a silent no-op until the realpath-before-match fix (Drush 8.5.3, shipped in BOA-5.10.3) made it actually deny — and in that build it filtered every invocation, over-blocking oN.ftp client shells; the boa_drush_extension_backend_identity() scoping that limits it to the aegir master and Octopus oN backend identities landed in the next bump (Drush 8.5.4). The leaf carries the version-by-version behaviour table.

Adjacent developing topics

  • Install & staged-setup internals — how aegir/tools/system/* and the crontab get onto a box: the staged AegirSetup* engine and the _xdrago_install_upgrade() path in system.sh.inc that this topic's deploy surfaces assume already ran.
  • Web & search stack internals — the Provision Nginx templates (server.tpl.php edge maps, vhost_include.tpl.php guard chain) that emit the 444s and access.log lines scan_nginx scores, and the $is_banned geo the ban pipeline feeds back into.
  • Aegir backend APIs — the Drush 8 fork whose preflight.inc loads the extension filter, and the Provision/Hosting backend the nightly drush8 @<site> calls drive.
  • Build & test — verifying a change to any of these scripts: a rebuild is the only path that re-runs _xdrago_install_upgrade and lands the edit on a box.

Reference

  • Variables and Commands — the consolidated _VAR and CLI tables, including the _NGINX_* abuse-guard knobs, _MONITOR_FANOUT_*, _NIGHT_*, _INCIDENT_REPORT, _PHP_FPM_DENY and the csf verbs cited throughout this topic.
  • Discontinued features — retired monitor scripts and superseded ban/escalation mechanics land there; check it before documenting anything found only in old trees, CHANGELOG.txt history, or the legacy docs/ctrl/*.ctrl reference files (several of those markers — e.g. .no.sql.cpu.limit.cnf, .ip.protected.vhost.whitelist.cnf — have zero live readers and must not be presented as current).