Load control & auto-pause

_load_control is BOA's per-host auto-pause safety net: a self-healing watchdog that measures the system load average every ~5 seconds and, when load stays dangerously high, throttles crawlers, pauses the web stack, and kills runaway processes — then restores everything automatically once load drops. It lives in second.sh (deployed at /var/xdrago/second.sh, source aegir/tools/system/second.sh) and is the load-shedding counterpart to the security-facing Abuse Guard, which is a different member of the same /var/xdrago/monitor/ machinery.

Its single design goal: a misbehaving site, a crawler storm, or a stuck Drush job should degrade gracefully — shed load and recover — rather than take the whole box down so hard that even SSH stalls.

Where it runs

second.sh is launched once per minute by the root crontab (aegir/tools/system/cron/crontabs/root) but self-loops 10 times with a sleep 5 between passes, so the load test fires roughly every 5 seconds across the whole minute rather than once a minute.

cron: * * * * *  bash /var/xdrago/second.sh
   │
   ▼
re-entrancy guard (lock.inc shared lock; legacy `pgrep -fc` fallback)
   │
   ▼
for _iteration in 1..10:        ← the 10× / 5 s loop
   ├─ _load_control            ← EVERY pass (cheap load sampling)
   ├─ heavy fan-out            ← only every Nth pass (see Cadence)
   │     _proc_control + hackcheck/hackftp/escapecheck
   └─ sleep 5

The re-entrancy guard is the shared _manage_single_lock pattern: it sources lock.inc and takes the shared single-instance lock if present, otherwise falls back to a legacy pgrep -fc count and exits if more than 2 copies of the script are already running (logged to /var/log/boa/too.many.log).

_load_control runs on EVERY pass of the loop, on every box, regardless of box class. The cadence throttle only reduces how often the heavy fan-out (_proc_control, the hack/escape scanners — see process guards) runs on small / idle / CI hosts. It never slows load sampling. Auto-pause stays fully responsive — a ~5 s reaction window — even on a throttled CI or SLOW box.

A standalone load-policy CLI, loadguard, is installed at /opt/local/bin/loadguard (symlinked into /usr/local/bin) with defaults matching the fleet and the verbs enforce | ok-tasks | ok-spiders | run-tasks — but it is not wired into any shipped crontab. It is manual-only; second.sh remains the live enforcement path. Do not treat loadguard as active enforcement.

The load metric

_get_load reads /proc/loadavg, takes the 1-minute and 5-minute load averages, and normalises each to a per-CPU percentage:

_O_LOAD = (loadavg_1min / nproc) * 100      # 1-minute, per-CPU %
_F_LOAD = (loadavg_5min / nproc) * 100      # 5-minute, per-CPU %

Normalising by nproc is what makes one set of thresholds valid across a 2-core VM and a 64-core bare-metal host: a per-CPU load of 100% means the run queue equals the core count regardless of how many cores there are. A box sitting at _O_LOAD=410% is running its 1-minute queue at ~4.1× its core count.

Both the 1-minute and 5-minute figures are tested. The 1-minute value reacts to a sudden burst; the 5-minute value catches a sustained grind that a fast burst-and-release would miss.

The four thresholds

Four per-CPU load ratios define the escalation ladder. Each is a ratio relative to a single CPU; _load_control multiplies it by 100 to get the percentage threshold compared against _O_LOAD / _F_LOAD.

All four are overridable in /root/.barracuda.cnf. They are sanitised on load (_sanitize_number strips anything but digits and a decimal point), so a malformed override falls back to the built-in default rather than breaking the arithmetic.

The defaults are deliberately well above 100%. A BOA box is expected to run its cores hot under normal traffic; auto-pause is a last resort for genuinely pathological load, not a load balancer. The drastic MAX/CRIT tiers carry extra headroom because the linear ratio × nproc model is right for CPU-runnable load but mis-scales I/O-wait load, which does not grow with core count. On a 2-core box, SPIDER trips when the 1-minute load average reaches ~4.2, MAX at ~12.2, CRIT at ~16.2 — by which point the box is effectively unresponsive and shedding load is strictly better than letting it thrash.

The TASK ratio (3.1) is the threshold above which backend tasks are skipped. second.sh does not read _CPU_TASK_RATIO at all — the variable lives in /root/.barracuda.cnf and is consumed by the queue scripts themselves (runner.sh, owl.sh, usage.sh, manage_solr_config.sh, purge_binlogs.sh — see process guards). The three tiers _load_control itself acts on are SPIDER, MAX and CRIT.

The escalation ladder

_load_control is a single if/elif chain evaluated highest-tier first, so only one action fires per pass. Crucially, every tier is re-verified after a sleep 9 cooldown before it acts: a one-off spike that has already passed by the time the cooldown elapses is ignored. Only sustained load triggers a response.

measure _O_LOAD (1m) and _F_LOAD (5m), per-CPU %
  │
  ▼
CRIT — _O_LOAD or _F_LOAD > 810%?
  │ yes ─► sleep 9, re-check ─► still high:
  │          _terminate_processes  (killall -9 php drush.php wget curl)
  │          then _hold_services   (stop nginx + php-fpm)
  │          (kill/pause skipped if a backup is running AND load is
  │           I/O-wait-bound — see below; the marker is still written)
  │          touch /run/critical_load.pid
  │ no
  ▼
MAX — _O_LOAD or _F_LOAD > 610%?
  │ yes ─► sleep 9, re-check ─► still high:
  │          _hold_services        (stop nginx + php-fpm)
  │          (pause skipped if a backup is running AND load is
  │           I/O-wait-bound — see below; the marker is still written)
  │          touch /run/max_load.pid
  │ no
  ▼
SPIDER — _O_LOAD or _F_LOAD > 210% and ≤ 610%?
  │ yes ─► sleep 9, re-check ─► still high:
  │          _nginx_high_load_on   (enable nginx_high_load.conf, block crawlers)
  │          touch /run/spider_load.pid
  │          _clear_pause_latch    (resume hysteresis — see NORMAL)
  │ no
  ▼
NORMAL — touch /run/normal_load.pid
         _clear_pause_latch: drop /run/max_load.pid + /run/critical_load.pid only
           once BOTH loads are below the resume threshold (MAX × _RESUME_FRACTION)
         if spider protection on and both loads ≤ 210%: _nginx_high_load_off

SPIDER — block crawlers

When per-CPU load sits above 210% but at or below 610% (checked on both the 1-minute and 5-minute figures), _nginx_high_load_on renames /data/conf/nginx_high_load_off.conf to /data/conf/nginx_high_load.conf and reloads nginx. That file is glob-included by aegir/conf/nginx/nginx_compact_include.conf (include /data/conf/nginx_high_load.c*;), so swapping the suffix toggles crawler blocking without rewriting any vhost. The web stack stays fully up for real users; only spiders are shed. This is the gentlest acting tier and the only one that does not set _skip_proc_control, so the heavy fan-out (_proc_control and the hack/escape scanners) still runs on its normal cadence. The MAX and CRIT tiers set _skip_proc_control=true, which suppresses only _proc_control for that pass (the hack/escape scanners are unaffected).

MAX — pause the web stack

Above 610%, _hold_services stops the entire web tier: service nginx stop, then force-quit on every installed php<NN>-fpm init script, then a belt-and-braces killall php-fpm and killall nginx. The box stops serving requests entirely so the run queue can drain. An ALERT-level incident is logged and (subject to policy below) emailed.

CRIT — terminate runaways, then pause

Above 810%, _terminate_processes runs first: killall -9 php drush.php wget curl. This targets the usual culprits of a runaway — a stuck PHP request, a Drush job in a loop, a wget/curl pulling something huge — before _hold_services pauses the web tier. Killing the runaway first is what lets the box actually recover instead of immediately re-spiking after the pause.

Backup exemption — MAX/CRIT skip only disk-bound backup load

Both the MAX and the CRIT branches consult two gates after the re-check: _backup_in_progress and _load_is_iowait_bound. Only when a backup is running and the load is measurably I/O-wait-bound do they log the high load and take no action — neither pausing the web tier nor killing processes. A backup's high load is expected, largely I/O-wait (D-state from dump/duplicity disk churn) and self-limiting, so pausing nginx/PHP-FPM would cut service for no benefit on a disk-bound job.

_load_is_iowait_bound samples /proc/stat twice, 0.2 s apart, and requires the system iowait share of the delta to reach at least _LOAD_IOWAIT_MIN% (default 10, sanitised by _sanitize_number, overridable in /root/.barracuda.cnf). The consequences:

• A CPU-bound runaway coincident with a backup window still gets acted on. CPU-bound load has low iowait, so pause/kill fire normally — the exemption never shields a genuine runaway just because a backup happens to be running.
• Fails closed. Any /proc read error or degenerate delta yields 0% iowait — below the threshold — so the drastic tiers still act on a failed measurement.
• The skip is logged to /var/log/boa/high.load.incident.log with the measured iowait%, e.g. Max load N% (1-minute) during a running backup, I/O-wait-bound (iowait X% >= 10%) - NOT pausing web.

The backup gate matches tightly: the BOA backup orchestrators by path (backboa, duobackboa, multiback, mysql_backup.sh, mysql_cluster_backup.sh), the dump engines mydumper/duplicity by exact name, or the /run/boa_sql_cluster_backup.pid marker — so a genuine non-backup overload is never mistaken for a backup. (The SPIDER tier has no such exemption; crawler shedding stays active during a backup.)

NORMAL — recovery, with resume hysteresis

When load is at or below the spider threshold on both the 1-minute and 5-minute figures, the box is marked healthy and, if spider protection is currently on, _nginx_high_load_off renames the file back and reloads. Recovery is automatic — there is no manual "un-pause" step. (_hold_services stops services rather than disabling them, so the service watchdogs bring nginx and PHP-FPM back; the spider config is the only piece _load_control explicitly reverts.)

Web resume is latched, not instant. The service watchdogs restart web only when both /run/max_load.pid and /run/critical_load.pid are absent, and _clear_pause_latch — called from the SPIDER and NORMAL branches — removes those markers only once load has fallen below the resume threshold = MAX threshold × _RESUME_FRACTION (default 0.8, so 488% with the default 6.1 MAX ratio) on both the 1-minute and 5-minute averages. Pause trips when either average exceeds a threshold; resume needs both below the lower bar. That asymmetry is a deliberate hysteresis gap: without it a box hovering at the MAX threshold would flap pause/resume. The per-pass report header prints the computed Web Resume Threshold alongside the three tier thresholds.

_RESUME_FRACTION is sanitised and clamped to the open interval (0,1) — anything outside resets to 0.8, so a bad override can neither pin the latch closed (a fraction of 0 would keep web paused forever) nor disable the hysteresis (≥ 1). The latch also fails toward resume: an empty or unusable threshold clears the markers rather than leaving web stuck paused.

State files in /run

Each pass writes its tier marker under /run and clears the stale ones, so the current load posture is readable from the marker(s) present. Several service watchdogs read these to decide whether to stand down.

The two pause markers are the hysteresis latch: a SPIDER or NORMAL pass removes max_load.pid / critical_load.pid only once both load averages are below the resume threshold (see NORMAL — recovery), so during the recovery window a spider or normal marker can coexist with a still-held pause marker.

boa_second_auto_healing.pid is a short-lived guard set by _hold_services for the duration of a pause and removed when it finishes. While it exists, the MAX/CRIT branches skip re-issuing _hold_services, so two overlapping passes can't stack service stops on top of each other.

To read the live posture:

ls -1 /run/*_load.pid 2>/dev/null
tail -n 20 /var/log/boa/high.load.incident.log

Process priority — _B_NICE

second.sh renices itself (and the _proc_control pass) to _B_NICE. The value is sanitised to an integer and clamped to the valid nice range -20..19; anything out-of-range or non-numeric falls back to 0. Set _B_NICE in /root/.barracuda.cnf to bias the whole monitor pass softer or harder against site traffic. The clamp matters because the renice runs unconditionally on a script that fans out helpers — an unbounded value would either be rejected by the kernel or starve the very watchdog meant to be relieving load.

Incident reporting

When a pause or kill fires, _incident_email_report appends to the incident log and may email an alert. Two gates control whether mail is actually sent.

Policy gate — _INCIDENT_REPORT. Normalised to upper-case and matched against three current levels, with legacy spellings mapped in:

Legacy values are remapped: NO → OFF, YES → CRIT, MINI → CRIT. Anything unrecognised also falls back to CRIT, so a typo fails safe to "critical alerts only" rather than to silence or to noise.

What "default" means here. second.sh's in-code default for _INCIDENT_REPORT is CRIT. The shipped /root/.barracuda.cnf does not leave it unset — it sets _INCIDENT_REPORT=MINI, and MINI normalises to CRIT for second.sh. Either way the effective behaviour out of the box is CRIT: you receive only ALERT-level mail from the load-control path. (The same MINI setting means something stricter on the per-service watchdogs — see auto-healing — so do not assume one _INCIDENT_REPORT value behaves identically across every monitor.)

This CRIT ladder is specific to second.sh. The per-service auto-healing watchdogs read the same _INCIDENT_REPORT variable but apply it with different per-script defaults and a stricter send gate (most mail only on ALL); see Service auto-healing watchdogs. The two tables describe different scripts, not conflicting rules.

Source note. The current case maps both YES and MINI to CRIT. The explanatory comment block just above it still describes the older intent (YES → MINI, with MINI as a distinct "most important alerts" level); there is no separate MINI runtime level any more — the code collapses it into CRIT. Trust the case, not the comment.

The pause/kill actions call _incident_email_report at level ALERT, so they are delivered under the default CRIT policy. The spider on/off events are coded at INFO and their email calls are commented out — they are logged but never mailed, to avoid alert spam from routine crawler shedding.

Uptime grace gate — _check_uptime_grace_period. Sourced from lock.inc, this is the very first thing _incident_email_report checks. It returns failure (suppressing the email and forcing _INCIDENT_REPORT=OFF) when any of these hold:

• system uptime is under 15 minutes;
• the Hostmaster alias /var/aegir/.drush/hm.alias.drushrc.php is missing, or csf / lfd aren't up yet;
• an install / upgrade is mid-flight (/run/octopus_install_run.pid, /run/boa_run.pid, /run/boa_wait.pid).

This is what stops a fresh boot or an in-progress BOA upgrade — both of which legitimately spike load — from flooding the mailbox with auto-pause alerts before the box has settled.

Destination — _MY_EMAIL (default notify@omega8.cc). Mail goes there via s-nail; if _MY_EMAIL is empty, _incident_email_report returns without sending. The body is the last 200 lines of the incident log at /var/log/boa/high.load.incident.log, so the current incident is not buried under history; the full log stays on disk and rotates weekly (see auto-healing). The log records every action (Web Server Paused, PHP/Wget/cURL terminated, spider protection enabled/disabled) with a timestamp and the offending load percentage regardless of whether mail was sent.

Tuning summary

All live in /root/.barracuda.cnf. Raise the ratios on a box that is expected to run hot; lower _CPU_SPIDER_RATIO on a box where crawler load is the main threat. Do not lower the ratios near or below 100% (1.0) — the box would auto-pause under ordinary traffic.

Related

• Cron cadence & idle-load throttle — the 10×/5 s loop, box classification (CI / SLOW / NORMAL), and the heavy fan-out throttle that load sampling is deliberately exempt from.
• Process guards & auth scanners — _proc_control and the task-queue runner gated by _skip_proc_control and _CPU_TASK_RATIO.
• Service auto-healing watchdogs — the watchdogs that read the /run/*_load.pid markers this page writes, and the per-service _INCIDENT_REPORT gate.
• Abuse Guard — the security-facing monitor in the same /var/xdrago/monitor/ family; it sheds malicious request load where _load_control sheds aggregate system load.
• Reference appendix — every override variable named above.