Nightly worker architecture (owl.sh + night/)

The nightly maintenance monolith that used to be /var/xdrago/daily.sh was renamed to owl.sh and split into an orchestrator plus a worker family under /var/xdrago/night/. The orchestrator keeps the run-level control flow; the workers carry the per-account, per-site and once-per-run bodies. Every body was moved verbatim during the split — behaviour is identical to the pre-split monolith on the serial path — so the split is a maintainability boundary, not a behaviour change.

Deployed layout:

/var/xdrago/owl.sh              orchestrator (was daily.sh)
/var/xdrago/night/night.inc.sh  shared helper library + run-freeze
/var/xdrago/night/10-account.sh per-Octopus-account worker (subprocess)
/var/xdrago/night/20-sites.sh   per-site family + the _daily_process loop
/var/xdrago/night/90-global-post.sh once-per-run global post steps

Source in the fork: aegir/tools/system/owl.sh and aegir/tools/system/night/*. 10-account.sh and 20-sites.sh also define functions only when sourced, so they are unit-testable in isolation; only the orchestrator and 90-global-post.sh carry top-level flow.

owl.sh — what stayed in the orchestrator

owl.sh runs from the root crontab at 04:15 and owns everything that must happen once per run, in order, before and after the account loop:

Prologue and gates. _check_root (ionice/renice self, disk-full abort at

90%), the proxy/pause skips (/root/.proxy.cnf, /root/.pause_heavy_tasks_maint.cnf → exit 0), and the queue wait (while [ -e /run/boa_run.pid ]; do sleep 5; done) all live here.
Run keystone. _NOW=$(date +%y%m%d-%H%M%S) and _DOW=$(date +%u) are computed once. _NOW names every *-${_NOW}.info idempotency guard and every per-account log, so it must be frozen and inherited, never re-derived in a worker (see The run-freeze).
Module lists and force flags. The Tuesday-gated (_DOW=2) _MODULES_ON_*/_MODULES_OFF_* lists, the always-on _MODULES_FORCE set, _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE, and the _O_CONTRIB* shared-tree roots are all resolved in the orchestrator.
Version check + notice. BARRACUDA release fetch and the "upgrade available" mail (gated on _SKYNET_MODE).
Hostmaster pre-work. The @hostmaster drush cache/cron/utf8mb4 block and the permissions-fix-post-up-${_xSrl}.info-gated /data/.../sites/all chmod sweep run once, under the /run/daily-fix.pid re-entrancy lock the orchestrator owns.
The account loop (_daily_action), described below.
Global post (90-global-post.sh bodies), incident detection, forward-secrecy refresh, _global_cleanup, and _archive_old_daily_logs.

The helper library and 90-global-post.sh are sourced with a hard abort if a command -v probe for a known function from each does not resolve — a half-installed night/ set aborts cleanly rather than running with missing bodies:

[ -r "/var/xdrago/night/night.inc.sh" ] && . /var/xdrago/night/night.inc.sh
if ! command -v _run_drush8_hmr_cmd > /dev/null 2>&1; then
  echo "FATAL ERROR: /var/xdrago/night/night.inc.sh not loaded; aborting"
  exit 1
fi

10-account.sh and 20-sites.sh are deliberately not sourced by the orchestrator: the account worker is invoked as a subprocess (it sources its own deps, including 20-sites.sh), and only 90-global-post.sh is sourced-and-called in place because its bodies touch shared/global resources that must never run inside the per-account fan-out.

night.inc.sh — the shared helper library

night.inc.sh is functions and constants only, with no top-level side effects; each caller owns its own PATH/HOME prologue and _check_root. It is sourced by the orchestrator and by every worker, so a single copy is shared instead of duplicated. What it defines:

Tier-0 constants. _WEBG, the _crlGet/_wgetGet fetch flags (iCab UA), the apt flags, and the drush config/variable verb shorthands _cGet/_cSet/_vGet/ _vSet used throughout the per-account and per-site work.
Pure helpers. _validate_safe_dir (realpath must resolve under /data/disk/*, /var/aegir/* or /home/* — the guard that stops a compromised aegir-context alias from redirecting a chown onto system paths), _sanitize_number, _detect_real_docroot (version-agnostic docroot detection: index.php at the root or under docroot/, html/, web/), _cnf_flag_yes (grep-not-source a control flag so load order never clobbers caller scope), _provision_running (the master cleanup interlock — pgrep -f provision), and the consecutive-ghost counter pair _ghost_seen_enough/_ghost_seen_reset.
Load helpers. _count_cpu, _get_load (loadavg-1 over CPU count, ×100), and _load_control (derives _O_LOAD_MAX from _CPU_TASK_RATIO, default 3.1, bumped to 4.1 while .force.sites.verify.cnf is present).
chattr helpers. _enable_chattr/_disable_chattr, which read _HM_U from the caller's scope at call time.
drush8 wrappers. _run_drush8_cmd, _run_drush8_hmr_cmd, _run_drush8_hmr_master_cmd, _run_drush8_nosilent_cmd — all read _HM_U/_Dom from caller scope and prefer the pinned /opt/php74/bin/php where present.
Night log + LE parsing. _acct_night_log, _le_extract_failures, _le_reason (see Per-account LE reporting).
The run-freeze. night_emit_run_env / night_load_run_env.

The run-freeze

Because each account is now a separate subprocess, the per-run state the orchestrator computed must cross the process boundary intact. night_emit_run_env writes it once, after the hostmaster pre-work and before the account loop, into /run/night/run.env (mode 0600) as a block of export lines:

night_emit_run_env    # owl.sh:189, defined night.inc.sh

The frozen set is the keystone _NOW, plus _DOW, _xSrl, _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE, the _O_CONTRIB* roots, all _MODULES_* lists, _hostedSys, _APT_UPDATE, the _PERMISSIONS_FIX/_MODULES_FIX/_CLEAR_BOOST toggles, _ENABLE_GOACCESS, the resolved admin address (_ADMIN_EMAIL, from _MY_EMAIL), _INCIDENT_REPORT, and _LE_CLIENT_NOTIFY.

A worker loads the context with night_load_run_env, which sources /root/.barracuda.cnf first (for cnf flags and any variable not in the freeze), then the freeze second so the frozen per-run set is authoritative for anything it sets:

night_load_run_env() {
  [ -e "/root/.barracuda.cnf" ] && . /root/.barracuda.cnf
  [ -r "/run/night/run.env" ] && . /run/night/run.env
}

The ordering matters: _NOW in particular must come from the freeze, never be re-derived, or a worker's idempotency stamps (*-${_NOW}.info) would key off a different timestamp than the orchestrator expects. _barracuda.cnf-only variables (e.g. the cleanup opt-in flags read directly by _cnf_flag_yes) are not in the freeze and are re-read live in the worker.

The worker contract

An owl.sh night worker is a script under /var/xdrago/night/ with a fixed shape. 10-account.sh and 90-global-post.sh are the two concrete templates:

Prologue. Same export HOME/SHELL/PATH/_tRee/_xSrl block as the orchestrator.
Source deps. night.inc.sh unconditionally (guarded [ -r ... ] && .); a per-account worker additionally sources 20-sites.sh.
Load run context (per-account worker only): night_load_run_env before doing any work, so _NOW and the module lists match the orchestrator's.
Define bodies, then either run them (subprocess/sourced-in-place worker) or leave them defined (a pure library like 20-sites.sh).

10-account.sh — the per-account worker

Invoked by the orchestrator as bash /var/xdrago/night/10-account.sh <account-path>, once per Octopus account — this subprocess is the unit of per-account parallelism. _account_process is the whole per-account sequence: derive _HM_U from the account path; drush prep; sync _MY_OCTO_EMAIL/_CLIENT_EMAIL out of the account .octopus.cnf; the @hostmaster _vSet block (cron intervals, queue frequencies, hosting_delete_force); relocate backups (see Backups relocation); the per-site loop _daily_process (in 20-sites.sh); platform GC (_check_old_empty_platforms, the per-account undo/ reaper); cruft purge (_purge_cruft_machine); hostmaster LE cert copy (_le_hm_ssl_check_update); goaccess; the LE account report (_le_account_report); and the final _enable_chattr relock. The account worker also carries the account-scoped ghost-platform reaper, which moves a ghost alias into <account>/undo/ behind _GHOST_PLATFORMS_CLEANUP (per-account .octopus.cnf override wins over the .barracuda.cnf default).

20-sites.sh — the per-site family

Sourced by 10-account.sh, never run directly. It defines the per-site (per-vhost) procedures and the loop driver _daily_process, which iterates the account's config/server_master/nginx/vhost.d entries, resolves _Dom/_Dir/_Plr from the drush alias, runs _validate_safe_dir on both derived paths (skip the site on a path outside the allowed roots), and calls the per-site fixups (control-file conversion, symlink and permission fixes, module force on/off, per-site LE, the _cleanup_ghost_vhosts/_cleanup_ghost_drushrc reapers). It reads the frozen _NOW/_DOW/_O_CONTRIB*/_MODULES_* context and the _usEr/_HM_U loop vars set by _account_process.

90-global-post.sh — once-per-run global post

Sourced by owl.sh and called in place after the account loop joins. Its bodies touch shared/global resources — the master /var/aegir tree, /data/all, /etc/ssl/private, a single service nginx reload — so they must never run inside the per-account fan-out; they run exactly once, after every account worker has finished. It carries _shared_codebases_cleanup, _ghost_codebases_cleanup, _check_old_empty_hostmaster_platforms, _incident_detection/_incident_email_report, _fix_nginx_forward_secrecy, _global_cleanup, _purge_shared_aegir_backups, and _archive_old_daily_logs. The /run/daily-fix.pid lock stays owned by the orchestrator and is not released here.

Per-account subprocess execution

_daily_action in owl.sh drives the account loop. Each account directory under /data/disk/ is a candidate; an account is processed only when it has a config/server_master/nginx/vhost.d tree and neither log/proxied.pid nor log/CANCELLED is present. For each eligible account the orchestrator re-samples CPU and load, then hands the account to the worker subprocess:

bash /var/xdrago/night/10-account.sh "${_usEr}" \
  >> "$(_acct_night_log "${_usEr}")" 2>&1

Every account's stdout/stderr is redirected to its own log, /var/log/boa/daily/acct-<user>-<NOW>.log, derived identically by orchestrator and worker from the frozen _NOW, so both sides agree on the filename and the worker can read its own log back to build the LE report. The orchestrator's own top-level output goes to /var/log/boa/daily/daily-<NOW>.log; 90-global-post.sh's incident scan reads that global log, while the per-account LE parsing reads the per-account log.

The run model: serial and parallel

The account loop has two modes, admission-gated identically on the per-core load ceiling (_O_LOAD < _O_LOAD_MAX, from _CPU_TASK_RATIO):

Serial (default, _NIGHT_PARALLEL=NO). One account subprocess at a time, called synchronously; behaviour-equivalent to the former inline call. If load is over the ceiling the loop logs and waits (re-sampling next iteration).
Parallel (_NIGHT_PARALLEL=YES). Accounts fan out concurrently up to _NIGHT_MAX_PARALLEL slots (default = CPU core count, sanitized via _sanitize_number; floor 1). Before each fan-out the loop spins on a combined gate — a free job slot and load headroom:

while :; do
  _count_cpu
  _load_control
  if [ "$(jobs -rp | wc -l)" -lt "${_NIGHT_MAX_PARALLEL}" ] \
    && (( $(echo "${_O_LOAD} < ${_O_LOAD_MAX}" | bc -l) )); then
    break
  fi
  sleep 5
done

The parallel path waits for headroom rather than skipping an overloaded account, so raising parallelism does not raise the silent-skip rate. Each worker writes to its own account log, so concurrent output never interleaves. After the loop, the orchestrator waits for all backgrounded workers before running the global-post step — this join is what guarantees 90-global-post.sh's shared-resource bodies never race an in-flight account worker.

_NIGHT_PARALLEL and _NIGHT_MAX_PARALLEL are the only two variables introduced by the split; both live in /root/.barracuda.cnf (defaults seeded idempotently on install/upgrade). Leaving _NIGHT_MAX_PARALLEL empty defaults it to the core count.

Backups relocation and the queue-hold interlock

_relocate_backups_to_static_fs (in 10-account.sh) moves an account's backups/ and backup-exports/ onto the static/files filesystem behind symlinks, so large dereferenced backups cannot fill the root partition. It is a deliberate no-op unless static/files resolves to a different device than the account root (the case for large accounts moved to attached storage), and it self-skips on either kill-switch (/data/conf/disable_backups_on_static_fs.cnf box-wide, or the per-account static/control/no_backups_on_static_fs.info).

The interesting path is the one-time migration of an existing real directory. It runs inside the parallel fan-out, so it must serialise against sibling account workers and must not let a new Aegir task start mid-move. Two interlocks stack:

A real flock on /run/.boa_backups_relocate.flock (flock -n, non-blocking; the kernel releases it on process death, so there is no stale-lock guessing). Only one account worker performs a migration at a time.
The task-queue hold, /run/boa_queue_stop.pid. The worker writes its own PID into the stop file only if it is absent, records that it created it (_madeStop), drains any in-flight provision task (bounded, ~60 s via _provision_running), does the rsync --remove-source-files move, then removes the stop file only if it created it and still owns the recorded PID:

[ -e "${_stop}" ] || { echo "$$" > "${_stop}" && _madeStop=YES; }
# ... drain, relocate ...
[ "${_madeStop}" = "YES" ] && [ "$(cat "${_stop}")" = "$$" ] && rm -f "${_stop}"

runner.sh (the frequent task-queue agent) honours /run/boa_queue_stop.pid two ways — a parent-level exit 0 when it is present, and a per-child skip inside its loop — so no new queue task starts while the move is in flight; the _provision_running drain handles any task already running. The stop file is self-healing: clear.sh purges a leaked one after checking its PID is dead, and /run is cleared on reboot, so a crashed relocation can never permanently freeze the queue. This same /run/boa_queue_stop.pid is the shared hold the filesystem migration tools (migratefs, updatesymlinks) use, so the night relocation is a cooperative participant in an existing interlock, not a new mechanism.

Per-account LE reporting

_le_account_report (in 10-account.sh) runs at the end of each account's sequence and builds the Let's Encrypt renewal report from that account's own log only — which is what guarantees a client notice can never be cross-attributed to another account. _le_extract_failures (in night.inc.sh) parses the dehydrated failure blocks and emits one record per failure with fields joined by the 0x1f control byte (not a tab — the alt-names field is empty for single-domain certs and a whitespace read IFS would collapse the empty middle field, shifting every later field left). _le_reason maps an ACME error type/detail to a plain-language cause.

The operator (_ADMIN_EMAIL) gets the full per-account report every night, gated by _INCIDENT_REPORT (normalized to OFF/ALL/MINI/CRIT in owl.sh; legacy NO→OFF, YES→MINI), plus a catch-all for non-validation backend/ACME errors so the move to per-account logs does not lose the old combined-log coverage. The affected client (_CLIENT_EMAIL) gets an actionable notice, throttled to once per 7 days per failing site via a marker under <account>/log/ctrl/, ON unless disabled with _LE_CLIENT_NOTIFY=NO (per-account .octopus.cnf override wins; only the literal NO disables). The client notice carries a Reply-To of the account owner (fallback: server admin) so replies do not bounce off the root@<host> envelope sender.

Deploy and fetch surfaces for the night/ family

The self-update path in BOA.sh.txt provisions the whole family. It creates the directory (mkdir -p /var/xdrago/night), then fetches each file with the serial-gated fetcher:

_fetch_versioned /var/xdrago/night/night.inc.sh  .../tools/system/night/night.inc.sh  fNN
_fetch_versioned /var/xdrago/night/20-sites.sh   .../tools/system/night/20-sites.sh    fNN
_fetch_versioned /var/xdrago/night/10-account.sh .../tools/system/night/10-account.sh  fNN
_fetch_versioned /var/xdrago/night/90-global-post.sh .../tools/system/night/90-global-post.sh fNN
_fetch_versioned /var/xdrago/owl.sh              .../tools/system/owl.sh               fNN

Each file carries its own fNN fetch serial, so a maintainer bumps only the serial(s) of the file(s) actually changed and only those boxes re-fetch. Serials count down (bump = decrement) and each is single-homed in BOA.sh.txt; do not cite a specific serial in a living doc — state the gating rule (each night file is serial-gated independently) instead. The self-update path also drops stale *.ctrl.*.pid stamps for any night file whose deployed copy is missing, so a partial install re-fetches cleanly.

Two migration guards make the daily.sh → owl.sh rename safe on a live box:

Disable state carries over. A box that had the nightly run disabled has the script renamed aside to /var/xdrago/.daily.sh.off (the same rename-aside disable the migration tools use, not a marker file). When .daily.sh.off is present, the self-update path renames the freshly fetched owl.sh aside to /var/xdrago/.owl.sh.off and removes .daily.sh.off, so the box stays disabled under the new name. The xoct/xcopy/xmass enable/disable of the nightly run now toggles between /var/xdrago/owl.sh and /var/xdrago/.owl.sh.off.
Legacy daily.sh retired only when the crontab has moved. daily.sh and its ctrl stamps are removed only once the live root crontab no longer invokes daily.sh (i.e. a full system upgrade has re-rendered the crontab to call owl.sh). Until then the deployed crontab still calls daily.sh, so deleting it on the self-update path would silently stop the nightly run mid-transition. Gating on the live crontab keeps this self-correcting and idempotent.

The root crontab entry is 15 4 * * * ... bash /var/xdrago/owl.sh. Note that weblogx self-skips while owl.sh is running (a pgrep -fc /var/xdrago/owl.sh guard), preserving the old mutual-exclusion the monolith had.

Adding a night worker

To add a new once-per-run global step, put the body in 90-global-post.sh and call it from owl.sh after the account loop — nothing that touches the shared /var/aegir or /data/all trees, /etc/ssl/private, or issues a service reload may run inside the per-account fan-out. To add per-account work, add the function to 10-account.sh and call it from _account_process; to add per-site work, add it to 20-sites.sh and call it from _daily_process. Rules the split enforces:

No top-level side effects in a sourced library. night.inc.sh and 20-sites.sh are sourced by multiple callers; a top-level statement would run in every one. Define functions only; let the caller invoke them.
Read the run context from the freeze, not the environment. Any per-run value a worker needs must be in night_emit_run_env's block, or it will be empty in the subprocess. If a new step needs a new per-run value, add its export line to the freeze (orchestrator side) and it appears automatically via night_load_run_env. Values that live only in .barracuda.cnf and are stable across the run can be read live with _cnf_flag_yes instead of freezing them.
Gate anything shared/global behind the join. A body that mutates a shared path belongs in 90-global-post.sh, which runs after wait. A body that races sibling account workers on a shared resource (like the backups relocation) must carry its own flock and, if it must quiesce the task queue, the /run/boa_queue_stop.pid hold with the create-and-own discipline shown above.
Bump the file's own fetch serial in BOA.sh.txt in the same change, so boxes actually re-fetch the modified worker.
Keep bodies verbatim when moving them. The split's invariant is behavioural identity with the monolith; a move that also changes logic breaks the ability to bisect a regression to "the split" vs "a later edit".

Monitor deploy surfaces — how the /var/xdrago monitor and worker binaries are fetched and serial-gated during self-update.
Abuse Guard internals — the real-time monitor family that shares the same /run pidfile and _fetch_versioned deploy conventions.
Monitor and abuse internals — the chapter overview for this topic.
Discontinued features — the retired Drupalgeddon daily hacked-site detector and its now-defunct control files, removed from this run.