Nightly maintenance (owl.sh)

owl.sh is BOA's nightly maintenance run — the successor to the daily.sh monolith. The rename came with a decomposition: /var/xdrago/owl.sh is now a thin orchestrator that freezes the run context, walks the Octopus accounts, and dispatches the real work to worker scripts under /var/xdrago/night/. Per-account work runs as a subprocess per account with its own log file, and can optionally fan out in parallel. The default remains serial and behaviour-equivalent to the old inline run.

This page covers the orchestrator: layout, gates, the run-context freeze, the per-account subprocess model, parallelism, and the daily.sh transition. The ghost/empty cleanup family that runs inside the same nightly window is the companion leaf Ghost & empty cleanup; Let's Encrypt renewal and the per-account failure reports the night workers assemble are covered in SSL operations.

Layout: orchestrator + night/ workers

owl.sh aborts with FATAL ERROR if night.inc.sh or 90-global-post.sh cannot be sourced — it never runs half-defined. All five files are serial-gated _fetch_versioned entries in BOA.sh.txt, so every box re-fetches the current copy automatically whenever one changes; you do not pull updates by hand.

Two legacy names are kept by design: the run lock is still /run/daily-fix.pid, and the log directory is still /var/log/boa/daily/.

Launch, gates and lock

The root crontab launches the run at 04:15, throttled below interactive work (aegir/tools/system/cron/crontabs/root):

15 4 * * *  /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/owl.sh  >/dev/null 2>&1

Before doing anything, owl.sh:

• exits 1 if the root filesystem is over 90% full (ERROR: Your disk space is almost full) — nightly maintenance does not run on a nearly-full disk;
• exits 0 immediately if /root/.proxy.cnf or /root/.pause_heavy_tasks_maint.cnf exists — proxy-only nodes and paused boxes run no nightly maintenance at all;
• takes the run lock /run/daily-fix.pid. If the lock already exists, it touches /var/log/boa/wait-for-daily and exits 1; the lock is removed at the very end of the run, after the global post-steps.

To disable the nightly run on a box, the toggle is /var/xdrago/.owl.sh.off — xoct and xcopy park the script there (mv owl.sh .owl.sh.off + pkill) during operations that must not collide with nightly maintenance, and restore it afterwards. xmass does not manage this toggle. A legacy .daily.sh.off left by the pre-rename tooling is carried over to .owl.sh.off during self-update (see the transition section below).

Run order

Inside the lock, one run proceeds as:

1. Global pre-steps — a once-per-upgrade permissions sweep over distro/*/*/sites/all/{libraries,modules,themes} (stamped permissions-fix-post-up-<serial>.info so it reruns only after a BOA upgrade), then hostmaster housekeeping as the aegir user: drush8 cc drush, disabling update/syslog/dblog on @hostmaster, hostmaster cron, two cache-clears, and utf8mb4-convert-databases.
2. _daily_action — logged to /var/log/boa/daily/daily-<NOW>.log: weblogx preparation (when goaccess is enabled), the run-freeze, the per-account loop (below), and the global cleanup family from 90-global-post.sh.
3. Post steps — incident detection, the Nginx forward-secrecy/DH-param refresh (single service nginx reload), global cleanup, log archiving, lock release, and a final INFO: Daily maintenance complete.

The run-context freeze: /run/night/run.env

Because each account now runs in its own subprocess, the run context must be inherited, not re-derived. After the global pre-steps and before the account loop, the orchestrator calls night_emit_run_env (night.inc.sh), which writes the exact per-run state as export lines to /run/night/run.env, chmod 0600:

• _NOW — the run keystone, date +%y%m%d-%H%M%S. It names every *-${_NOW}.info idempotency guard and every per-account log file, so a worker must see the orchestrator's value, never compute its own;
• _DOW, _xSrl (the BOA serial), _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE;
• the module lists: _O_CONTRIB* per core version and the _MODULES_ON_* / _MODULES_OFF_* / _MODULES_FORCE sets;
• the cnf flags a worker acts on: _APT_UPDATE, _PERMISSIONS_FIX, _MODULES_FIX, _CLEAR_BOOST, _ENABLE_GOACCESS, _hostedSys;
• the reporting identity: _ADMIN_EMAIL (from _MY_EMAIL), _INCIDENT_REPORT, _LE_CLIENT_NOTIFY.

A worker loads context with night_load_run_env: it sources /root/.barracuda.cnf first (for cnf flags and anything not in the freeze), then /run/night/run.env — the freeze is authoritative for the frozen per-run set.

Per-account subprocess model

The orchestrator walks /data/disk/ and processes each eligible account — one that has config/server_master/nginx/vhost.d and neither log/proxied.pid nor log/CANCELLED — by running:

bash /var/xdrago/night/10-account.sh <account>  >> /var/log/boa/daily/acct-<user>-<NOW>.log 2>&1

The log path comes from _acct_night_log (night.inc.sh) — derived identically on both sides from the frozen _NOW plus the account basename, so the orchestrator that redirects and the worker that later reads its own log back (to build the Let's Encrypt incident report) always agree on the file name. The orchestrator's own log for the run is /var/log/boa/daily/daily-<NOW>.log.

Before each account, the orchestrator re-samples CPU count and load (_count_cpu + _load_control); an account is only launched below the load ceiling (_O_LOAD_MAX, derived from _CPU_TASK_RATIO — see Load control).

/var/log/boa/daily/ stays bounded without deleting anything: at the end of each run, _archive_old_daily_logs (90-global-post.sh) moves top-level *.log files whose mtime month differs from the current month into MM-YYYY/ subdirectories. The current run's daily-*.log and acct-*.log stay in place, already-archived files are never re-scanned, and nothing is ever deleted — the logs remain available for incident analysis.

Optional parallel fan-out

Two /root/.barracuda.cnf variables control per-account parallelism (lib/settings/barracuda.sh.cnf):

The admission gate differs between the modes in one operator-relevant way:

• Serial (default): an account whose turn arrives while load is over the ceiling is logged (...we have to wait...) and passed over for this run — the historical daily.sh behaviour.
• Parallel: each launch waits in a 5-second retry loop until there is both a free slot (fewer than _NIGHT_MAX_PARALLEL running workers) and load headroom, then fans out. Parallelism therefore does not raise the silent-skip rate — an overloaded moment delays an account instead of dropping it.

After the loop, a parallel run waits for all workers to join before the global post-steps run — 90-global-post.sh touches shared resources (the master /var/aegir tree, /data/all, /etc/ssl/private, one Nginx reload) and must run exactly once, after all per-account work has finished.

Task-queue interplay

The Aegir task-queue runner defers to the nightly run: runner.sh refuses to drain the queue while owl.sh is running (pgrep -fc owl.sh) or while a SQL dump is in flight (mysql_backup.sh, mysql_cluster_backup.sh, or any mydumper process). It touches /var/log/boa/wait-runner.pid, logs Another BOA task is running, we will try again later..., and exits — the queue resumes on a later tick. See Tasks & queue for the runner's full back-off list.

Transition from daily.sh

The rename is designed so that no box loses its nightly run mid-transition (BOA.sh.txt):

• The self-update path installs owl.sh and the four night/ workers via _fetch_versioned.
• A leftover /var/xdrago/.daily.sh.off (a box an operator or xoct/xcopy had disabled under the old name) is converted: owl.sh is parked as .owl.sh.off and the legacy marker removed — a disabled box stays disabled.
• Legacy /var/xdrago/daily.sh and its ctrl stamps are deleted only once the live /var/spool/cron/crontabs/root no longer invokes daily.sh — i.e. after a full system upgrade has re-rendered the crontab to call owl.sh. Until then the deployed crontab still calls daily.sh, so removing it early would silently stop nightly maintenance; gating on the live crontab keeps the retirement self-correcting and idempotent. Whether a given box has already crossed that gate is per-box state — check its live root crontab.

Related

• Ghost & empty cleanup — the opt-in, dry-run-by-default cleanup family (_GHOST_*_CLEANUP, _SHARED_CODEBASES_CLEANUP) that runs inside this nightly window.
• SSL operations — Let's Encrypt renewal and the per-account failure reports assembled from the acct-*.log night logs.
• Tasks & queue — runner.sh and its back-off while owl.sh or a SQL dump runs.
• Load control — _load_control and the _CPU_TASK_RATIO ceiling that gates each account launch.
• Host control files & INI — /root/.proxy.cnf and /root/.pause_heavy_tasks_maint.cnf, the box-level skip gates.
• Reference appendix — _NIGHT_PARALLEL / _NIGHT_MAX_PARALLEL and the other /root/.barracuda.cnf variables.