The serial & fetch pipeline (SKYNET)

This page is the maintainer's contract for how a merged commit physically reaches every live box. There are exactly two transports, and every change you ship rides one of them:

1. Per-file, serial-gated fetches — _fetch_versioned in BOA.sh.txt, driven by the SKYNET self-update tick. Covers aegir/tools/bin/*, aegir/tools/system/* monitors, and the privileged fix-drupal-* helpers. Shipping a change here means decrementing an fNN serial — in both fetch sites if the tool has two.
2. Whole-release rides — lib/functions/*.sh.inc, lib/settings/*.sh.cnf, Nginx/PHP config templates, and everything else in the build tree. No per-file serial exists; the code lands when a box runs a full barracuda up-* pass against the release. Their "serial" is the release identity itself (_xSrl / _X_VERSION).

Pick the wrong transport model and the failure is silent: the commit merges, the mirror serves it, and no box ever installs it.

The heartbeat: clear.sh → BOA.sh.txt → autoupboa

The self-update tick is a root cron line, not a daemon (aegir/tools/system/cron/crontabs/root):

*/5 * * * * bash /var/xdrago/clear.sh >/dev/null 2>&1

Each pass, clear.sh prunes stale pidfiles, exits immediately on proxy-only boxes (/root/.proxy.cnf), and — only when no installer run is in flight (/run/boa_run.pid absent) — re-executes the meta installer straight off the mirror, then the auto-upgrade scheduler:

wget -qO- https://${_USE_MIR}/versions/${_tRee}/boa/BOA.sh.txt | bash
bash /opt/local/bin/autoupboa

BOA.sh.txt is therefore never stale on a box — it is streamed fresh from the mirror every tick, and everything below hangs off whatever identity block that fresh copy carries. Inside it, the dispatch at the bottom of the script:

• _if_boa_key_tools_update_allowed — parks the whole update path while a distro migration marker (/root/.run-to-<codename>.cnf) is present.
• _SKYNET_MODE=OFF → _if_update_boa_key_tools_only runs _update_boa_tools (key tools still refresh) and exit 0. This is the documented "critically important tools are still updated" floor.
• SKYNET active → _boa_setup (mirror health check, _update_boa_tools when its _bTs gate is stale), then — only on an installed box (/var/log/barracuda_log.txt exists) — the _fix_* series, Drupal core patch checks (_if_drupal_patches_update, _fix_drupal_core_ten / _fix_drupal_core_eleven) and _update_agents.

Separately, the nightly owl.sh (04:15 cron) fetches ${_urlHmr}/conf/version/barracuda-release.txt (shipped from aegir/conf/version/) and mails _MY_OCTO_EMAIL when /var/log/barracuda_log.txt does not yet contain that release string — the "new release available" notice. It runs only when _SKYNET_MODE is unset or ON.

The identity block

The first exported lines of BOA.sh.txt are the release identity every other mechanism keys off:

export _tRee=dev              # tree: dev | pro | lts (per published branch copy)
export _xSrl=<NNNN><tree>TNN  # whole-release serial — in EVERY ctrl stamp name
export _rLsn="BOA-X.Y.Z"      # release label
export _bTs=<NNNN>vNN         # _update_boa_tools function serial (counts UP)

(Concrete values churn every release — read them from BOA.sh.txt at HEAD; NNNN squashes the version, e.g. X.Y.Z → XYZ-style digits.)

• _tRee selects the mirror tree: all serial-gated fetches resolve against _urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir". The value is baked into each branch's published copy of the installer, so a box follows whichever tree's BOA.sh.txt its clear.sh streams.
• _xSrl appears in every control stamp (<name>.ctrl.<serial>.${_tRee}.${_xSrl}.pid). Bumping it at a release invalidates all stamps at once — the first tick after a release refetches the entire serial-gated tool set exactly once. One-shot migration blocks (e.g. cv-phar-symlink.ctrl.${_tRee}.${_xSrl}.pid) and the dragon backup names in lib/functions/ use the same key, which is why whole-release code needs no fNN of its own.
• _rLsn feeds _rlsE="${_rLsn}-${_tRee}"; the barracuda/octopus/ boa wrappers pin it into the fetched settings file as _X_VERSION=${_rlsE} (see the release-ride section below). The same values are duplicated at the top of the wrappers themselves — a release commit updates all of them together.
• _bTs gates _update_boa_tools as a function: _boaToolsPid="${_pthLog}/updateBOAtools.${_bTs}.ctrl.${_tRee}.${_xSrl}.pid". The function only runs when that stamp is missing (or xcopy is missing, or the box has no Hostmaster alias yet); on completion all updateBOAtools*.pid stamps are wiped and the current one is touched. Any behavioural change to _update_boa_tools must bump _bTs in the same commit or installed boxes with a current stamp never re-enter the function. Unlike fNN, _bTs counts UP: the vNN counter increments within a release; the release bump resets the NNNN prefix. (The same up-counting vNN shape gates other one-shot ctrl stamps, e.g. the /etc/crontab rewrite markers in autoupboa — same rule: bump means increment, never re-use.)

All stamps live in _pthLog=/var/log/boa (a one-time migration block at the top of BOA.sh.txt moved them from the legacy /var/xdrago/log).

_fetch_versioned — the per-file transport

One helper replaces every hand-rolled fetch block (defined in BOA.sh.txt, above _update_agents):

_fetch_versioned <dest> <url> <serial> \
  [--mode 700] [--owner root:root] [--symlink PATH] [--guard PATTERN]

Semantics, in order:

1. Up-to-date test. Return 0 when the stamp ${_pthLog}/<basename>.ctrl.<serial>.${_tRee}.${_xSrl}.pid exists and the destination is non-empty (-s) and, when --symlink was given, the symlink is intact. Missing file or broken link with a present stamp = self-heal refetch.
2. Run guard. --guard PATTERN does a pgrep -fc; a running match skips the fetch entirely (no stamp written) — a live tool is never clobbered, and the fetch retries next tick.
3. Fetch with rollback. The old copy moves to <dest>.old, then curl ${_crlGet} "${_url}" -o "${_dst}". A non-empty result gets chmod/chown (defaults 700 root:root), the optional symlink, all old <basename>.ctrl.* stamps removed, the new stamp touched, and the .old backup dropped. An empty/failed result restores <dest>.old and writes no stamp — mirror lag or a transient failure retries on the next 5-minute tick instead of being skipped forever.

Retry is two-layered: _crlGet itself carries --retry 9 --retry-delay 9 --fail (so a non-2xx yields an empty destination, triggering the rollback path), and the stamp-only-on-success rule makes the cron tick the outer retry loop.

This replaced two pre-5.10.3 mechanisms. The /opt/local/bin bulk curl loop (~30 tools) tested -e instead of -s, so a 0-byte --fail download could overwrite a good tool and delete its .prev backup, and it had no retry. Separately, the six privileged helpers (fix-drupal-*-permissions/ownership.sh, lock-local-drush-permissions.sh, fix-drupal-site-symlinks.sh) sat behind one updateFx31.ctrl one-shot marker that stamped even on a failed fetch, so a partial pull was never retried. Both are now per-file _fetch_versioned calls (/opt/local/bin group and privileged helpers alike); the old updateFx*.ctrl markers are unused and harmless. Two residual hand-rolled blocks remain in _update_agents (manage_ltd_users.sh, manage_solr_config.sh) — same stamp shape, inline logic; treat their serials identically.

fNN serials count DOWN — and are bumped in BOTH fetch sites

The in-code rule (comment above the /opt/local/bin group in _update_boa_tools): "bump a script's serial (counts DOWN, f99->f98...) to force a refresh after editing it."

• Bump = decrement. f99 → f98 → … → f01. Concrete values churn every release — read them from BOA.sh.txt at HEAD, never from docs.
• Never re-use, never increment. The mechanism itself is direction-free (_fetch_versioned wipes old stamps on every successful fetch, per its header comment), but only for boxes that actually completed the previous fetch. A box that skipped an intermediate serial — offline, mirror lag, guard always busy — may still hold a stamp bearing the older, higher value. "Bump" a serial back up to a previously-used value and that box matches its stale stamp and silently skips the refetch. Monotonic decrement makes stamp collision impossible.
• Shared tools have two fetch sites and ONE stamp. The core tool group (barracuda, boa, octopus, autoupboa, xcopy, xoct, the synproxy family, ~30 more) is fetched by both _update_agents and _update_boa_tools, against the same <name>.ctrl.<serial>.<tree>.<xSrl>.pid stamp. The serials must be changed to the same value in both call sites in the same commit. If they diverge, each function sees the other's stamp as stale, wipes it, and refetches — every tool in the pair flip-flops on every 5-minute tick, fleet-wide. Both _update_agents sub-groups that overlap (autoupboa/fixmounts, the synproxy family) carry an explicit "bump serials in both functions together" comment at the call site.
• A serial mistake is corrected by decrementing again — in both sites, in one commit — never by putting an earlier value back. The direction rule has been enforced this way in past release trains; treat any correction the same as a normal bump.

_update_agents vs _update_boa_tools — duplication is load-bearing

Do not "DRY up" these two functions. The header comment in _update_boa_tools states the contract:

The first group is ALSO fetched by _update_agents. That duplication is INTENTIONAL and must stay: _update_boa_tools runs ALWAYS — including before BOA is installed, and when _SKYNET_MODE=OFF makes the run update key tools and exit 0 BEFORE _update_agents is ever reached — so it must independently fetch the core tools; _update_agents runs only when Skynet is active. The two share one <name>.ctrl.<serial> stamp, so each serial here MUST stay in sync with the same tool in _update_agents — a divergent serial makes each run wipe the other's stamp and refetch every cycle. Bump both together; do NOT try to merge the two functions.

Whole-release rides: lib/functions and the build tree

lib/functions/*.sh.inc carry no fNN and are never fetched per-file. They reach a box only inside a full installer run:

1. The barracuda wrapper (itself a serial-gated tool) maps up-lts/up-pro/up-dev to _tRee and sets _rgUrl="https://files.boa.io/versions/${_tRee}/boa", _rlsE, _bRnh (_set_tree_vars). _proceed then fetches BARRACUDA.sh.txt and lib/settings/barracuda.sh.cnf fresh into /var/backups/ and calls _up_action, which seds the release pins into the fetched settings file: _X_VERSION=${_rlsE}, _BRANCH_PRN=5.x-${_tRee}, _AEGIR_VERSION=${_tRee}. boa and octopus do the equivalent for their chains.
2. BARRACUDA.sh.txt sources the fetched barracuda.sh.cnf, then _download_helpers_libs stages the whole build tree at _bldPth=/opt/tmp/boa: _DL_MODE=BATCH (default) pulls boa.tar.gz from https://<mirror>/dev/<tree>/ via _get_dev_ext (10 attempts, 9 s apart); _DL_MODE=GIT clones ${_BOA_REPO_GIT_URL}/boa.git at ${_BRANCH_BOA} instead.
3. The include list is fixed: _FL="helper dns system sql valkey redis nginx php solr master xtra firewall hotfix", each sourced from ${_bldPth}/lib/functions/ with _panic_exit on a missing file.
4. _xdrago_install_upgrade (lib/functions/system.sh.inc) then wholesale copies ${_bldPth}/aegir/tools/system/* over /var/xdrago/, after snapshotting the old tree and root crontab to ${_vBs}/dragon/…-pre-${_xSrl}-${_X_VERSION}-${_NOW}.

Consequence for shipping: a monitor or system script in the fNN set updates both ways (serial fetch within minutes, wholesale copy at the next full upgrade); a lib/functions change updates only at the next full barracuda up-* — there is no faster path, so never put an urgent fleet fix solely in lib/functions if a serial-gated tool can carry it.

The _*_VRN block at the top of BARRACUDA.sh.txt (_CURL_VRN, _NGINX_VRN, _CSF_VRN, _GIT_VRN, _MSS_VRN, …) is the software-pin layer of the same ride: the pin is defined there, but the build step that consumes it lives in the sourced lib/functions/*.sh.inc, and its build/rebuild stamp embeds the pin value together with _xSrl/_X_VERSION (e.g. the MySecureShell build in lib/functions/system.sh.inc gates on ${_pthLog}/mss-build-${_MSS_VRN}-${_xSrl}-${_X_VERSION}.log). Changing a pin therefore re-triggers exactly that component's build on the next full run.

Mirrors and the _crlGet convention

Every fetch in the pipeline goes through one flag set, defined near the top of BOA.sh.txt and duplicated verbatim in the barracuda/boa/octopus wrappers and the BARRACUDA.sh.txt/OCTOPUS.sh.txt meta-installers (the lib/settings/*.sh.cnf files inherit it from the sourcing *.sh.txt, they do not redefine it):

_crlGet="-L --max-redirs 3 -s --fail --retry 9 --retry-delay 9 -A iCab"
_wgetGet="--max-redirect=3 -q --tries=9 --wait=9 --user-agent='iCab'"

--fail is load-bearing (non-2xx → empty destination → _fetch_versioned rollback instead of an HTML error page saved as a tool). -A iCab is the fetch identity every BOA downloader sends; the mirror cluster is operated around that convention, so when hand-testing a mirror URL, test with curl -A iCab — a default UA is not a representative probe.

Mirror selection: _find_fast_mirror_early seeds /var/backups/boa-mirrors-2026-05.txt with the current pool — files.boa.io (default/fallback), files.o8.io, files.host8.biz — and runs ffmirror to pick the fastest; the winner becomes _USE_MIR. The list filename is date-stamped, so shipping a new pool means a new filename (old boxes regenerate rather than merge). Two URL bases derive from it:

• _urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir" — per-tree tool and config fetches (paths mirror the repo's aegir/ subtree).
• _urlDev="https://${_USE_MIR}/dev" — source tarballs and boa.tar.gz.

For the pro and dev trees, _verify_boa_keys additionally checks a per-hostname key (enc/2024/<md5(hostname)> on the mirror) before proceeding; lts has no license gate.

How a tagged release propagates

Tags exist only on 5.x-pro and 5.x-lts; 5.x-dev (with its -base / -edge feeders) is consumed untagged by dev-tree boxes. The mirror cluster serves each tree's installers, tools and boa.tar.gz under versions/<tree>/ and dev/<tree>/; tagging a release is what promotes the new file set into the pro/lts trees (mirror-side publication — the repo carries the content, the tag marks it releasable). From the box's point of view, everything after that is the mechanics above:

1. Within one 5-minute tick, clear.sh streams the new BOA.sh.txt. Its new identity block does the rest: a bumped _bTs re-runs _update_boa_tools; a bumped _xSrl invalidates every ctrl stamp, so _update_agents refetches the full tool/monitor set once; individually decremented fNN serials refetch just their files.
2. The same tick runs autoupboa, which maintains the scheduled full-upgrade cron entries (barracuda up-${_AUTO_VER} … / octopus up-${_AUTO_VER} … in /etc/crontab) from the _AUTO_UP_* variables, and the weekly wrapper mode (autoupboa weekly-system up-<tier> system … noscreen). Two populations, one mechanism: on hosted-fleet boxes (_if_hosted_sys — /root/.host8.cnf present or hostname ending .aegir.cc) autoupboa hard-sets the schedule in-code, with the full-upgrade date parked on _AUTO_UP_MONTH=2 / _AUTO_UP_DAY=29 (Feb 29 — a leap-day-only slot) while the weekly system upgrade stays live; release ops edit the shipped autoupboa to open a real fleet window and re-park it afterwards. On operator boxes the same variables are read from /root/.barracuda.cnf — no month+day set means _crontab_cleanup strips the full-upgrade lines. The crontab rewrite is itself a one-shot per ${_tRee}.${_xSrl} (an up-counting vNN ctrl stamp), reset when the /root/.turn.off.auto.update.cnf lock toggles; dev-server boxes (/root/.dev.server.cnf) never get auto-upgrade lines.
3. Nightly, owl.sh compares barracuda-release.txt against /var/log/barracuda_log.txt and mails the operator until the box has been upgraded.
4. The full stack upgrade — new lib/functions, rebuilt services, /var/xdrago wholesale refresh — happens when barracuda up-lts|up-pro actually runs: manually, or at the scheduled autoupboa window.

So: tools and monitors converge fleet-wide in minutes; the deep stack converges at the next full run. Design any fix's rollout around that split.

Shipping checklist (serial mechanics only)

• Tool under aegir/tools/bin/ or aegir/tools/system/ changed → decrement its fNN in BOA.sh.txt, in both fetch sites when it is in the shared group (grep the tool name; the call sites carry matching-serial comments). Same commit as the change.
• Never increment or re-use an fNN value; never copy a serial from another tool.
• _update_boa_tools body changed → bump _bTs (<NNNN>vNN, up) in the same commit.
• lib/functions/*.sh.inc / lib/settings/*.sh.cnf / template change → no serial exists; it rides the next full barracuda up-*. If the fix is urgent fleet-wide, carry it (also) in a serial-gated tool.
• New tool → add a _fetch_versioned line starting at f99, plus the chmod group and symlink block in _update_boa_tools if it is operator-facing.
• Verify fetch URLs against the repo layout (_urlHmr maps to the aegir/ subtree) and test with curl -A iCab.

Related

• Code style & conventions — the lint gates and commit rules every serial-bumping commit must also pass.
• Contributing — how changes reach the public branches this pipeline publishes from.
• Reference appendix — consolidated _VAR tables, including the _AUTO_UP_* scheduler variables.