Hostmaster upgrade orchestrator (AegirUpgrade)

aegir/scripts/AegirUpgrade.sh.txt (521 lines at current HEAD) is the fifth staged script: where AegirSetupM.sh.txt installs the Master Aegir frontend, AegirUpgrade.sh.txt upgrades it. It runs as the aegir user via su -s /bin/bash - aegir out of the root-side _aegir_master_upgrade() (master.sh.inc:752-924), refreshes Drush and the Provision backend under /var/aegir/.drush, then moves the frontend to the next numbered root host_master/NNN with drush hostmaster-migrate.

One myth to clear before reading the code: despite the function names, there is no dispatch and no major/minor branch. The script defines four local functions, pulls a fifth from master.sh.inc, and calls all five unconditionally, in a fixed order, inside a single settings.php existence guard (AegirUpgrade.sh.txt:496-504). Every run of the orchestrator is the full pipeline.

Invocation from the barracuda upgrade flow

Transport first: the barracuda wrapper fetches BARRACUDA.sh.txt and lib/settings/barracuda.sh.cnf into /var/backups/ from _rgUrl="https://files.boa.io/versions/${_tRee}/boa" (barracuda:1834-1835, :1968), and the staged build tree at _bldPth=/opt/tmp/boa — which is where AegirUpgrade.sh.txt executes from — arrives as part of boa.tar.gz (or the git clone in _DL_MODE=GIT). The orchestrator rides the whole-release transport: it has no per-file fNN serial, so a fix here reaches boxes at their next barracuda up-* run, not on the SKYNET tick (see the serial & fetch pipeline).

The wrapper's second positional argument pre-arms two mode toggles by sed-ing them into both /var/backups/barracuda.sh.cnf and /root/.barracuda.cnf:

• barracuda up-<tree> system → _SYSTEM_UP_ONLY=YES when a frontend exists (hm.alias.drushrc.php + nginx/subdir.d witnesses, barracuda:1024-1037) — host stack only, frontend upgrade skipped.
• barracuda up-<tree> aegir → _AEGIR_UPGRADE_ONLY=YES (barracuda:1057-1069) — frontend only.

Inside BARRACUDA.sh.txt, _STATUS=UPGRADE is derived by _mode_detection() from the existence of /var/aegir/.drush/hm.alias.drushrc.php (helper.sh.inc:472-481). The path to the orchestrator then runs through these gates:

Past all gates: _php_cli_cron_update → _aegir_master_upgrade → _php_cli_cron_update (master.sh.inc:1405-1407).

Root-side prep: _aegir_master_upgrade()

Everything that needs root happens before the su (master.sh.inc:752-924), gated on _prompt_yes_no "Do you want to upgrade Ægir Master Instance?" (auto-yes when _AUTOPILOT=YES, helper.sh.inc:23-25):

1. mysql_upgrade -u root --force on Percona 5.7 only (:759-764); ProxySQL detection via /var/aegir/use_proxysql.txt → 127.0.0.1:6033 (:765-770).
2. Sudoers wiring for the aegir user: nginx init script plus the six fix-drupal-*/lock-local-drush-permissions helper scripts (:772-784).
3. _hostmaster_dr_up (:786) — see the quirk note in step 2 of the pipeline below.
4. Backend presence repair: if /var/aegir/.drush/sys/provision/http or /var/aegir/drush/includes is missing, wipe .drush/{sys,xts,usr} and re-clone Provision at ${_BRANCH_PRN} from ${_gitHub}/provision.git (:788-794). This is what the in-script preflight later re-checks.
5. Parse _THIS_HM_ROOT/_THIS_HM_SITE out of /var/aegir/.drush/hm.alias.drushrc.php (:795-804; note this is the root-side hm.* alias — the aegir-side script later parses hostmaster.alias.drushrc.php instead), create the sites/all skeleton, normalize ownership, chattr -i the immutable .drush/php.ini so the upgrade may replace it (:805-820).
6. _php_cli_drush_update rewrites the #! shebang of drush.php/drush to the configured /opt/phpNN/bin/php (:822-825, function at :626-653).
7. State-bus append (:830-849) — see below.
8. mysqladmin flush-hosts, syncpass fix aegir, _find_correct_ip, _provision_backend_dbpass_sync, _hostmaster_frontend_dbpass_sync (:851-857).
9. _master_download_for_local_build (:858, function at :547) — stages /opt/tmp/make_local with the Hostmaster, Hosting and Drupal core sources the migrate will build from (make_local.tar.gz in BATCH mode; per-repo git clone --branch 5.x-${_tRee} clones plus _get_dev_stc contrib tarballs otherwise).
10. _mrun "webserver up" (:860-861), then the launch:

AegirUpgrade="${_bldPth}/aegir/scripts/AegirUpgrade.sh.txt"
su -s /bin/bash - aegir -c "/bin/bash ${AegirUpgrade} aegir"
wait

(master.sh.inc:864-866). The positional aegir argument is not consumed — unlike AegirSetup{A,B,C}, which key their settings-snapshot suffix on $1, AegirUpgrade always sources the unsuffixed barracuda.sh.cnf; the argument is symmetry with the setup scripts, nothing more.

After the wait: if /opt/tmp/status-AegirUpgrade-FAIL exists → FATAL ERROR + touch /opt/tmp/status-Barracuda-FAIL + _clean_pid_exit (:869-873). On success: chattr +i the php.ini back, re-run _hostmaster_frontend_dbpass_sync (:874-879), refresh the /data/conf/global.inc symlink, rotate Valkey/Redis passwords, run _force_advanced_nginx_config, reset the master nginx/pre.d overrides, and sweep permissions across /var/aegir (:880-918).

The state bus across the su

su - wipes the environment, so /var/backups/barracuda.sh.cnf is the only variable channel into the orchestrator — the same contract as the staged setup engine. The append block at master.sh.inc:830-849 writes exactly these keys: _AEGIR_VERSION, _AEGIR_XTS_VRN (= ${_tRee}), _BOA_REPO_GIT_URL, _BOA_REPO_NAME, _BRANCH_BOA, _BRANCH_PRN, _DEBUG_MODE, _DL_MODE, _DOMAIN (= ${_THIS_FRONT}, the Master frontend domain), _DRUPAL7, _DRUSH_VERSION, _NOW (set/exported by the wrapper at barracuda:10-11), _PHP_CLI_VERSION, _PHP_FPM_VERSION, _SMALLCORE7_V, _STRONG_PASSWORDS, _THIS_DB_HOST, _USE_MIR, _xSrl, _X_VERSION. A variable the aegir-side script needs and that is not in this list does not exist past the su — add an echo line here, not an export in the parent.

Bootstrap inside the script

The prologue mirrors the staged-setup convention (AegirUpgrade.sh.txt:41-77): pins _bldPth="/opt/tmp/boa", _filIncB="barracuda.sh.cnf", _libFnc="${_bldPth}/lib/functions", _vBs="/var/backups", export _tRee=dev (re-pinned per published tree copy). _panic_exit() (:51-60) echoes EXIT: lines, touches /opt/tmp/status-AegirUpgrade-FAIL and exits 1; it guards both the settings source (:66-67) and the include loop _FL="helper master" (:73-77). This is the Master-side pairing — satellite.sh.inc is never loaded here, and the master/satellite libraries are deliberately parallel copies: fixes are mirrored by hand, never DRY'd.

Local settings (:83-109): _THIS_DB_HOST=FQDN resolves to the box hostname; the PHP CLI ladder tests exactly 8.4 → 8.5 → 8.3 against _PHP_CLI_VERSION and the matching /opt/phpNN/bin/php (:86-95). There is no else and no fallback — same short ladder as AegirSetupM, unlike the full /opt/php{85..56} walk in AegirSetup{A,B,C}. An unsupported _PHP_CLI_VERSION leaves _T_CLI empty, _DRUSHCMD degenerates, and the run dies at the drush gate (step 4 below). Then: _ROOT="${HOME}" (= /var/aegir under the su), _DRUSHCMD="${_T_CLI}/php ${_ROOT}/drush/drush.php", PATH pinned with ${_T_CLI} first, _crlGet/_wgetGet with the mandatory iCab UA, _gCb="git clone --branch", and the two exported mirror bases _urlDev="https://${_USE_MIR}/dev" / _urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir" (:108-109; this path only fetches through _urlDev — _urlHmr is exported for the include-side helpers, per the shared prologue shape).

Picking the target root: host_master/NNN and the zombie sweep

The Master frontend lives in numbered generation roots /var/aegir/host_master/NNN (a 000 placeholder generation is seeded with placeholder_dont_remove.txt, :392-399). The body (:388-495) resolves four variables before the pipeline runs:

• _PREV_HM_ROOT — the live root. Seeded from the legacy flat layout (find ${_ROOT} -maxdepth 1 -type d | grep hostmaster, :390) and overwritten with host_master/${_LASTNUM} when the numbered layout exists.
• _LASTNUM — last existing generation: cd host_master; _list=([0-9]*), take the last glob entry (:402-405).
• _DISTRO — the next generation number, zero-padded to three digits via _nextnum=00$((10#…)); _nextnum=${_nextnum: -3}. The arithmetic is forked on the bash version (:406-411): under bash 5.1/5.2 an extra 0 is interpolated (10#0${_last%%[^0-9]*}) so the 10# base prefix never sees an empty operand — $((10#)) hard-errors there when the glob is unmatched and the suffix-strip yields an empty string.
• _HM_ROOT="${_ROOT}/host_master/${_DISTRO}" — the migrate target (:478). It must not pre-exist (:479-485).

The liveness witness for a generation is sites/${_DOMAIN}/settings.php. A last generation without it is a zombie — a dead root left by a failed prior run — and gets moved to ${_ROOT}/backups/system/empty-hm-${_LASTNUM}-${_NOW} before a rescan (:423-475; a second zombie gets the -sec suffix). The sweep runs at most twice; if the computed _HM_ROOT still exists afterwards the script aborts: FATAL ERROR: Too many zombies to delete! Try again... + FAIL marker + exit 1 (:479-485). Zombie detection, and an old-generation tree detected via ${_PREV_HM_ROOT}/modules/path_alias_cache, both force _DEBUG_MODE=YES for the rest of the run (:420-424, :449).

Preflight (:487-495): host_master is created 0711, and /var/aegir/.drush/sys/provision/http must exist (the root-side prep at master.sh.inc:788-794 repairs it if missing) — else FATAL + marker + exit. Finally the single guard at :496: ${_PREV_HM_ROOT}/sites/${_DOMAIN}/settings.php present → run the pipeline (:500-504); absent → FATAL + marker + exit (:505-512).

The five-step pipeline

_hostmaster_mv_up      # AegirUpgrade.sh.txt:121
_hostmaster_dr_up      # master.sh.inc:723 (the only include-side step)
_hostmaster_go_up      # AegirUpgrade.sh.txt:156
_hostmaster_dr_tt      # AegirUpgrade.sh.txt:222
_hostmaster_mi_up      # AegirUpgrade.sh.txt:235

The order is load-bearing: archive the old backend before fetching the new one, install Drush before testing it, test Drush before migrating with it.

1. _hostmaster_mv_up — archive the old backend extensions (:121-154)

Moves the Drush/Provision backend extension directories aside — it does not touch the hostmaster platform directory. For each of the 16 names in _D_EXT (provision clean_missing_modules drupalgeddon drush_ecl make_local provision_boost provision_cdn provision_civicrm provision_site_backup provision_tasks_extra registry_rebuild remote_import safe_cache_form_clear security_check security_review utf8mb4_convert) it checks all four layout locations — ${_ROOT}/.drush/$e, .drush/xts/$e, .drush/usr/$e, .drush/sys/$e — and moves hits to ${_ROOT}/backups/system/$e-pre-${_DISTRO}-${_NOW}. If you add a new backend extension to the fetch set, add its name to _D_EXT too, or the previous copy stays live in .drush next to the new one.

2. _hostmaster_dr_up — Drush swap (master.sh.inc:723-748)

Moves ${_ROOT}/drush to backups/system/drush-pre-${_DISTRO}-${_NOW}, then prefers the system-wide copy: if /opt/tools/drush/8/drush/drush.php exists (staged earlier in the run by _drush_system_install_update, system.sh.inc:4042), cp -af it into place; otherwise fetch drush-${_DRUSH_VERSION}.tar.gz from the mirror and fix permissions (_DRUSH_VERSION="${_DRUSH_EIGHT_VRN}", BARRACUDA.sh.txt:158).

Quirk to know before refactoring: this function is called twice per upgrade — root-side at master.sh.inc:786 and aegir-side at AegirUpgrade.sh.txt:501. The aegir-side call is the operative one (_ROOT=/var/aegir). Root-side, no _ROOT assignment exists anywhere in BARRACUDA.sh.txt or its sourced include set (grep-verified at HEAD), so that call runs with _ROOT empty and its move/copy targets degenerate. Treat the root-side call as a known wart, not a pattern to extend.

3. _hostmaster_go_up — fresh Provision backend (:156-220)

Despite the name, this installs the backend, not the hostmaster platform (that is step 5). It rebuilds ${_ROOT}/.drush/{sys,xts,usr} per _DL_MODE:

• BATCH (default): _get_dev_ext "backend.tar.gz" — one tarball containing backend/{sys,xts,usr}, moved up a level; the staging dir is only removed after sys/provision/provision.inc, xts/security_review and usr/registry_rebuild all verify (:164-178).
• GIT: seven pinned clones — provision.git at ${_BRANCH_PRN}, drupalgeddon 7.x-1.x-dev, drush_ecl 7.x-1.x, security_review 7.x-1.x, provision_boost 7.x-2.x, registry_rebuild 7.x-2.x, safe_cache_form_clear 7.x-1.x — plus _get_dev_ext for clean_missing_modules and utf8mb4_convert (:179-198).
• anything else: nine individual tarballs via _get_dev_ext into sys/usr/xts (:199-215).

_get_dev_ext/_get_dev_stc (helper.sh.inc:259-323) fetch from https://${_USE_MIR}/dev/${_tRee}/… (…/static/… for stc) with 10 attempts 9 s apart, -A iCab; a total failure is an OOPS: + return 1, not a fatal — the pipeline continues and the breakage surfaces at the next step's gate. The function ends by stripping .git dirs, sed-ing the chosen mirror into sys/provision/aegir.make (files.boa.io → ${_USE_MIR}, :218 — same op as AegirSetupM.sh.txt:138) and cd-ing back to ${_PREV_HM_ROOT}.

Duplication warning: this is a near-copy of _provision_backend_up() (master.sh.inc:656), which serves the INIT path via AegirSetupM. A fix to backend fetching must be applied to both copies (and considered for the satellite-side equivalents) — same rule as the master/satellite include split.

4. _hostmaster_dr_tt — drush sanity gate (:222-233)

Not disaster recovery — a go/no-go test of what steps 2–3 just installed:

${_DRUSHCMD} help | grep "^ provision-install"

Success logs ${_STATUS} B: Drush test result OK under debug; failure is FATAL ERROR: Drush is broken, FAIL marker, exit 1. Identical to the gate in AegirSetupM.sh.txt:127-136. This is the last stop before the migrate — nothing after it can run with a broken Drush/Provision pair.

5. _hostmaster_mi_up — repairs, then hostmaster-migrate (:235-384)

The actual frontend move, wrapped in pre-flight repairs of known bad states:

1. Neutralize security_review for the duration: its security_review.drush.inc is renamed to foo.txt (:237-239) and restored at the end (:370-371) — the plugin breaks the migrate when active.
2. Broken Entity module repair (:249-279): witness is module_filter.module sitting inside profiles/hostmaster/modules/contrib/entity/ of the previous root — a corrupted extraction. Repair: wipe the dir, refetch entity-7.x-1.12 via _get_dev_stc, en entity, disable the SSL module set (hosting_ssl, hosting_le, hosting_le_vhost, hosting_nginx_ssl), rebuild the http row in hosting_service by raw sqlq (DELETE/INSERT/UPDATE to type='nginx'), force-verify @server_master, hosting-dispatch, then three hosting-tasks --force passes 5 s apart.
3. Broken-SSL vhost repair (:280-305): if the master nginx.conf still greps as nginx default ssl server, run the same disable-SSL + service-row + verify/dispatch/tasks dance and touch the witness /var/aegir/disable_nginx_ssl.log.
4. Custom-settings module swap (:306-330): old-generation tree (modules/path_alias_cache present) → disable + pm-uninstall aegir_custom_settings, disable hosting_advanced_cron, en ctools, registry-rebuild; current tree → disable + uninstall hosting_custom_settings, registry-rebuild.
5. DB housekeeping (:331-340): cc drush, drop ~/.tmp/cache, load ${_bldPth}/aegir/helpers/hosting_cron.sql via sqlc, prune hosting_task_log older than 3 months + OPTIMIZE TABLE, purge stale delete tasks.

6. The migrate (:341-345):

   ${_DRUSHCMD} hostmaster-migrate ${_DOMAIN} ${_HM_ROOT} -y      # + -d under debug

   _DOMAIN comes off the state bus, _HM_ROOT is the freshly computed host_master/${_DISTRO} target. Note: the exit status is not checked and output is discarded outside debug — a failed migrate does not set the FAIL marker; the steps below run regardless.

7. Post-migrate (:346-383): parse the new root out of ${_ROOT}/.drush/hostmaster.alias.drushrc.php and create its sites/all/{modules,themes,libraries} skeleton; registry-rebuild, en hosting_cron, cache-clear all, updb -y; restore security_review; archive legacy frontend backups matching *host8*, *o8.io*, *boa.io*, *aegir.cc* into backups/system/old_hostmaster/ (0700 dir, 0600 files).

Failure model

One marker, one direction: /opt/tmp/status-AegirUpgrade-FAIL. There is no -OK marker — the parent treats absence of FAIL after the wait as success (master.sh.inc:869-874) and escalates FAIL to status-Barracuda-FAIL + _clean_pid_exit. Every abort path must therefore touch the marker before exiting; a bare non-zero exit is invisible to the orchestrating run. Touch points at HEAD:

Two failure classes deliberately do not trip the marker: mirror-fetch exhaustion inside _get_dev_ext/_get_dev_stc (returns 1, caught by the next gate at best) and a failed hostmaster-migrate (step 5.6 above). When a box comes out of barracuda up-* with a dead frontend and no status-AegirUpgrade-FAIL, the migrate itself is the first suspect — check the run log, and remember debug mode (touch /root/.debug-boa-installer.cnf) reruns the migrate with -d and without output suppression.

Maintainer contract

• New variable needed inside the script? Add an echo line to the state bus append block (master.sh.inc:830-849). Environment exports do not survive the su.
• New abort path? Touch /opt/tmp/status-AegirUpgrade-FAIL first — exit codes alone are swallowed.
• Backend fetch changes go to both _hostmaster_go_up (AegirUpgrade.sh.txt:156) and _provision_backend_up (master.sh.inc:656); a new backend extension additionally needs its name in _hostmaster_mv_up's _D_EXT list (:128-131).
• _hostmaster_dr_up is shared between the root-side and aegir-side call sites — it must stay safe under both (and see the empty-_ROOT wart in step 2 before touching it).
• PHP CLI ladder (:86-95) must track the supported CLI set in lockstep with AegirSetupM and _php_cli_drush_update/_php_cli_cron_update (master.sh.inc:626, :928) — four copies of the same version knowledge.
• Do not reorder the five calls — each step's precondition is the previous step's output (old backend archived → new backend fetched → drush proven → migrate).
• Transport: the script rides the staged build tree; no fNN serial to bump. Changes land on boxes at their next barracuda up-* run — see the serial & fetch pipeline before deciding where a hotfix belongs.

Related

• Staged setup engine (AegirSetup*) — the INIT-side counterpart: the four setup stages, the settings-snapshot bus and the marker-file contract this script shares.
• The serial & fetch pipeline — which transport a fix rides and when it reaches production boxes.
• Code style & conventions — _msg prefix rules (INFO:/OOPS:/FATAL ERROR:) and the bash conventions used throughout the orchestrator.
• Variables and Commands — consolidated tables for _DL_MODE, _PHP_CLI_VERSION, _AEGIR_UPGRADE_ONLY, _SYSTEM_UP_ONLY and the barracuda CLI verbs referenced above.