Stack architecture for maintainers

This is the wiring diagram you need in your head before changing BOA code: which repo owns which behaviour, where each file lands on a live box, and the exact call chain from a click in the Aegir UI to a provision-* verb mutating a site. Every path and line reference below is verified against the current source; when a line number drifts, the function name is the stable anchor.

The moving parts

Five codebases, all independent forks under github.com/omega8cc — omega8cc is the upstream; there is no pre-fork upstream to track. Public branch model: 5.x-dev (main development) with 5.x-dev-base/5.x-dev-edge, and the release lines 5.x-pro* / 5.x-lts*; tags land only on 5.x-pro and 5.x-lts and trigger SKYNET fleet self-update.

Companion repos pulled into the frontend platform build: hosting_civicrm, hosting_custom_settings, hosting_deploy, hosting_git, hosting_le, hosting_remote_import, hosting_site_backup_manager, hosting_tasks_extra, aegir_objects, and the eldir theme — cloned at the branch matching the active tree during the local build (lib/functions/satellite.sh.inc:3155-3167, _satellite_download_for_local_build).

Hostmaster is Drupal 7 and the backend is Drush 8: legacy boxes still run the backend under old PHP CLIs, so the design floor for Provision/Hosting/Drush PHP is 5.6 — no ??, no scalar type hints, no arrow functions, nothing newer than what PHP 5.6 parses.

Master and Satellite

One host runs one Master plus N Satellites — a BOA-specific layering. Each is a complete Aegir instance (own Hostmaster site, own DB user, own Drush, own aliases, own task queue). The Master is a thin coordinator holding the host-level server entities; tenant sites live on Satellites.

Two traps that keep resurfacing in older docs — both wrong against source: the Satellite instance root is not /home/o1 (that is only the lshell SFTP home o1.ftp), and the Hostmaster platform directory is not hostmaster-current/ or hostmaster-7.x-3.x on Satellites — it is the numeric aegir/distro/NNN counter. The Provision clone sits under .drush/sys/, not bare .drush/provision.

The bash side is split the same way

BARRACUDA.sh.txt sources _FL="helper dns system sql valkey redis nginx php solr master xtra firewall hotfix" (BARRACUDA.sh.txt:275); OCTOPUS.sh.txt sources _FL="helper dns satellite" (OCTOPUS.sh.txt:266). master.sh.inc and satellite.sh.inc are never loaded in the same process, and the near-duplicate functions between them (e.g. _ensure_hosting_dispatch_enabled at master.sh.inc:364 and satellite.sh.inc:2868) are separate copies by design. When you fix one, mirror the fix into the other copy — do not try to factor them into a shared include.

Hosting vs Provision — the frontend/backend boundary

The boundary is crossed in exactly two files: dispatch.hosting.inc (forks the queue workers) and task.hosting.inc (translates one task node into one backend call via provision_backend_invoke(), defined at provision.inc:1274). Nothing else in hosting shells out to the backend; if you need frontend→backend behaviour, that is where it goes.

Bootstrap contract: every provision-* command declares its own level in the provision.drush.inc command table (DRUSH_BOOTSTRAP_DRUSH for context-level verbs like provision-save, DRUSH_BOOTSTRAP_DRUPAL_ROOT / DRUSH_BOOTSTRAP_DRUPAL_SITE for platform/site verbs). Deeper bootstraps are escalated inside the verb: e.g. verify bootstraps the target site to DRUSH_BOOTSTRAP_DRUPAL_FULL for Drupal < 12 and caps at DRUSH_BOOTSTRAP_DRUPAL_CONFIGURATION for Drupal 12+ (platform/verify.provision.inc:22-28; the same pattern in install.provision.inc, import.provision.inc, deploy.provision.inc).

The Aegir context model

Every entity — server, platform, site — is persisted as a Drush alias file in the invoking user's ~/.drush:

<name>.alias.drushrc.php        # Provision/Config/Drushrc/Alias.php:36

The same file is simultaneously a Drush alias (drush @foo.example.com …) and a Provision context. provision_drush_init() hydrates it into the d() accessor at the start of every backend invocation (provision.drush.inc:52-58); d() and the named-context classes live in provision.context.inc, with the three types in Provision/Context/{server,platform,site}.php. Services attach to contexts via provision.service.inc, with the drivers under db/Provision/Service and http/Provision/Service.

Concrete instances on a box: hm.alias.drushrc.php (Master frontend), hostmaster.alias.drushrc.php (Satellite frontend), server_master.alias.drushrc.php (referenced from migrate.hostmaster.inc:41), plus one alias per platform and site.

Contexts are write-through from the frontend, never authoritative: before executing any task, task.hosting.inc:208 re-runs provision-save with the node's context_options, regenerating the alias file. A hand-edited alias survives only until the next task touches that entity. Site-local overrides belong in the site's local.settings.php / drushrc.php, not in the alias.

Task flow, end to end

Aegir UI form submit
  → hosting_add_task($nid, $type, …)                task/hosting_task.module:550
  → hosting_task node, status QUEUED                (Hostmaster DB)
      │  once per minute
      ▼
root crontab: runner.sh                             aegir/tools/system/cron/crontabs/root
  gates: /run/boa_queue_stop.pid pause · load < _CPU_TASK_RATIO×100 (default 3.1)
         /run/boa_run.pid blocks · /run/octopus_install_run.pid forces through
  → bash /var/xdrago/run-<user>                     runner.sh:122-152
      │  per-user flock; skip if a platform build is live for that user
      ▼
su - <user>:  drush8 cc drush ; bash /data/disk/<user>/aegir.sh
      ▼
<php-cli> ${_ROOT}/tools/drush/drush.php @hostmaster hosting-dispatch
      │  fork per due queue                         dispatch.hosting.inc:78
      ▼
drush @self hosting-tasks
      │  per queued node                            task/hosting_task.module:785
      ▼
drush @self hosting-task <nid>                      task.hosting.inc
      ├─ provision-save @<context>   (sync context from node)     :208
      └─ provision_backend_invoke($alias, 'provision-<type>', …)  :226
            ▼
provision-<type> in the backend, bootstrapped per verb
      ▲  log stream + backend result parsed back into the task node

Step by step, with the decisions that matter when you change any of it:

1. Node creation. Every mutating UI action calls hosting_add_task() (task/hosting_task.module:550), producing a hosting_task node with task_type, arguments, and HOSTING_TASK_QUEUED status. The UI task log is just this node re-read.

2. Queue kick — no daemon. There is no resident queue process. On the Master, the aegir user's crontab runs hosting-dispatch directly every minute (master.sh.inc:927-941). For Satellites, root cron runs /var/xdrago/runner.sh every minute; _runner_action iterates every /var/xdrago/run-* file (runner.sh:122-125). Cadence modifiers: /root/.fast.cron.cnf loops 10 passes 5 s apart (runner.sh:241-246), /root/.slow.cron.cnf runs one delayed pass — and is auto-created immutable on boxes with ≤ 4096 MB RAM (runner.sh:194-203). CI-flagged boxes (/etc/boa/.look.like.jenkins.cnf) keep the queue off unless explicitly allowed (runner.sh:223-234, _if_allow_aegir_queue).

3. Per-Satellite glue. /var/xdrago/run-<user> is the aegir/scripts/run-xdrago template with EDIT_USER substituted, installed by satellite.sh.inc:4371-4374. It takes a non-blocking per-user flock, skips the dispatch while that user has a live platform build (a _tmp_ dir in ${_ROOT}/.tmp with a running drush.php), then drops privileges: su - <user> → drush8 cc drush → bash /data/disk/<user>/aegir.sh. That one-liner script is generated at satellite.sh.inc:5466-5473 and contains the actual @hostmaster hosting-dispatch call under the instance's own PHP CLI and Drush copy.

4. Dispatch. drush_hosting_dispatch() reads hosting_get_queues() (hosting.queues.inc:56), applies each queue's frequency gate under the dispatch semaphore (with a variable_initialize() re-read of last_run so racing dispatchers serialize instead of double-forking), and forks drush @self hosting-<queue> (dispatch.hosting.inc:78). Exactly three hook_hosting_queues implementations exist: tasks (task/hosting_task.module:475 — serial, every minute, 5 items), cron (cron/hosting_cron.module:22), and task_gc (task_gc/hosting_task_gc.module:13).

5. Task execution. The hosting-tasks worker calls hosting_task_execute() per queued node, which forks drush @self hosting-task <nid> — the argument is the node ID, not an alias/verb pair (task/hosting_task.module:785).

6. Crossing into the backend. The hosting-task command (task.hosting.inc) re-saves the context (:208), resolves the backend command — $task->task_command, defaulted inline to 'provision-' . $task->task_type when unset (task.hosting.inc:222-223); the same provision-$TYPE fallback is also stamped onto task-type metadata by hosting_task_hosting_tasks_alter() (task/hosting_task.module:908-914) — and calls provision_backend_invoke($alias, $command, …) (:226). Backend log lines stream back into the task node; structured results come back through parse.backend.inc. On failure the hosting_TASK_TYPE_task_rollback hook fires (:249); on completion post_hosting_TASK_TYPE_task (:262).

The boa repo layout (github.com/omega8cc/boa)

BOA.sh.txt              meta installer + fleet update agent (see serials below)
BARRACUDA.sh.txt        host stack + Master installer; thin phase driver
OCTOPUS.sh.txt          Satellite installer; thin phase driver
CHANGELOG.txt           the genuine release changelog
lib/
  settings/             barracuda.sh.cnf, octopus.sh.cnf — functional defaults,
                        SOURCED at runtime (BARRACUDA.sh.txt:251, OCTOPUS.sh.txt:238);
                        config, not documentation — wrong text here is a code bug
  functions/            13 shared libs: helper, dns, system, sql, valkey, redis,
                        nginx, php, solr, master, satellite, xtra, firewall,
                        hotfix (.sh.inc) — sourced fresh from the staged clone
aegir/
  scripts/              AegirSetup{A,B,C,M}.sh.txt staged-setup engine,
                        AegirUpgrade.sh.txt, run-xdrago template
  tools/bin/            operator CLIs + helpers → /opt/local/bin
  tools/system/         cron-driven scripts + monitor/ tree → /var/xdrago
  tools/system/cron/crontabs/root   the root crontab, verbatim source
  conf/                 config templates: nginx, php, solr4/solr9, redis,
                        valkey, dns, apparmor, ftpd, ini, global, tpl, var, …
  helpers/  makefiles/  patches/

Delivery model — there is no package manager in the loop:

• The barracuda / octopus wrappers (themselves deployed to /opt/local/bin) select the tree from the verb (up-dev/up-pro/up-lts → _tRee, aegir/tools/bin/barracuda:1940-1947), then curl the installer and its settings file from https://files.boa.io/versions/<tree>/boa/ into /var/backups/ and execute them (barracuda:1834-1835, run at :1042). All mirror fetches use the -A iCab user agent (_crlGet); other UAs are blocked by the mirrors.
• The installer stages the full BOA tree at /opt/tmp/boa by downloading and extracting a tarball, not a git clone: _download_boa_code() curls ${_urlDev}/<tree>/boa.tar.gz and tar xzfs it into /opt/tmp (_get_dev_ext → _extract_archive, lib/settings/barracuda.sh.cnf:1595-1611; _download_helpers_libs() at :1613 pulls the same way), then sources lib/functions/*.sh.inc from that staged tree. Library changes therefore need no serial — every run re-fetches and re-extracts the whole tree, so a new .sh.inc is picked up on the next run with no per-file version gate.
• BOA.sh.txt doubles as the standing self-update agent. clear.sh (root cron, every 5 min — aegir/tools/system/cron/crontabs/root:13) re-pipes it from the mirror whenever no BOA task is running, then runs autoupboa (aegir/tools/system/clear.sh:235-237, both under if [ ! -e /run/boa_run.pid ]). autoupboa (aegir/tools/bin/autoupboa) is the actual upgrade executor: the weekly auto-update cron invokes it as autoupboa weekly-system up-<tier> system … noscreen, and it calls barracuda up-<tier> as a leaf (never the reverse — barracuda self-calling would loop). All of this is gated by _SKYNET_MODE; with _SKYNET_MODE=OFF autoupboa refuses to update.
• The nightly owl.sh (root cron 04:15 — crontab :24) is a notifier, not an updater: under _SKYNET_MODE on/unset it fetches barracuda-release.txt, compares it to /var/log/barracuda_log.txt, and on a newer release emails an upgrade notice (owl.sh:349 ff.). It does not run any upgrade itself.

Serial mechanics — the part you must not get wrong

Because BOA.sh.txt re-runs on every box every few minutes, each tool it deploys is gated by _fetch_versioned <dest> <url> <serial> (BOA.sh.txt:1056):

• On success it writes a stamp <name>.ctrl.<serial>.<tree>.<xSrl>.pid under /var/log/boa and removes all older stamps for that name; while the stamp exists and the destination is non-empty, the fetch is skipped.
• The deploy map is the _update_agents call list (BOA.sh.txt:1364-1423) — aegir/tools/bin/* → /opt/local/bin, each with its fNN serial and an optional --guard pattern that refuses to clobber a running tool.
• Editing a tool under aegir/tools/bin/ without changing its fNN in BOA.sh.txt ships a no-op: deployed boxes keep their stamped copy forever. Change the serial in the same commit as the tool. Never reuse a value that ever shipped for that tool (a box still stamped with it would skip the fetch); the tree convention is a monotonic decrement from f99.
• The stamp name embeds ${_tRee} and the header serial ${_xSrl} (BOA.sh.txt:48, e.g. 5103devT01) — so bumping _xSrl on a release invalidates every stamp at once and forces a full re-fetch. The sibling header values _rLsn (release label, :49) and _bTs (build timestamp, :50) are not part of the stamp name; only _xSrl gates re-fetch.
• aegir/tools/system/* → /var/xdrago is not serial-gated: it is a bulk cp -af on every barracuda run (lib/functions/system.sh.inc:9179, _xdrago_install_upgrade), which also installs the root crontab from /var/xdrago/cron/crontabs/root (system.sh.inc:9203) and appends /var/xdrago/cron/custom.txt if present.

Where each piece lands on a live box

Runtime state directories you will read while debugging: /var/log/boa (installer logs + serial stamps), /run/boa_run.pid / /run/boa_wait.pid / /run/boa_queue_stop.pid (global gates every queue and installer path honours), /root/.barracuda.cnf and /root/.<user>.octopus.cnf (per-box and per-instance overrides — see the variables reference).

Read the code in this order

1. BOA.sh.txt — the top ~100 lines (env, tree, serials), then _fetch_versioned (:1056) and _update_agents (:1364 ff.). This is the deploy map and the update contract; everything else hangs off it.
2. aegir/tools/bin/barracuda and octopus — verb parsing, tree selection, staging (_set_tree_vars, _up_start). Then aegir/tools/bin/boa, the umbrella CLI (case dispatch at boa:3037).
3. BARRACUDA.sh.txt + lib/functions/system.sh.inc + master.sh.inc — host stack and Master. The installers are thin; the libs are the substance.
4. OCTOPUS.sh.txt + lib/functions/satellite.sh.inc — the Satellite build. The second-largest lib after system.sh.inc and the whole of the Satellite surface; almost every Satellite path cited on this page is defined here.
5. aegir/scripts/AegirSetup{A,B,C,M}.sh.txt — the su-side staged setup the installers hand off to — plus run-xdrago and aegir/tools/system/runner.sh for the queue glue.
6. provision.drush.inc — command table with per-verb bootstrap levels and the hook docs in the file header — then provision.context.inc (d() and the context classes), then one full verb (platform/verify.provision.inc) and the service drivers under db/ and http/.
7. hosting.queues.inc + dispatch.hosting.inc → task/hosting_task.module → task.hosting.inc — the whole frontend→backend crossing in four files.
8. Only then the periphery: /var/xdrago monitors (second.sh, minute.sh, owl.sh, monitor/check/*) and aegir/conf templates, as the task at hand demands.

Related

• Developing BOA — guide index — the other maintainer topics.
• Variables reference — every _VAR control setting the installers and monitors read.
• Commands reference — the operator-facing CLI surface these internals implement.
• Discontinued features — dropped subsystems you will still find traces of in older trees.