Task queue engine

Every change Aegir makes is a Task node, processed asynchronously. BOA drains that queue from cron, not from a daemon: a load-aware runner.sh fires a per-Octopus dispatch chain that ends in drush @hostmaster hosting-dispatch. There is no long-running queue process — the optional upstream hosting_queued daemon ships in the fork but BOA never starts it. This page is the operator reference for the queue engine: the lifecycle, the cron chain, the fast/slow cadence, the three registered queues, and the controls.

Task lifecycle

1. Front-end (Hostmaster UI) creates a Task node
       e.g. "Install site foo.example.com"  → status=Queued
2. Cron fires /var/xdrago/runner.sh (once a minute, load permitting)
3. runner.sh runs each Octopus's /var/xdrago/run-<USER>
4. run-<USER> runs /data/disk/<USER>/aegir.sh
       → drush @hostmaster hosting-dispatch
5. hosting-dispatch forks the due queue workers
       → hosting-tasks picks up Queued task nodes
6. The task forks drush @self hosting-task <NID>, which resolves to
       the matching provision-<type> backend command
7. Provision runs server-side, returns log + exit code
8. The Task node is updated:
       success → status=Successful, log attached
       failure → status=Failed, log + error attached
9. Front-end refresh shows the new status (UI polls the task node)

A failed task stops the queue for that entity (a failed Migrate on foo.example.com blocks further tasks on that site until resolved). The operator must Reset the task in the UI (or delete it) to unblock.

How BOA runs the queue

The engine is a chain of cron-fired shell scripts ending in a single Drush dispatcher. Nothing here is resident — every layer is a short-lived process started by cron.

1. Cron (aegir/tools/system/cron/crontabs/root) runs four scripts every minute:

* * * * *   bash /var/xdrago/second.sh
* * * * *   bash /var/xdrago/minute.sh
* * * * *   bash /var/xdrago/guest-fire.sh
* * * * *   /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/runner.sh

runner.sh is the only queue-relevant entry — the task-queue orchestrator, run niced + ioniced so it never competes with site traffic. (second.sh / minute.sh / guest-fire.sh are the other per-minute agents; none touches the task queue.)

2. runner.sh is load-aware. It:

  • Bails / waits when the box should not process tasks: a proxy node (/root/.proxy.cnf), an explicit maintenance pause (/etc/boa/.pause_tasks_maint.cnf), the dedicated queue-stop file (/run/boa_queue_stop.pid — see Pause / resume), a high/critical load flag (/run/max_load.pid, /run/critical_load.pid), a running SQL backup (mysql_backup.sh / mysql_cluster_backup.sh / mydumper), the nightly owl.sh run (formerly daily.sh), a MySQL restart, or another BOA task already holding the queue (/run/boa_run.pid, /run/boa_cron_wait.pid). On the SQL-backup / nightly back-off it touches /var/log/boa/wait-runner.pid, logs "Another BOA task is running, we will try again later...", and exits — queue draining defers until the dump or maintenance run finishes.
  • Throttles on load: it compares one-minute load to _CPU_TASK_RATIO (default 3.1, i.e. a ceiling of ratio × 100% per CPU) and skips a pass when over the ceiling.
  • Iterates the per-Octopus runners (_runner_action): for every /var/xdrago/run-<USER> it re-checks load, runs the script, then pauses a random 2–10 s before the next instance.
  • Accelerated mode overrides the load gates during a genuine Octopus install/upgrade. The CLI touches the marker /run/octopus_install_run.pid (set by octopus and by boa up-* runs, removed on completion) so the queue keeps running regardless of load and the freshly upgraded backend config and Drush aliases reload promptly: runner.sh skips the max_load.pid / critical_load.pid exit and runs each run-<USER> regardless of the _CPU_TASK_RATIO gate. The marker is honoured only when backed by a live /opt/local/bin/octopus process or when it is fresh (under 15 minutes); a stale marker is deleted, so a crashed run can never permanently bypass load protection. run-<USER> cooperates: while the marker is fresh it overrides its own build-in-progress skip so dispatch still fires. Normal-operation load protection is unchanged, and the SQL-backup / nightly back-off above is not bypassed by accelerated mode.

3. Per-Octopus run-<USER> (/var/xdrago/run-<USER>, generated from aegir/scripts/run-xdrago by satellite.sh.inc with EDIT_USER → the instance user, mode 0700) does cache/tmp cleanup, optional SFTP-password rotation, an optional Octopus self-upgrade, then runs drush8 cc drush and bash /data/disk/<USER>/aegir.sh as the instance user. It records /var/log/boa/last-run-<USER> and bails if /run/boa_wait.pid exists. Two guards shape when it actually dispatches:

  • Per-instance serialization: it takes a non-blocking flock on fd 9 (/run/run-xdrago-<user>.lock) so overlapping cron runner.sh passes, the fast-schedule loop, and the accelerated install/upgrade loop cannot fire concurrent hosting-dispatch + drush8 cc drush storms for one instance. A concurrent run exits cleanly (exit 0); the lock is released automatically on process exit, so there is no stale lock to clear (the lockfile itself persists harmlessly), and it fails open — runs unlocked — if flock or a writable /run is unavailable. The front-end dispatcher's own serialization (step 5 below) is the authoritative no-double-launch guard; this flock is best-effort de-duplication.
  • Build-in-progress skip: a _tmp_ Drush working dir under /data/disk/<USER>/.tmp marks a platform build in progress and makes run-<USER> skip that instance's dispatch (touching /var/log/boa/skip-run-<USER> instead of last-run-<USER>). It counts as an active build only while a live drush.php process for that instance user exists — an orphaned _tmp_ dir left by a crashed build does not freeze the instance queue until an unrelated cache clear. The check is re-evaluated on every run and deletes nothing.

4. Per-Octopus aegir.sh (/data/disk/<USER>/aegir.sh) is the actual dispatch call:

php .../tools/drush/drush.php @hostmaster hosting-dispatch
touch /data/disk/<USER>/<USER>-task.done

5. drush @hostmaster hosting-dispatch (hosting/dispatch.hosting.inc) is the real queue processor. It reads the registered queues (hook_hosting_queues) and, for each enabled queue whose frequency is due, forks the matching drush hosting-<queue> worker for up to N items, guarded by Drupal lock_acquire / lock_wait semaphores. One cron entry, frequency tunable from the front end, no resident process. The semaphore semantics:

  • The per-queue semaphore is named hosting_dispatch_<queue>_running and is acquired with a lifetime of HOSTING_QUEUE_LOCK_TIMEOUT = 900 s (15 min; reduced from the historical 3600 s). The lock is held only for the brief fork-and-dispatch, so the timeout is purely a recovery bound: a dispatcher that dies holding the lock blocks that queue for at most 15 minutes.
  • The dispatcher is serialized: while holding the dispatch semaphore it reloads the D7 variable table (variable_initialize(); D7 caches $conf per request) and re-reads hosting_queue_<queue>_last_run, then stamps last_run synchronously before forking the worker — so two overlapping dispatch runs are designed no longer to both pass the frequency gate and each fork a full batch (previously up to 2× the configured concurrency).
  • Concurrency is capped as calc_items minus running_items, where running_items counts PROCESSING tasks whose executed stamp is within the last 28800 s (8 h). This window is deliberately generous and dual-role — shortening it lets long installs/migrations age out of the count while still PROCESSING and re-opens the 2×-batch hole. Stuck-PROCESSING tasks are never re-dispatched (only QUEUED tasks are); they age out of the concurrency count after 8 h.

Dispatcher enablement. hosting-dispatch processes queues only while the hosting_dispatch_enabled variable is TRUE — otherwise it logs "Dispatching disabled." and exits. hosting-setup (run during install/upgrade) disables the dispatcher, self-tests one dispatch, and re-enables it on success; Hosting remembers the prior state and keeps the dispatcher enabled when the self-test fails on a system where dispatch was already enabled — logging "Dispatch self-test failed, but dispatch was already enabled; leaving it enabled." — while a fresh install whose self-test fails stays disabled as before. BOA adds belt-and-braces repair: _ensure_hosting_dispatch_enabled() checks and re-enables the variable via drush ev immediately before the hosting-dispatch calls on Barracuda master install/upgrade finalize (lib/functions/master.sh.inc) and, as a deliberately separate satellite copy, before hosting-dispatch on Octopus instance install/upgrade finalize (satellite.sh.inc; su mode when the caller is root pre-install, bare mode when already the instance user), logging INFO only when a correction was made. Manual one-line check/repair (also works on older versions), as the instance user:

drush8 @hostmaster vget hosting_dispatch_enabled
drush8 @hostmaster vset hosting_dispatch_enabled 1

Fast vs slow cadence

How aggressively runner.sh fires is set by BOA control files, not by a daemon poll interval:

Mode Trigger Behaviour Concurrency
Slow /root/.slow.cron.cnf present (and no /root/.force.queue.runner.cnf) one _runner_action pass, padded 15 s before and after 1
Fast /root/.fast.cron.cnf (or /root/.force.queue.runner.cnf) loops _runner_action ×10 with 5 s sleeps 8
default neither flag a single _runner_action pass 8

Boxes with ≤ 4096 MB RAM are forced Slow: runner.sh writes /root/.slow.cron.cnf (made immutable with chattr +i) plus a .slow.cron.cnf.protected marker, so the slow cadence survives plan changes. The Octopus installer sets .fast.cron.cnf / .slow.cron.cnf from the instance's plan tier.

On CI / Jenkins-style boxes (/etc/boa/.look.like.jenkins.cnf) the automatic queue is off by default. It runs only when a higher Octopus plan (POWER / PHANTOM / CLUSTER / ULTRA / MONSTER) is present, or /etc/boa/.allow.aegir.queue.cnf exists, and at least one instance opts in with /data/disk/*/static/control/run-aegir-queue.info.

The three registered queues

This hosting fork registers exactly three queues via hook_hosting_queues: tasks, cron, and task_gc. hosting-dispatch forks the matching hosting-<queue> worker when each is due — the same path for all three:

  • tasks — the main task queue (hosting-tasks); processes Queued task nodes into provision-<type> calls.
  • cron — hosting_cron runs advanced Drupal cron on hosted sites. BOA additionally hardens this with its own reliability patch (aegir/patches/hosting_cron_queue-reliability.patch).
  • task_gc — hosting_task_gc garbage-collects old completed task nodes / task logs (keeps the last N completed per entity, deletes the rest; configurable per instance).

There is no separate backup or statistics queue in this fork: a per-site backup is a backup task node processed by the tasks queue, not its own dispatch queue. New task types are added by a hosting module implementing hook_hosting_tasks() (the object token — site, platform — is an array key, $tasks['site']['backup'], not part of the hook name); see the Reference appendix for the hook surface.

A note on hosting_queued

Upstream Aegir offers two ways to drain the queue: the cron-driven dispatcher (above) and an optional long-running daemon (hosting_queued). The hosting/queued/ module ships in the omega8cc/hosting fork because it is inherited from upstream, but BOA does not run it — there is no init script, no service, no enablement anywhere in the BOA stack. So on a BOA host there is no hosting_queued daemon process, no /var/log/hosting_queued*.log, and no service hosting-queued-* … unit. Upstream docs describing a queue daemon do not apply.

Pause / resume

To pause all task processing on a host (maintenance window, manual DB fix), create the BOA maintenance-pause control file — runner.sh checks it on every run and exits immediately while it exists:

# Pause: runner.sh (and several other BOA jobs) stop touching the queue
touch /etc/boa/.pause_tasks_maint.cnf

# Resume
rm -f /etc/boa/.pause_tasks_maint.cnf

This is the same flag BOA's self-upgrade (autoupboa) sets while it works. Under high load BOA also pauses the queue automatically via /run/max_load.pid / /run/critical_load.pid until load recedes. There is no per-queue hostmaster-pause Drush command — the hosting-pause / hosting-resume commands are unrelated (they bracket a migration of the Hostmaster site itself to a new platform).

BOA's own nightly maintenance uses a separate, dedicated queue-stop file: /run/boa_queue_stop.pid holds the Aegir task queue while a maintenance operation moves data (set by the backups-to-static-fs relocation in night/10-account.sh, which records its own PID in the file and removes it only if it still owns it, and by updatesymlinks --auto-fix). runner.sh honours it twice: the parent exits immediately at startup, and each per-instance child dispatch is skipped if the file appears mid-pass ("Task queue paused: /run/boa_queue_stop.pid present -- skipping"). It is self-healing by design and can never freeze the queue for good: the file lives in /run (cleared on reboot) and clear.sh (cron */5) purges it as soon as the owner PID recorded inside it is gone, while a legitimately long move keeps its hold. Treat it as a mechanism you may observe, not a control to set by hand — for planned maintenance use /etc/boa/.pause_tasks_maint.cnf as above.

When tasks fail

Two common operator-facing failures:

  • Task hangs ("spinning") — usually Drush could not reach Percona (a restart mid-task) or hit a network timeout. Confirm runner.sh is firing, then re-run.
  • Task fails immediately — read the task log in the UI; it carries the Drush + Provision output. Most common causes: bad file ownership, a network issue, or DB corruption.

To force the queue forward by hand:

# Run one Octopus instance's runner immediately
bash /var/xdrago/run-o1

# Or call the dispatcher directly for that instance
su -s /bin/bash - o1 -c "drush @hostmaster hosting-dispatch"

# Is the cron runner firing at all? (last-run marker + live process)
ls -l /var/log/boa/last-run-*
pgrep -fc 'runner.sh'

A crashed dispatch self-heals: the dispatch semaphore (hosting_dispatch_<queue>_running) is acquired with a 15-minute lifetime, so a dispatcher that died holding it blocks that queue for at most 15 minutes before the lock expires on its own. If you need the queue back sooner than the automatic bound, clear the semaphore for that instance by hand:

su -s /bin/bash - o1 -c \
  "drush @hostmaster sqlq \"DELETE FROM semaphore WHERE name LIKE 'hosting_dispatch_%'\""

The per-Octopus Hostmaster alias is @hostmaster (equivalently @hm), resolved from that instance's own ~/.drush/ — so run it as the instance user as above. There is no @hm-oN / @hm-o1 alias; that form does not exist and will fail with an unknown-alias error.

See Troubleshooting for the broader task-failure playbook.

Related