Aegir task failures

The BOA task queue is cron-driven, not daemon-driven: there is no queue daemon to restart. runner.sh scans /var/xdrago/ for run-<USER> dispatchers and runs each — under a load gate (_O_LOAD < _CPU_TASK_RATIO × 100, default ratio 3.1) — which sus to the Octopus user, runs the instance's aegir.sh (driving drush @hostmaster hosting-dispatch), and touches /var/log/boa/last-run-<USER>. A "stuck" task is therefore almost always either a dispatch that is not firing (load hold, maintenance pause), a task whose backend Drush run aborted, or stale on-disk state Aegir cannot reconcile. The recovery paths below act on those three classes directly.

Task spinning — status "Processing", nothing happening

Diagnosis

# Is cron firing the dispatcher? (expect a fresh marker within ~1 min)
ls -l /var/log/boa/last-run-<USER>
pgrep -af 'runner.sh'

# Is processing paused for maintenance, or load-throttled?
ls -l /etc/boa/.pause_tasks_maint.cnf /run/max_load.pid /run/critical_load.pid

# Per-instance Aegir log/state dir (task markers + run state)
ls -lt /data/disk/<USER>/log/ | head

There is nothing to "restart". If a maintenance pause marker is present, remove it to resume dispatch:

rm -f /etc/boa/.pause_tasks_maint.cnf

If /run/max_load.pid or /run/critical_load.pid is present, the box is in a load-control hold (see Load control) and task dispatch is deliberately deferred until load drops — do not force a run into a saturated box; let it clear. One exception: a genuine in-progress Octopus install/upgrade runs the queue despite the hold, honoured only while /run/octopus_install_run.pid is under 15 minutes old or a live /opt/local/bin/octopus process exists; a stale marker is deleted, so a crashed run cannot permanently bypass load protection.

What now heals itself

Several formerly manual recoveries are self-clearing on current code — check the BOA version before reaching for the older workarounds:

  • Dispatcher died holding the queue semaphore. The dispatch lock times out after 15 minutes (HOSTING_QUEUE_LOCK_TIMEOUT, 900 s), not the old hour. The manual semaphore clear shown on the task queue engine page is only needed inside that window.
  • Orphan _tmp_ build directories. A _tmp_ working dir under /data/disk/<USER>/.tmp counts as an active platform build only while a live drush.php process for that instance user exists. The check is re-evaluated on every runner pass and deletes nothing, so a _tmp_ dir left behind by a crashed build no longer wedges the instance queue until an unrelated cache clear — dispatch resumes on the next pass by design.
  • Dispatcher disabled after an upgrade or hostmaster reinstall. On older versions, a failed hosting-setup self-test could leave hosting_dispatch_enabled off, piling tasks up as Queued forever; the manual check is drush8 @hostmaster vget hosting_dispatch_enabled and, if disabled, drush8 @hostmaster vset hosting_dispatch_enabled 1. Current code self-repairs this: hosting-setup preserves a previously-enabled dispatcher when its self-test fails, and BOA checks-and-repairs the variable on Barracuda master and Octopus instance install/upgrade finalize (_ensure_hosting_dispatch_enabled, logging "task queue dispatcher re-enabled" only when it corrected something).

What does not heal: a task stuck in Processing is never re-dispatched. After 8 hours (28800 s) it merely ages out of the dispatcher's running-task concurrency count; the task itself stays Processing until you Reset or delete it in the UI.

Force a manual dispatch

# As root. run-<USER> requires root and su's to the instance user itself.
bash /var/xdrago/run-<USER>

This dispatches every queued task for that instance in one pass. The dispatcher serialises itself per instance with a non-blocking flock on /run/run-xdrago-<USER>.lock (held on fd 9, released automatically on exit, failing open if /run or flock is unavailable). If a runner pass is already in flight for that instance, the command exits cleanly and silently (flock -n 9 || exit 0) — that is de-duplication, not a failure; the in-flight pass is already dispatching the queue.

"Could not delete this site" / "Could not delete this platform"

The Delete task fails when files were removed manually from the platform's sites/ directory before the Aegir Delete task ran — Aegir tries to back up or verify a tree that is already gone.

Recovery — clean the Aegir record by hand

{hosting_site} carries no title column; the human-readable name is the {node} title, and the site-status column is status (not site_status — site_status is only the loaded-node PHP property). So the record cleanup must join against {node} on title, exactly as the companion task-delete already does:

# Mark the site node deleted (HOSTING_SITE_DELETED = -2), keyed via {node}.title
drush @hostmaster sqlq "UPDATE hosting_site SET status = -2 \
  WHERE vid = (SELECT vid FROM node WHERE title = 'foo.example.com');"

# Drop the orphaned tasks for that node
drush @hostmaster sqlq "DELETE FROM hosting_task \
  WHERE rid = (SELECT nid FROM node WHERE title = 'foo.example.com');"

-2 is the correct value (define('HOSTING_SITE_DELETED', -2)). Confirm the join target first if the same title exists across revisions:

drush @hostmaster sqlq "SELECT nid, vid, status FROM node n \
  JOIN hosting_site h USING (vid) WHERE n.title = 'foo.example.com';"

Then re-run Delete on the now-clean record.

sites/<domain>.restore left after a Clone/Migrate failure

A failed Clone or Migrate leaves a sites/<domain>.restore directory — the site's pre-task state, kept so the task can be rolled back. The real platform tree lives under /data/disk/<USER>/static/<platform>; the chroot-visible copy is the SFTP symlink under /home/<USER>.ftp/static/<platform>.

Source site intact

# Remove the stray .restore directory (real platform path)
rm -rf /data/disk/<USER>/static/<platform>/sites/<domain>.restore

Then Verify the site to reconcile Aegir's records.

Site half-migrated (exists broken on both platforms)

  1. Determine the good copy (compare timestamps and DB row counts).
  2. Disable the broken copy.
  3. Verify the good copy.
  4. Delete the broken copy.

See Cross-host migration for the full migrate mechanics.

CiviCRM site: backend task blocked after 5.10.1 (*.drush.inc filter)

Symptom. Aegir backend tasks (Verify, Migrate, Clone) on a CiviCRM site start failing after the 5.10.1 upgrade, while the site keeps serving traffic normally under PHP-FPM.

Cause. BOA's *.drush.inc command-file filter — shipped inert in 5.9.5, active from 5.10.1 — denies Drush command files located under tenant-writable site paths after realpath() resolution. CiviCRM legitimately ships its own command files (civicrm.drush.inc, cv.drush.inc, civicrm_drush.drush.inc) inside the site tree, so the backend Drush run that drives those Aegir tasks refuses to load them. This affects Drush command-file loading during backend tasks only; normal web requests never go through Drush.

Fix — per-Octopus opt-in allowlist

# As root. The Octopus user is the one owning the backend task's $HOME under /data/disk/.
touch /data/conf/<octopus-user>_civicrm.txt

When the file exists, the three CiviCRM command-file basenames are loaded during backend tasks; otherwise they stay blocked (off by default). Re-run the failed task once the file is in place. The filter mechanism is documented in lshell & limited users.

Verify fails after permission drift

Symptom. Verify fails with "Could not write to …" or similar ownership errors after Composer or a manual file operation left files with the wrong owner/mode.

Fix — BOA permission/ownership scripts

These scripts accept only the --flag=PATH equals form. The space-separated form (--root /path) hits the invalid-argument branch and exits 1 before doing any work. Run them as the SFTP user oN.ftp, against the real platform path under /data/disk/<USER> (or, inside the chroot, the /home/<USER>.ftp symlink):

# Whole platform
fix-drupal-platform-permissions.sh --root=/data/disk/<USER>/static/<platform>
fix-drupal-platform-ownership.sh   --root=/data/disk/<USER>/static/<platform>

# Single site (--site-path must contain settings.php or the script exits 1)
fix-drupal-site-permissions.sh --site-path=/data/disk/<USER>/static/<platform>/sites/<domain>
fix-drupal-site-ownership.sh   --site-path=/data/disk/<USER>/static/<platform>/sites/<domain>

Then re-run Verify. See lshell & limited users for the full fix-drupal-* script set and the oN.ftp limited shell.

Every Verify fails with "Access denied for user ''@'localhost'"

Symptom. Every Verify (and any other backend task) on every site fails with Access denied for user ''@'localhost' (using password: NO).

Cause. A botched hostmaster Migrate left the {hosting_db_server} credentials row empty or deleted, so the master_db credential URL rendered as mysql://:@host and every provision-verify failed. Previously the condition was self-perpetuating, with no UI recovery.

Post-5.10.3 Hosting carries three reinforcing guards: credentials are written via an atomic upsert instead of delete-then-insert, so the row is never momentarily absent; blank credentials are refused, with the last known-good db_user/db_passwd recovered from the current or, failing that, the latest revision of the DB-server node; and a degenerate mysql://:@host alias can no longer blank stored values on context import (the server presave hook carries db_user forward alongside db_passwd). On current code the condition is designed to self-recover on the next task that saves server credentials (e.g. Verify), as long as any revision of the DB-server node still holds the values — which is the botched-migrate case.

Operator action on current code: none beyond re-running the failed task. If the symptom appears at all, confirm the box is on a post-5.10.3 build before attempting deeper surgery. See Database for credential storage and Cross-host migration for the trigger scenario.

D10/D11 tasks: updatedb, cache rebuild, and the fixed container-poison fatals

Aegir's own Drush 8 cannot bootstrap a D10/D11 site, so on those platforms the deploy pipeline runs updatedb and the cache rebuild through the platform's site-local Drush. Four behaviours to know when reading a failed task log:

updatedb skipped on a degraded platform. When a D10+ platform has no executable site-local Drush, provision-deploy skips updatedb with a single actionable warning — updatedb skipped: site-local Drush not executable at <path> and Aegir Drush 8 cannot bootstrap D10+; apply the DB update manually — instead of a doomed Drush 8 attempt that only failed noisily. Recovery: run the platform's Unlock Local Drush task, then vdrush @alias updb (or repair the site-local Drush; vdrush is the pre-existing BOA wrapper). On this degraded path the automatic update_fetch_task clear also logs-and-skips, so the manual DELETE FROM key_value WHERE collection = 'update_fetch_task' workaround applies only here — every normal Clone, Migrate, platform Migrate, Rename, and Restore path clears it automatically after updatedb.

The fixed container-poison fatal family (post-5.10.3). Older builds fully unlocked vendor/drush for the cache-rebuild/updatedb windows, reverting the Aegir patches; the D8+ service container then compiled in stock state and baked Drush's DrushLog logger into the persisted (Valkey) container, which the patched web runtime could not resolve — ServiceCircularReferenceException on D11, Class Drush\Log\DrushLog not found on D10 — breaking every site on the platform after routine tasks (disable cron, Clone, Verify, deploy/updatedb) until a manual non-Drush cache clear. Current code opens a chmod-only exec window that flips only the exec bits and keeps the Aegir patches applied, so the container is never compiled in stock state; every early-abort path relocks, so a failed task never leaves the shared platform vendor/drush unlocked. The old fatal presented as a 500-class WSOD, not a 502 — see 502 Bad Gateway for separating the two.

could not bootstrap drupal after updatedb in a DEPLOY log. On D10/D11 this line no longer means stale caches were shipped: the cache rebuild runs drush cr through the site-local modern Drush (non-fatal by design), inside the same exec window as updatedb. D8/D9 keep the Aegir Drush 8 cache-rebuild via backend invoke.

Fresh D11+ installs fail visibly. A failed site-install now aborts the install task (PROVISION_DRUPAL_INSTALL_FAILED) instead of reporting a broken site as installed. On success, the one-time /user/reset/ login link is derived through the site-local Drush, and automatic_updates / package_manager are uninstalled right after install (Aegir never lets web UI tools overwrite the codebase).

Task log won't load in the UI

A task log too large for Drupal to render is still on disk. The most recent .log files under the instance log dir are the task logs:

ls -lt /data/disk/<USER>/log/ | head
less /data/disk/<USER>/log/<task-id>.log

Reading "Drush command terminated abnormally" in task logs

Since the Drush 8.5.3 bundle shipped with BOA 5.10.3, drush_shutdown() reports a cause for an abnormal exit only when the last recorded PHP error is fatal-class (E_ERROR | E_PARSE | E_CORE_ERROR | E_COMPILE_ERROR | E_USER_ERROR | E_RECOVERABLE_ERROR). Previously any lingering recorded error — typically a PHP 8.x implicitly-nullable deprecation emitted at autoload by the old vendored psy/psysh and correctly suppressed by error_reporting — was printed as Drush command terminated abnormally due to an unrecoverable error: ... Implicitly marking parameter $config as nullable, mislabeling a harmless deprecation and hiding the real cause. The common trigger was exiting the psysh REPL (drush php / core-cli), which calls exit() before drush_main() completes.

Operator takeaway: on current code the cause line is trustworthy — if "terminated abnormally due to an unrecoverable error" names an error, it is a genuine fatal. In older task logs, an ...unrecoverable error: ... deprecated/nullable... line was noise; look for the real failure elsewhere in the log.

Related