Recovery cycles & cache faults

Two operationally distinct failure families share one symptom surface ("a task suddenly fails", "the site WSODs intermittently"): platform/Drush state drift, fixed by the Verify/Unlock/Verify recovery cycle; and the APCu/Valkey/cron plugin-discovery cascade, fixed by sizing and cron discipline. Diagnose which one you have before acting — the recovery cycle does nothing for a Valkey memory ceiling, and resizing Valkey does nothing for a platform whose local Drush is locked.

The platform recovery cycle

When a platform is "in a weird state" after a failed task — Drush version mismatch errors, lock state out of sync, Migrate/Clone/Verify aborting unexpectedly — run the three-step cycle:

1. Platform → Verify + Lock Drush
2. Platform → Unlock Local Drush
3. Platform → Verify + Lock Drush     (yes, again)

The middle Unlock Local Drush step un-patches Drupal core and removes the codebase permission locks; the final Verify + Lock re-establishes the canonical Verified+Locked state Aegir expects (dunlock/relock on Verify is implemented in the BOA hosting_platform/provision fork, not a generic toggle). This resolves:

  • LoggerChannel::log() declaration errors — typically from running Unlock Local Drush on an already-unlocked platform, or running a site task before the platform was Verified+Locked.
  • Site-local vdrush "doesn't work" — when the platform's local Drush is locked (the default post-Verify state), run Unlock Local Drush first; run Verify + Lock Drush again when finished. (The other cause is connecting as oN instead of oN.ftp — see lshell & limited users.)
  • Orphan site node with no working Drush alias after a failed Migrate/Clone — drush sa | grep -i foo.example.com empty means no on-disk alias; Verify the site to regenerate it from the node, and if that fails, run the full cycle at the platform level first.

Permission-drift Verify failures

Composer or manual file ops leave files with the wrong owner. Repair with the fix-drupal-* scripts (equals-form flags only) against the real platform path, then run the recovery cycle:

# As oN.ftp
fix-drupal-platform-permissions.sh --root=/data/disk/<USER>/static/<platform>
fix-drupal-platform-ownership.sh   --root=/data/disk/<USER>/static/<platform>

Composer broke the platform ("code path X doesn't exist")

Composer run on a platform after sites were attached modifies vendor/ and composer.lock in ways Aegir does not expect. Do not try to repair in place: roll the platform back to its pre-Composer state from git or backup, build a new platform carrying the Composer changes, and Migrate the affected sites onto it.

Anti-pattern — blanket re-Verify

Re-Verifying everything on a stuck box can make it worse: a Verify that hits a permission error mid-run can leave files in a worse state, and one that crashes mid-DB-update can corrupt schema state. Read the task log, fix the named root cause, then re-Verify.

APCu / Valkey / cron — the PluginNotFoundException cascade

The most-cited operational fault on busy hosts: intermittent PluginNotFoundException / PluginException / FieldException bursts that self-resolve after a few minutes and recur. BOA wires Drupal's bootstrap/discovery/config cache bins to cache.backend.chainedfast:

APCu (per-worker, fast)  →  Valkey/Redis (shared)  →  Database

Three distinct root causes, diagnosed in priority order. Do not override the chainedfast defaults to "fix" this — see the anti-pattern below.

Always check Valkey first

The cache password is in /root/.valkey.pass.txt (Redis hosts: /root/.redis.pass.txt); the PHP side reads /data/conf/valkey/pass.inc. There is no _VALKEY_PASSWORD control variable.

_VPASS=$(tr -d '\r\n' < /root/.valkey.pass.txt)
valkey-cli -a "${_VPASS}" config get maxmemory
valkey-cli -a "${_VPASS}" info memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'
valkey-cli -a "${_VPASS}" info stats  | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'

Healthy: used_memory_human well below maxmemory_human, evicted_keys low or zero, hit rate above 85–90 %. Unhealthy: usage at the ceiling, evicted_keys climbing, or hit rate below 50 % on a busy host — all point to maxmemory too low.

Cause 1 — Valkey memory starvation (most common)

When Valkey is hard-capped too low, volatile-lfu eviction runs continuously, the discovery bin gets evicted, concurrent rebuilds write incomplete entries, and workers reading them throw PluginNotFoundException. Tell-tale: the same plugin named across incidents.

The shipped cap formula:

_MAX_MEM_VALKEY = _RAM / 3      # written to valkey.conf as ${_MAX_MEM_VALKEY}MB

_RAM is usable RAM in MiB (from free -mt) after the BOA reserve is subtracted: _RESERVED_RAM defaults to _RAM / 4 (overridable in /root/.barracuda.cnf). So on a 24 GB box the cap is ~RAM/3 of usable RAM after the reserve, not a clean 24/3. BOA also sets maxmemory-policy volatile-lfu and disables RDB snapshots (save "") — this is a cache, not a store. Very old installs used _RAM / 6; if you are still carrying that, raise it.

Fix:

valkey-cli -a "${_VPASS}" config set maxmemory 8gb
valkey-cli -a "${_VPASS}" config rewrite
watch -n 10 "valkey-cli -a ${_VPASS} info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'"

evicted_keys should drop to zero within minutes; hit rate re-warms over 15–30 min. If the box is genuinely under-RAMmed for its site count, this buys time but the real fix is more RAM.

Cause 2 — Valkey cache poisoning via CLI drush

Investigate only after Valkey memory and hit rate check out. Signature: bursts at regular intervals (e.g. hourly), different plugins failing across incidents, a full cache clear reliably ending the burst, eviction stats fine. Hypothesis: a CLI drush cache rebuild writes partial discovery entries into Valkey mid-rebuild; FPM workers read the poisoned entries until the next clean rebuild.

Mitigation:

  1. Disable Drupal core's Automated Cron on all Aegir-managed sites.
  2. Use Aegir's wget-based cron instead — it writes from FPM context with a fully rebuilt cache.

  3. If wget-cron times out (e.g. Scheduler module misses its window), raise the PHP execution-time band in the FPM common pool files (these overwrite on barracuda upgrade — re-apply afterward):

    # /opt/etc/fpm/fpm-pool-common.conf (and -legacy.conf / -modern.conf)
php_admin_value[max_execution_time]     = 300
php_admin_value[max_input_time]         = 300
php_admin_value[default_socket_timeout] = 300

Cause 3 — class-not-found / unreadable on HDD-backed hosts

PluginException: Plugin instance class "…" does not exist while the .php file is on disk, self-resolving after minutes or on Verify. This is HDD I/O saturation: the autoloader hits a filesystem latency spike and times out. Common triggers: cache rebuilds, drush ops, Composer runs. No software fix for under-resourced hardware — reduce drush-based cron (Cause 2), keep the chainedfast defaults so reads stay in the memory tiers, and migrate rotational storage to NVMe. That is the actual fix.

Anti-pattern — disabling chainedfast

// /sites/<domain>/local.settings.php  — do NOT do this
$settings['cache']['bins']['discovery'] = 'cache.backend.database';

This bypasses the cache stack entirely: a discovery DB hit on every request. Fix the root cause instead and revert any such override.

APCu sizing & the self-service graceful FPM reload

APCu is a per-worker cache inside the FPM worker processes; it cannot be flushed remotely. A stale APCu entry (an old field definition, plugin registry, or Solr-core mapping) clears only when the worker is recycled — so after a config change, platform update, or Solr core rename, stale APCu is a common source of FieldException / PluginNotFoundException even when Valkey is perfectly healthy.

Sizing. Default apc.shm_size=256M, set in each version's per-version php.ini (/opt/php<ver>/etc/php<ver>.ini, with /opt/php<ver>/lib/php.ini as the alternate build layout — there is no separate apcu.ini); for 100+ sites raise the apc.shm_size line to 512M and reload that version's FPM. APCu sizing is secondary to Valkey sizing — raise Valkey first.

Self-service flush. BOA exposes a sentinel that triggers a graceful reload of every installed PHP-FPM version (clearing APCu pool-wide) without dropping active connections:

# As the site owner, from the Octopus account
touch ~/static/control/run-php-fpm-reload.pid

The monitor (monitor/check/php.sh) detects the sentinel within seconds, performs the reload, and removes the file. A cooldown (_FPM_COOLDOWN_SECS, default 30 s, tracked via /run/php84-fpm.cooldown) prevents reload storms. The sentinel is honoured only for accounts on POWER, PHANTOM, CLUSTER, ULTRA, or MONSTER plans, or on hosts carrying /etc/boa/.allow.php.fpm.reload.cnf; on any other plan the plan gate returns before the file is even inspected, so the sentinel is simply ignored and left in place (not removed) — if you create it and it never clears, the account's plan does not qualify. This is the correct fix for stale per-worker APCu — but only after ruling out Valkey starvation (Cause 1) and poisoning (Cause 2). Flushing APCu does nothing for a maxmemory set too low.

Correct Drush usage on BOA

Two mistakes produce errors easily misread as cache faults:

  • Run drush as oN.ftp under lshell, not as oN under bash.
  • For D8+, use site-local Drush (vdrush), not system drush8 — system Drush 8 is for D7 only.

See lshell & limited users for the limited-shell and Drush-version selection details.

Related