Cache & cron tuning

The most-cited operational failure mode on busy BOA hosts: intermittent PluginNotFoundException errors, self-resolving after a few minutes, recurring in bursts. Three distinct root causes, diagnosed here in priority order, plus the APCu-flush sentinel for clearing stale per-worker cache.

Applies to BOA 5.x, Drupal 8/9/10, the PHP-FPM + APCu + Valkey/Redis stack on Aegir-managed sites.

The cache stack

The default cache backend is cache.backend.redis (Valkey/Redis). On a standalone (non-cluster) host BOA then overrides the bootstrap / discovery / config bins to cache.backend.chainedfast:

APCu (per-worker, fast)  →  Valkey/Redis (shared)  →  Database

The chainedfast override is gated on not being a cluster node (if (!is_readable('/data/conf/clstr.cnf'))): on a cluster node those three bins stay on cache.backend.redis (the shared tier only), so the per-worker APCu tier does not apply and the APCu-flush guidance below is moot there. The redis_exclude_bins site/platform INI can additionally push named bins onto cache.backend.database (also only off a cluster node).

When Valkey is starved for memory the discovery cache is evicted; rebuilds under concurrent traffic can produce incomplete entries; plugin-not-found errors cascade. Do not override the chainedfast defaults without understanding the trade-offs (see the anti-pattern below).

Diagnose Valkey first

Memory starvation is the most common root cause on busy hosts. Get the Valkey password from /root/.valkey.pass.txt (the Redis variant uses /root/.redis.pass.txt; the PHP side reads /data/conf/valkey/pass.inc). There is no _VALKEY_PASSWORD variable in /root/.barracuda.cnf.

_PASS=$(cat /root/.valkey.pass.txt)

# Memory ceiling vs. current usage
valkey-cli -a "${_PASS}" config get maxmemory
valkey-cli -a "${_PASS}" info memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'

# Hit rate + eviction count
valkey-cli -a "${_PASS}" info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'

Healthy: used_memory_human well below maxmemory_human; evicted_keys low or zero; hit rate above 85–90% (keyspace_hits / (hits + misses)).

Unhealthy: used_memory_human at or near maxmemory_human (ceiling too low); evicted_keys non-zero and climbing (active eviction — the cause of discovery-cache loss); hit rate below 50% on a busy host (almost always maxmemory too low).

Issue 1 — Valkey memory starvation (most common)

Symptoms. Intermittent WSOD bursts that self-resolve after a few minutes; watchdog Drupal\Component\Plugin\Exception\PluginNotFoundException entries; the same plugin named across multiple incidents (vs. cache poisoning, where it varies); a full cache clear resolves temporarily but errors return.

Root cause. Valkey hard-capped at maxmemory. With the ceiling too low, eviction runs continuously (maxmemory-policy volatile-lfu), the discovery bin gets evicted, worker requests trigger rebuild from DB, concurrent rebuilds produce incomplete entries, and a worker reading an incomplete entry throws PluginNotFoundException.

The shipped default formula:

_MAX_MEM_VALKEY=$(( _RAM / 3 ))

_RAM is usable RAM in MiB (from free -mt) after the BOA reserve is subtracted: _RESERVED_RAM, when left at its default 0, is auto-set to a quarter of total RAM, then _RAM = total − _RESERVED_RAM. The cap is written to valkey.conf as ${_MAX_MEM_VALKEY}MB. So on a 24 GB host the cap is roughly a third of usable RAM after the reserve — not a clean 24/3 = 8 GB. The instance is also configured maxmemory-policy volatile-lfu with RDB snapshots disabled (save ""). The current formula is hard-coded to _RAM / 3 and rewritten on every tune run, so there is no per-install variable to carry forward.

Fix:

_PASS=$(cat /root/.valkey.pass.txt)
valkey-cli -a "${_PASS}" config set maxmemory 8gb
valkey-cli -a "${_PASS}" config rewrite

watch -n 10 "valkey-cli -a ${_PASS} info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'"

evicted_keys should drop to zero within minutes; hit rate re-warms over 15–30 min.

RAM sizing for D8/9/10 hosts:

Sites Recommended RAM
≤ 50 16 GB
50–150 32 GB
150–300 48 GB
300+ 64 GB+

If the host is under-RAMmed for its site count, increasing Valkey within existing RAM helps temporarily; the real fix is more RAM.

Issue 2 — Valkey cache poisoning via CLI drush

Investigate only after confirming Valkey memory + hit rate healthy.

Symptoms. Burst pattern at regular intervals (e.g. every hour); different plugins fail across incidents (vs. Issue 1's same-plugin pattern); a full cache clear reliably ends the burst; Valkey eviction stats look fine.

Hypothesis. CLI drush running a cache rebuild can write partial discovery entries into Valkey mid-rebuild; FPM workers read poisoned entries and fail until the next clean rebuild.

Actions:

  1. Disable Drupal core's Automated Cron on all Aegir-managed sites.
  2. Switch Aegir to web-based cron instead of the drush backend. In the Hostmaster front-end this is the hosting_cron_use_backend setting: it defaults to TRUE (the drush/CLI backend — the poison source). For a regular managed site in backend mode hosting_cron_queue (hosting/cron/hosting_cron.module) runs two drush invokes, provision_backend_invoke … "elysia-cron", then sleep(3), then provision_backend_invoke … "cron"; the Hostmaster site itself is instead a single "cron" invoke, and it always uses the drush backend regardless of this toggle. Setting the toggle to FALSE makes the module hit each site over HTTP instead (https://<site>/cron/<cron_key>, or cron.php?cron_key= on pre-D8). That runs cron in FPM/web context against the live, fully rebuilt cache, which is what avoids the mid-rebuild poisoning.
  3. If web-cron times out (e.g. Scheduler module misses its schedule), raise PHP execution time in /opt/etc/fpm/fpm-pool-common*.conf: 
    php_admin_value[max_execution_time]     = 300
php_admin_value[max_input_time]         = 300
php_admin_value[default_socket_timeout] = 300

The fpm-pool-common*.conf files are overwritten on barracuda upgrade — re-apply after upgrades.

Issue 3 — class-not-found / file-unreadable on HDD-backed hosts

Symptoms. Drupal\Component\Plugin\Exception\PluginException: Plugin instance class "…" does not exist, even though the .php file IS on disk; self-resolves after a few minutes or on Verify.

Cause. HDD I/O saturation under disk pressure → the class autoloader hits a filesystem latency spike → PHP autoloader times out → "class not found". Common triggers: cache rebuilds, drush ops, Composer runs.

Mitigation. No software fix for under-resourced hardware. Reduce drush-based cron (Issue 2). Keep the chainedfast defaults intact so frequent reads stay in memory tiers, not disk. If the host has rotational storage, migrate to NVMe — that is the actual fix.

Anti-pattern — don't disable chainedfast

The tempting "fix":

// /sites/<domain>/local.settings.php
$settings['cache']['bins']['discovery'] = 'cache.backend.database';

This bypasses the cache stack entirely — a performance regression on every request, since discovery now hits the DB. Fix the root cause instead.

Correct Drush usage

Two mistakes produce errors easily misread as cache issues:

  1. Run drush as oN.ftp under lshell, not as oN under bash. Running drush as the bare oN system user under bash bypasses the limited-shell wrapper and produces errors easily mistaken for cache faults.
  2. For D8+, use the site-local Drush (vdrush), not system drush8 — system Drush 8 is for D7 only.

APCu memory sizing

apc.shm_size=256M    # template seed in phpNN.ini, NOT the live value

The 256M in phpNN.ini is only the shipped template seed. _tune_memory_limits rewrites it to the box-wide _USE_APC budget on every tune run and on barracuda upgrade (sed "s/256M/${_USE_APC}M/g"), so the live apc.shm_size already scales with RAM — a hand-edit to 512M is reverted on the next upgrade. _USE_APC = _USE_FPM is derived purely from usable RAM (the budget ladder in FPM capacity sizing) with no operator override, so the durable lever for more APCu is more RAM (or a smaller _RESERVED_RAM), not a phpNN.ini edit. For a quick one-version test you can still set it and service phpNN-fpm reload, but treat it as transient. APCu sizing is secondary to Valkey sizing — increase Valkey first.

Nightly Boost cache clearing (legacy D6/D7 platforms)

A different cache entirely: the Boost module's static page cache on legacy Drupal 6/7 platforms — unrelated to the D8+ chainedfast (APCu/Valkey) stack the rest of this page covers. On platforms carrying the o_contrib (Drupal 6) or o_contrib_seven (Drupal 7) bundles, the nightly per-site maintenance (aegir/tools/system/night/20-sites.sh) clears the platform-level Boost cache: _fix_boost_cache wipes <platform>/cache/* plus the cache/.boost and cache/.htaccess markers, creates the cache dir if missing (only when sites/all/drush/drushrc.php exists), and ensures it is owned oN:www-data mode 02775.

The call is gated by _CLEAR_BOOST=YES inside the o_contrib/o_contrib_seven platform branch. Default is YES (the shipped barracuda.sh.cnf default; the nightly run also falls back to YES when the variable is unset). Set _CLEAR_BOOST=NO in /root/.barracuda.cnf to keep Boost static caches across nights — the shipped cnf comment warns that with NO, boost cron and/or the Expire module must handle cache removal. _CLEAR_BOOST is a first-class persistent barracuda.cnf variable (written on fresh install, back-filled on upgrade) — see its entry in the barracuda.cnf reference.

Self-service APCu flush (graceful FPM reload)

APCu is a per-worker cache inside the FPM worker processes. Unlike Valkey it cannot be flushed remotely — a stale entry (old field definition, plugin registry, Solr core mapping) only clears when the worker is recycled. After a config change, platform update, or Solr core rename, stale APCu is a common source of FieldException / PluginNotFoundException even when Valkey is healthy.

BOA can graceful-reload every installed PHP-FPM version without dropping active connections, driven by a sentinel file the monitor watches. The sentinel lives in an Octopus account's control dir, so it is touched from that account (~ = /data/disk/<oN>):

# From the Octopus account (~ resolves to /data/disk/<oN>):
touch ~/static/control/run-php-fpm-reload.pid

The monitor (aegir/tools/system/monitor/check/php.sh) detects the sentinel, performs the graceful reload (clearing APCu pool-wide), and removes the file automatically. A cooldown (_FPM_COOLDOWN_SECS, default 30 s, shared across versions) prevents reload storms if the sentinel is recreated repeatedly.

Availability gate. Enabled only for accounts on POWER, PHANTOM, CLUSTER, ULTRA, or MONSTER plans (matched in /root/.<oN>.octopus.cnf), or on hosts where the operator has created /etc/boa/.allow.php.fpm.reload.cnf. On any other host the sentinel is ignored. This is the correct fix for stale per-worker APCu — after you have ruled out Valkey starvation (Issue 1) and cache poisoning (Issue 2). Flushing APCu does nothing for a maxmemory ceiling that is too low.

Diagnostic log paths

PHP-FPM error logs are a single directory, one file per version:

/var/log/php/phpNN-fpm-error.log     # e.g. /var/log/php/php84-fpm-error.log

There is no per-version /var/log/phpNN/ directory and no php-fpm.log file — every version logs to /var/log/php/phpNN-fpm-error.log (aegir/conf/php/phpNN-fpm.conf error_log = …; the directory is created once by mkdir -p /var/log/php).

Checklist

  • Check Valkey hit rate + eviction count.
  • If unhealthy: raise maxmemory to ~RAM/3.
  • If RAM under-allocated for site count: escalate a RAM bump.
  • Disable Drupal core's Automated Cron site-wide.
  • If drush-based cron triggers bursts: set hosting_cron_use_backend = FALSE (web-based Aegir cron).
  • If web cron times out: raise PHP execution time in fpm-pool-common*.conf.
  • Revert any local.settings.php discovery-bin override.
  • Confirm drush ops run as oN.ftp, not oN.
  • Confirm site-local vdrush for D8+ sites.
  • If APCu utilisation >75%: add RAM (lifts _USE_APC); a manual apc.shm_size bump is reverted on the next tune/upgrade.
  • If stale APCu suspected (post config/platform/Solr-core change): touch ~/static/control/run-php-fpm-reload.pid for a graceful all-versions FPM reload (qualifying plans or .allow.php.fpm.reload.cnf).

Related