Nginx debugging — 502/504, high-load 503, install loops

Three recurring Nginx-level failure modes on a BOA host:

  • 502 Bad Gateway / 504 Gateway Timeout on specific URLs or a whole site — almost always PHP-FPM or downstream, not Nginx.
  • High-load 503 — the reduced-feature mode second.sh switches on under CPU pressure.
  • Redirect loop to /install.php after a clone-then-migrate.

The diagnostic value here is using the correct runtime paths: BOA's FPM sockets and logs are not where a generic Nginx guide would put them.

502 vs 504

Error What Nginx saw
502 Bad Gateway PHP-FPM accepted the request, then closed the connection abnormally before responding (segfault, kill, or pool not running).
504 Gateway Timeout PHP-FPM did not respond within the FastCGI read timeout.

The FastCGI timeout is governed by _PHP_FPM_TIMEOUT (default AUTO, sized from the box class) in octopus.cnf, not a fixed 180 s. Both errors point at PHP-FPM or downstream (Drupal bootstrap, DB, Redis/Valkey), not at Nginx itself.

Where to look, in order

1. Nginx error log

tail -f /var/log/nginx/error.log

The upstream-related lines around the failure time:

upstream prematurely closed connection while reading response header from upstream   (→ 502: PHP segfaulted or was killed)
upstream timed out (110: Connection timed out) while reading response header          (→ 504: PHP is slow)
connect() to unix:/run/<pool>.fpm.socket failed                                       (→ 502: FPM pool not running)

The socket path in the third line is the real BOA FPM socket form (below) — if you see a different path, the vhost is referencing a stale upstream.

2. PHP-FPM error log

All PHP/FPM logs go to a single /var/log/php/ directory with version-tagged filenames — there is no per-version /var/log/phpNN/ directory:

tail -f /var/log/php/php84-fpm-error.log         # error_log, per version
tail -f /var/log/php/fpm-<pool>-slow.log         # slowlog, per pool
tail -f /var/log/php/opcache-<pool>-error.log    # opcache, per pool

Substitute the PHP version (php74, php83, php84, …) and pool name as needed. Look for:

  • child <pid> exited on signal 11 (SIGSEGV) — segfault, often an APCu / opcache issue.
  • child <pid>, script '…' executing too slow — a slow request approaching the timeout.
  • server reached max_children setting — the pool is saturated; raise the pool size (see PHP-FPM & performance).

3. Drupal watchdog

drush @<site-alias> watchdog-show --count=20

When PHP-FPM logged nothing useful, Drupal's own log often carries the DB or cache failure behind the 502.

4. Confirm services are up

for v in 5.6 7.0 7.1 7.3 7.4 8.0 8.1 8.3 8.4; do
  vv=${v//./}
  echo -n "PHP-FPM $v: "; service php${vv}-fpm status 2>/dev/null | head -1
done
service mysql status | head -1
service redis-server status 2>/dev/null | head -1
service valkey-server status 2>/dev/null | head -1

Restart anything stopped with service <name> start and watch whether the 502/504 clears.

5. Check the high-load 503 path

Under high CPU load, second.sh activates the rule by renaming /data/conf/nginx_high_load_off.conf to nginx_high_load.conf — the inactive _off copy is the deployed default, and dropping the _off suffix is what arms it (_nginx_high_load_on(), second.sh:183, the mv at :187; the file is pulled in via the include /data/conf/nginx_high_load.c*; glob at vhost_include.tpl.php:271). It reloads Nginx, switching on reduced-feature mode: the fragment's if ($deny_on_high_load) { return 503; } returns 503 only (not 502) to the matched crawlers/spiders — the $deny_on_high_load map in the master config (server.tpl.php:717-719) flags only crawler-class user-agents (crawl|spider|tracker|click|parser|google|yahoo|yandex|baidu|bing) and leaves the default empty, so ordinary human visitors pass and bots are shed. The toggle pair _nginx_high_load_on() / _nginx_high_load_off() (the reverse rename, second.sh:199-200) lives at second.sh:183-209; the load checks arm it at :397-398 (1-minute load) and :413-414 (5-minute load) and disarm it at :422-425 once both readings are back at or below the spider threshold. It logs to /var/log/boa/high.load.incident.log:

tail -f /var/log/boa/high.load.incident.log

Recent entries near the failure time mean the box was shedding load.

Distinct mechanism, same incident log: loadguard (aegir/tools/bin/loadguard) is a separate CPU-tier guard whose MAX/CRIT action fully stops Nginx and every PHP-FPM via init.d (_action_pause_web) — that produces connection-refused, not 502/503. A 502 is a third, unrelated case (FPM crashed). Don't conflate the three.

Common 502/504 causes by frequency

Cause Symptom Fix
APCu corruption Random 502 on different URLs, SIGSEGV in the FPM error log Drush cache-rebuild; restart the FPM pool
FPM pool not running All requests 502 instantly service phpNN-fpm start
Slow DB query 504 once the FastCGI timeout is hit Optimise the query; raise _PHP_FPM_TIMEOUT in octopus.cnf
Memory limit exceeded Sporadic 502 on specific URLs, OOM-killer in dmesg Raise PHP memory_limit; chase the leak
Redis/Valkey down 502, cache failures in watchdog Restart Redis/Valkey; check the daemon log
Disk full Random 502, write failures in watchdog Free disk space
max_children saturated Intermittent 502 during traffic spikes Raise pool size — see PHP-FPM & performance

Redirect loop to /install.php

Symptom: after a clone-then-migrate, Nginx redirects every request to /install.php, which itself returns 404.

Cause: the site directory is missing the sites/<domain>/settings.php that Aegir writes on Install — the migrate did not re-create it on the new platform.

Fix

  1. Confirm the site directory exists on the new platform (Octopus instance root is /data/disk/<USER>, not /home/<USER>): 
    ls -la /data/disk/oN/static/<platform>/sites/<domain>/
  2. If settings.php is missing, re-emit it — run Verify on the site, which re-writes settings.php from the Aegir template: 
    drush @<site-alias> provision-verify
  3. If Verify also fails, the site's DB connection may be misconfigured — check /data/disk/oN/log/<site>.log for the error.
  4. Last resort: restore the site from a known-good backup.

Diagnostic recipes

Is Nginx the problem at all?

Bypass Nginx and hit PHP-FPM directly. Use the real FPM socket and the real Hostmaster path:

SCRIPT_NAME=/index.php \
SCRIPT_FILENAME=/var/aegir/aegir/distro/<N>/index.php \
REQUEST_METHOD=GET \
  cgi-fcgi -bind -connect /run/<aegir-pool>.fpm.socket | head -20

BOA's Hostmaster root is ${_ROOT}/aegir/distro/<N> — for the Master, /var/aegir/aegir/distro/<N>/. (The /var/aegir/hostmaster-7.x-3.x/ form is pre-fork upstream Aegir and does not exist on a BOA host.) If this succeeds, Nginx is the issue; if it fails the same way, PHP-FPM / Drupal is.

Is the upstream socket reachable?

FPM sockets live under /run/, named <pool>.fpm.socket (PHP 7/8) or /run/php5-fpm.sock (the single PHP 5 socket). /opt/etc/fpm/ holds pool config, not sockets.

ls -la /run/*.fpm.socket
nc -U /run/o1.fpm.socket < /dev/null

If ls shows the socket but nc hangs, the FPM daemon is not reading it — restart the pool.

Force a fresh vhost

drush @<site-alias> provision-verify

Verify regenerates the vhost (see Config templates) and reloads Nginx — clears a stale vhost.

Related