Ghost & empty-artifact cleanup

Long-lived BOA boxes accumulate ghosts: orphaned Composer trees with no working docroot, legacy shared codebases no platform links to any more, platform aliases whose platform is gone, nginx vhosts with no matching site, stale alias copies for sites deleted long ago. The nightly owl run and the every-3-minutes manage_ltd_users.sh cron detect all of these — and, only when explicitly enabled, move them aside.

Three rules govern every mover on this page:

• Opt-in, dry-run by default. Every flag defaults to NO: out of the box each check only logs what it would move and changes nothing. You flip one _VAR=YES per check, per box or per account, after reviewing the dry-run output.
• Moves, never deletes. An enabled check relocates the item to a backup or undo/ path; recovery is moving it back.
• Safeguards sit above the flags. The Provision interlock and the fail-closed liveness tests below apply on every run, enabled or not; the nightly per-site reapers additionally demand ghost state across consecutive nights before acting.

The reaper family

The six nightly checks belong to the owl run (04:15 by default): the per-account workers handle platforms, vhosts, and sites for their account; after all accounts finish, the two codebase checks and the Hostmaster platform sweep run once globally. The ltd-alias reaper lives in manage_ltd_users.sh (see lshell + manage_ltd_users) because that script already owns the tenant alias copies it inspects.

Version-agnostic detection

Whether a tree is "real" is judged by _detect_real_docroot (night/night.inc.sh): an index.php at the platform root or under docroot/, html/, or web/. That is the front controller present in every Drupal docroot from D6 to D11, regardless of build recipe, so Composer-based D8+ platforms — whose web root differs from the application root and which no longer ship sites/all — are never misread as ghosts. Ghost detection never keys on sites/all. The -f test follows symlinks and is false for a dangling one, so a decoy like web -> /etc cannot register as a docroot.

For the marker-gated platform reapers a vendor/ tree alone also counts as live: a codebase mid-composer-build has no docroot yet, but it is not a ghost.

Always-on safeguards

These apply before any move, regardless of flags:

• Provision interlock. While any Aegir/Provision task runs (install, clone, migrate, verify, backup, restore) the site and platform trees are transiently inconsistent — files mid-repoint, aliases mid-rewrite, vhosts staged under a leading dot. Every cleanup checks pgrep -f provision first (_provision_running) and skips entirely, logging a notice, when one is found. It retries on the next pass once the box is quiet.
• Consecutive-night markers. The nightly site, vhost, and platform reapers never act on a single snapshot: _ghost_seen_enough requires the item to look ghost on 2 consecutive nightly runs, tracked in counter files under <account>/log/ctrl/ (ghost-*.seen). The moment an item looks valid the counter is reset (_ghost_seen_reset), so a recovered item never carries a stale count into a later reap. A transient state — a momentarily-unmounted store, a symlink mid-repoint — gets at least one grace run to recover.
• Symlink-aware liveness. A site is judged live on drushrc.php alone, present as a file or a symlink. files/ and private/ are deliberately not required: under native files-symlinking those point into a static store whose target can be transiently absent, and that must never count as gone.
• Fail-closed parsing. A degraded or mid-rewrite alias that parses to an empty or non-/data/disk site_path keeps the site and resets its marker — a bad parse never reaps.
• Freshness and staging skips. Vhosts younger than 24 h (mid-import/activation) reset their marker; leading-dot companion vhosts (.example.com — staged or rollback originals from xoct/xcopy) are always skipped; the vhost sweep skips the whole account while log/exported.pid marks a migrate/export in flight; the ltd reaper skips aliases written under 60 minutes ago and re-checks the Provision interlock inline on its 3-minute cadence.
• No instant re-bake. On any manage_ltd_users.sh pass that reaped, the tenant Drush yml site-cache rebuild is skipped, so a deregistration is never immediately baked into the tenant's Drush state.

One platform check predates the markers: the legacy empty-platform sweep in the per-account worker (and its Hostmaster twin) carries no consecutive-night marker — it is age-gated instead, examining only platform_* alias files older than _DEL_OLD_EMPTY_PLATFORMS days (see the octopus.cnf reference). Only its ghost-alias branch belongs to the flag family above; the sweep's original job — deleting an old legacy (sites/all) platform no site uses, through a Hostmaster task — is that variable's own feature, not gated by any _GHOST_* flag. The marker-gated platform reapers run independently of the age gate.

Control flags

All default NO. The two codebase checks walk the whole /data tree, so their flags are system-wide only. Semantics for the dual-scope flags: YES in either file enables the move for that account — the per-account file lets you opt in one account at a time, but a per-account NO cannot veto a system-wide YES (only YES enables, anything else is ignored). The Hostmaster platform sweep reads /root/.barracuda.cnf only.

Flags are read at act time by grepping the control file (case-insensitive YES; a missing file or unset variable reads as disabled) — no reload or restart is involved, and the next nightly pass simply picks up the current value. BOA is designed to seed the NO defaults into the live control files on install and to append any still-absent flag on the next barracuda up-* / octopus up-* upgrade, never overwriting a value you have set; a box whose control files predate the flags behaves identically, since unset reads as disabled.

Reading the dry-run output

Where each check logs:

Typical dry-run lines, verbatim:

GHOST platform /data/disk/o1/static/foo detected (dry-run; set _GHOST_PLATFORMS_CLEANUP=YES in /root/.o1.octopus.cnf or /root/.barracuda.cnf to move)
Ghost /data/disk/o1/distro/002/foo detected (dry-run; set _GHOST_CODEBASES_CLEANUP=YES in /root/.barracuda.cnf to move)
Unused /data/all/001/foo detected (dry-run; set _SHARED_CODEBASES_CLEANUP=YES in /root/.barracuda.cnf to move)
GHOST vhost for foo.example.com with no drushrc detected (dry-run/grace; set _GHOST_VHOSTS_CLEANUP=YES to move)
GHOST drushrc for foo.example.com detected (grace run, not moved)
GHOST foo.example.com.alias detected (dry-run; set _GHOST_ALIASES_CLEANUP=YES to move)

(grace run, not moved) means the consecutive-night marker is still counting — the item looked ghost tonight but has not yet been ghost long enough to act on even with the flag enabled. (dry-run/grace) covers both states in one line: flag off, or marker not yet satisfied.

Recovering a moved item

Nothing is deleted, so recovery is moving the item back from its destination:

• ghost codebases — mirrored under /var/backups/ghost-codebases-cleanup/ at their original path, in a <timestamp>/ subdirectory per move;
• shared codebases — /var/backups/codebases-cleanup/… (or /data/disk/codebases-cleanup/… on symlinked /data/all);
• platform aliases, platform dirs, vhosts, site aliases — the owning account's undo/ dir (/var/aegir/undo/ for Hostmaster); a reaped site's vhost lands as undo/ghost-vhost-<domain> and its files dir as undo/ghost-site-<domain>. The ~/.ftp/.drush mirror copy the ltd reaper deletes is derived data: restore the master alias and a live site's copy is re-created on the next 3-minute pass.

One retention caveat: the nightly run prunes the shared-codebase backup dirs after 7 days. Review what landed there within that window; the ghost-codebase mirror and the undo/ dirs are not aged out by the night family.

Enabling on production

The codebase and platform checks are the older, long-exercised pair. The per-site reapers (_GHOST_VHOSTS_CLEANUP, _GHOST_SITES_CLEANUP, _GHOST_SITE_FILES_CLEANUP, _GHOST_ALIASES_CLEANUP) act on live serving artifacts and are newer — exercise them in dry-run, then on a disposable VM, before enabling on production. A sane rollout:

1. Watch the dry-run output across at least two nights (the markers need two consecutive runs anyway) and confirm every listed path really is a ghost.
2. Enable per account in that account's octopus.cnf first; go system-wide in barracuda.cnf only once per-account runs are clean.
3. Leave _GHOST_SITE_FILES_CLEANUP for last — it is split out precisely because it moves site data, not just registrations.

Related

• Nightly owl maintenance — the 04:15 run that hosts the six nightly checks: per-account fan-out, load gating, and the global post-account pass.
• lshell + manage_ltd_users — the orchestrator that owns the tenant alias copies and carries the _GHOST_ALIASES_CLEANUP reaper.
• barracuda.cnf and octopus.cnf — the control files the flags live in, and _DEL_OLD_EMPTY_PLATFORMS.
• Reference appendix — the catalogued _GHOST_* / _SHARED_CODEBASES_CLEANUP variables.