Orphan stores & the .archived archive

Deleting a site removes its docroot — including the in-site files/private symlinks — but not its per-site store under static/files/<url>/. What remains is a ghost store: uploaded data with no site attached. This page covers how the subsystem detects ghosts, the two situations in which it moves them aside, the .archived layout they land in, the accumulation alert, and why pruning the archive is deliberately left to you. The store layout itself belongs to the overview, the tools and their logs to the tools reference, and the cron/opt-in plumbing to nightly automation.

One rule governs everything here: nothing on this page ever deletes data. Every path archives — normally a free same-filesystem rename into a hidden directory — and every archiving logs a du/df disk-space snapshot, even when space is ample.

Do not confuse these ghost stores with the registration ghosts (platforms, vhosts, site aliases) handled by the opt-in ghost & empty-artifact cleanup — that family moves stale registrations to undo/; this page is about the site's data.

The strict orphan test

The account sweep scans static/files/ for entries that look like site storage (a directory containing files/ or private/, or any symlinked entry) and have no matching active site. For each candidate <name> it checks two registration artifacts inside the account:

  • the Drush alias — /data/disk/<account>/.drush/<name>.alias.drushrc.php
  • the nginx vhost — /data/disk/<account>/config/server_master/nginx/vhost.d/<name>

Only a store whose name has neither is archived — a genuinely deleted site. The double test exists because of disabled sites: a merely disabled site keeps its alias and a placeholder vhost, so it fails the test on both counts, is treated as active, and its store is left in place — the files stay live for a later re-enable. A partial/broken state (only one of alias/vhost survives) is reported but also left alone, with an explicit line in the daily report:

[REPORT] ORPHAN static/files entry in account <acct>: <path> (size=<N>K) – a surviving alias/vhost remains (disabled or partial state); left in place, NOT archived

Hidden entries (.archived itself), lost+found, and the known non-site entries dbackup and foo.com are skipped outright. symlinkinfo <site> (see the tools reference) prints the same classification — active / disabled / deleted / partial — for any name you ask about.

The .archived layout

Every archive, from either trigger below, lands under the store itself:

/data/disk/<account>/static/files/.archived/<UTC-stamp>/<site>/{files,private}

The stamp is %Y%m%dT%H%M%SZ-<pid> and is run-scoped, so a site's files and private (and all orphans of one sweep) share one stamp directory. Two deliberate properties of the location:

  • Under static/files, not a sibling. static/files may be a symlink onto attached storage; a static/files-* sibling would land on the possibly-tiny root filesystem and turn a free rename into a cross-FS copy that can fill the disk. Under the store, the move is always a same-filesystem rename.
  • Leading dot. The orphan/site scan ignores .* entries, so the archive is never itself mistaken for a live or orphaned site — yet the tools still track it (the accumulation report below, and symlinkinfo's archive history). The same convention hides the relocated .backups/.backup-exports dirs — see backups on the static filesystem.

The move degrades safely: if the archive base somehow sits on a different device and the ghost would not fit in its free space, or if the mv fails, the reuse-time path renames the store aside in place and the nightly path leaves the orphan where it is — a [WARN] line either way, never a disk-filling cross-FS copy, and never a blocked conversion.

Reuse-time archiving

Ghost stores used to block reusing a deleted site's name. Now, when an install, clone, or migrate/rename reuses a name whose stale store still exists, autosymlink archives it aside automatically and the new site converts cleanly — no manual step. The dry-run announces it beforehand:

[PLAN] Existing (stale) store target will be archived before move: <...>/static/files/<site>/files -> <...>/static/files/.archived/<stamp>/<site>/files

The same archiving covers break-sharing collisions: when a clone or migrate whose deployed link still points at another site's store needs its own copy (--force-unshare), a pre-existing store at the target name is archived before the copy — handled identically to the local-convert path, so it lands in the same reported, prunable archive instead of cluttering the live store. See site cloning and cross-host migration for the task-side view.

Note the asymmetry on migrate/rename: the target name's stale store is archived at reuse time, but the old name's store is simply left behind — it becomes a normal reported orphan, picked up by the nightly sweep below.

The nightly sweep

With the nightly automation enabled (_AUTOSYMLINK_NIGHTLY=YES, updatesymlinks --auto-fix — cadence and opt-ins on nightly automation), the apply modes auto-archive each deleted-site leftover that passes the strict orphan test, logging one incident per store:

[REPORT] ORPHAN archive incident: <site> deleted-site leftover size=<N>K -> static/files/.archived/<stamp>/ (target FS avail=<M>K)

(The reuse-time variant reads <site>/<type> stale store instead of deleted-site leftover.)

The sweep runs with the Aegir task queue paused and drained (the self-healing /run/boa_queue_stop.pid honoured by runner.sh — see task queue), so an in-flight install or clone can never make a live site momentarily look deleted to the scan.

The read-only paths are unchanged: updatesymlinks --orphan-report and autosymlink report still only list orphans — useful as a preview, or as the steady state when the auto-fix is off.

Opt-out: create /data/conf/disable_orphan_store_archiving.cnf to revert the nightly sweep to report-only; remove it to re-enable. It gates only the sweep — reuse-time archiving is unaffected, since without it a reused name could not convert at all.

Accumulation alert

Archiving never reclaims space: a big site whose name is reused many times leaves many copies. The daily report therefore sums each account's static/files/.archived/ pile (read-only, moves nothing) and, once the total crosses the threshold, emits an ORPHAN-class line:

[REPORT] ORPHAN archived-store pile in account <acct>: <...>/.archived total=<N>K >= threshold <T>K - prune old entries to reclaim space

updatesymlinks emails the report whenever it contains a [REPORT] ORPHAN line (subject ORPHANS detected), so crossing the threshold means an email; below it, a plain informational Archived-store pile ... (below threshold ...) line is logged but does not itself trigger the email.

The threshold defaults to 1 GiB (1048576 KB) and is tunable box-wide via /data/conf/native_files_archive_alert_kb.cnf — a single KB integer on the first line; remove the file to restore the default. It controls only the alert, never any archiving behaviour. The control files are catalogued on the configuration leaf.

Pruning is operator-only

No age-based auto-clean is applied to .archived/ — deliberately. The age-based purge BOA uses for /data/disk/<account>/backups is safe there because backup timestamps are always fresh; an archived orphan may hold data that is already very old yet still wanted, so purging it by age cannot be guaranteed safe. The tools archive and alert; deleting is your call, after review:

du -sh /data/disk/<account>/static/files/.archived/*        # oldest-first by stamp
rm -rf /data/disk/<account>/static/files/.archived/<UTC-stamp>

Before pruning a stamp, symlinkinfo <site> shows which archive events produced it (reuse vs deleted-site orphan) and which archived copies are still on disk, with sizes.