Backup subsystem overview & architecture

A BOA host carries two generations of off-site backup tooling, all built on Duplicity and all installed together by BOA.sh.txt. They differ in config surface, bucket scheme, provider coverage, and whether they are license-gated.

The orchestrators

dcysetup is the installer/wiring tool for the multiback suite; it is also license-gated. backboa and duobackboa carry no _verify_boa_keys call, so they are physically present and runnable on every host regardless of tier — "LTS-only" is a usage recommendation, not an enforced restriction. The practical split is: multiback/dcysetup are the modern, license-gated, multi-provider path; backboa/duobackboa are the simpler single-bucket AWS-S3 legacy path.

Why Duplicity

All four tools shell out to /usr/local/bin/duplicity (pinned 3.0.6, Python 3.13.9 for all four tools — dcysetup install and backboa install pin the identical versions; built/installed by dcysetup or backboa install). Duplicity provides:

• Encryption before upload. multiback/mybackup use a symmetric passphrase read from .secret.txt into PASSPHRASE; backboa/duobackboa use _AWS_PWD into the same env var. Some backend paths run --no-encryption (the custom path-set on hosted systems).
• Full + incremental chains. _set_mode picks full vs incremental per run from cache state and whether today's full-log already exists — not by reading FULL_BACKUP_FREQUENCY — never by a CLI verb. The full cadence is enforced separately by Duplicity's --full-if-older-than ${FULL_BACKUP_FREQUENCY}, embedded in the default backup command (_DCY_BUP_CMD). On hosted systems the global/data path-sets instead use _FBF_BUP_CMD, which omits --full-if-older-than and forces a full only on the first of the month (_DOM=1).
• Point-in-time restore via --time / --path-to-restore against a volsize-300 archive.

The Duplicity invocation runs at --concurrency scaled by core count (_useCpu = 1/2/4 for up-to-4 / up-to-8 / over-8 CPUs) and --volsize 300.

What multiback actually captures

multiback does not back up a hard-coded path list. It backs up whatever the selected path-set (<user> argument) resolves to, sourced from a paths.txt file via _load_paths. The root-side path-sets are generated by dcysetup setup (create_global_paths_config.sh):

• global — _SOURCE="/etc /opt/solr4 /var/aegir /var/solr7 /var/solr9 /var/www /var/xdrago" plus generated include globs for /data/disk/arch, /var/backups/csf, /var/backups/dragon, /var/backups/reports, and an include-regexp for /root/.*.cnf.
• data — empty _SOURCE; an auto-generated per-/data/disk/<user> include set, which also appends --include /data/all, --include /data/conf and --include /home. It excludes each tenant's .tmp, backup-exports, backups, clients, src, u, undo, static/{.tmp,restores,tmp,trash}, each /home/<user>.ftp account's own .tmp, backups, clients, platforms and static, plus /var/www. The excludes are written as literal paths (/data/disk/<user>/backups, /data/disk/<user>/backup-exports); there is no exclude for static/files/.backups or static/files/.backup-exports, so on a relocated account the excluded names are only symlinks while the real tarball stores sit inside the included tree — one more reason to read the generated paths.txt.
• custom — operator-supplied include/exclude lists only.

A per-tenant mybackup run uses /data/disk/<user>/remote_backups/paths/paths.txt.

Because _SOURCE and the include/exclude sets are config-driven, the exact captured set is host-specific; read the generated paths.txt to know what a given host backs up, rather than assuming a fixed list.

Per-account backup directories may be symlinks

Aegir's per-site backup tarballs land in /data/disk/<account>/backups (Provision derives backup_path as <aegir_root>/backups) and backup-download exports in /data/disk/<account>/backup-exports; the nightly per-account run re-asserts aegir_backup_export_path to <account>/backup-exports right after running the relocation check below.

Post-5.10.3, when an account's static/files resolves to a different device than the account root (stat -c '%d' on the account vs stat -L -c '%d' on static/files), the nightly per-account run moves both directories onto the static filesystem and replaces them with symlinks:

/data/disk/<account>/backups         -> /data/disk/<account>/static/files/.backups
/data/disk/<account>/backup-exports  -> /data/disk/<account>/static/files/.backup-exports

Backups of native-symlinked sites dereference the files/private symlinks, so the tarballs are large and would otherwise fill the root partition; on a default single-filesystem box the device check makes this a deliberate no-op. The Aegir paths keep working through the symlinks, and both stores are deliberately kept on one filesystem so the backup-download hardlinks between backups and backup-exports keep working — hardlinks cannot cross filesystems. The leading-dot store names are skipped by the site/orphan scan.

To check whether an account is relocated: readlink /data/disk/<account>/backups — empty output means a plain directory, not relocated. Kill-switches: box-wide /data/conf/disable_backups_on_static_fs.cnf, per-account /data/disk/<account>/static/control/no_backups_on_static_fs.info.

The migration mechanics — the incremental rsync --remove-source-files move, the /run/.boa_backups_relocate.flock serialisation, and the /run/boa_queue_stop.pid task-queue pause — are owned by Backups on the static filesystem.

The local-dump layer underneath

Off-site backup archives a host that already has local MySQL dumps in place. mysql_backup.sh (cron 15 1 daily) writes per-DB dumps to /data/disk/arch/sql/<host>-<date>/<db>/ — mydumper by default (--sync-thread-lock-mode=AUTO, forced FTWRL on Percona 5.7), with the mysql system schema always taken by mysqldump --routines --events. Those dumps live under /data/disk/arch, which the global path-set includes — so the off-site copy carries the local dump tree. Local-dump retention (_DB_BACKUPS_TTL, default 14 days) is independent of off-site retention. See Database.

Known failure — backup binary permissions

/usr/bin/mysqldump and /usr/bin/rsync must be mode 755 so the unprivileged aegir/Octopus user can execute them. A transient 700 (or 750) on these binaries leaves them unexecutable for that user and breaks all Octopus/Aegir per-site backups with mysqldump: Permission denied. autoupboa restores 755 in normal operation and deliberately sets 700 only on a restricted VM — when /root/.restrict_this_vm.cnf is present (killall -9 rsync; chmod 700 /usr/bin/rsync /usr/bin/mysqldump). If site backups fail with a permission error, check these two modes before anything else. See Troubleshooting.

Concurrency and gating

Every tool refuses to run a second Duplicity in parallel: a positive pgrep -fc duplicity aborts the run (multiback logs to /var/log/mybackup_waiting_queue.log and, on _INCIDENT_REPORT=ALL, emails a waiting report). All tools also honour /root/.pause_heavy_tasks_maint.cnf (exit 0 immediately) and abort if root disk usage is over 90%. mysql_backup.sh additionally short-circuits on /root/.proxy.cnf.

Related

• multiback operations — config tree + install.
• Retention — KEEP_WITHIN / FULL_BACKUP_FREQUENCY.
• CLI reference — verbs and argument order.
• Reference appendix — consolidated variable table.