Cross-host migration — xoct, xcopy, xmass

Three root-side tools move accounts between BOA hosts. All three are fetched and auto-updated by BOA.sh.txt via _fetch_versioned serials (plus a chmod 700 sweep), so the binaries live at /opt/local/bin/{xoct,xcopy,xmass} and all three — xcopy included — self-refresh on every BOA update.

Tool Moves DB transport Source after Write-freeze
xoct one Octopus account mydumper / myloader converted to proxy http-off 503
xcopy one Octopus account (duplicate) mydumper / myloader stays live none
xmass the whole host (all accounts + Solr) xtrabackup snapshot + GTID replication converted to proxy http-off 503

xoct is a move (the source account is decommissioned to a proxy); xcopy is a copy (the source account keeps serving — no http-off, no source proxy conversion); xmass is a whole-server move for evacuating a box.

Account identity is preserved across the hop: BOA does not support cross-account migration in the general case (o1 lands on o1, o2 on o2). The one rename axis is xoct/xcopy's optional fourth argument, which renames the moved account on the target (see below).

xoct — single-account move

xoct is a multi-step dispatcher, not a two-verb prep/import model. Almost every step runs on the source with the target IP as the argument — including create, which SSHes to the target to provision the new Octopus account there (boa in-octopus). Only import and post-mig are run on the target; pre-mig is run on both. The optional fourth argument renames the account on the target.

# Phase 0 — pre-mig, run on BOTH source then target
xoct pre-mig <source-fqdn>

# On SOURCE — provision target shared infra + first-pass transfer.
# transfer/pretransfer default to a read-only DRY run: review the
# [DRY-PLAN] output, then re-run the same command with --live.
xoct transfer    shared <target-ip>                # DRY plan
xoct transfer    shared <target-ip> --live
xoct create      o1 <target-ip> [o2]
xoct pretransfer o1 <target-ip> [o2]               # DRY plan
xoct pretransfer o1 <target-ip> [o2] --live

# On SOURCE — export (writes http-off 503 + dumps) then transfer
xoct export   o1 <target-ip>
xoct transfer o1 <target-ip> [o2]                  # DRY plan
xoct transfer o1 <target-ip> [o2] --live

# On TARGET — import, then post-mig (restores runner scripts on target)
xoct import   o1 <target-ip> [o2]
xoct post-mig <source-fqdn>

# On SOURCE — convert old host to proxy, then post-mig
xoct proxy    o1 <target-ip> [o2]
xoct post-mig <source-fqdn>

# Then update DNS A records to <target-ip>; the proxy hop drops away
# once DNS propagates.

The dispatch verbs are: export, create, import, pretransfer, transfer, proxy, pre-mig, post-mig, and ssl-gen (re-issue proxy SSL, gated on /root/.ssl.proxy.cnf). transfer shared provisions the account-independent shared infrastructure on the target; the per-account verbs take the Octopus username (o1, o2, …) as their second argument and the target IP as their third. When the optional fourth argument differs from the source username, xoct enters rename mode on import (renameaegirhost rewrites every reference from the source to the target account name). The transfer/pretransfer verbs are gated by the DRY/--live model — see Storage-aware transfers and the DRY/--live gate below.

export dumps the entire hostmaster database (${_HDB}, derived from the hostmaster site's drushrc.php) into /data/disk/<oN>/src/prev_hostmaster.sql, using the same mysqldump option set renameaegirhost uses (--single-transaction --quick --no-autocommit --skip-add-locks --no-tablespaces --hex-blob). The previous hand-curated table allowlist — tied to the removed direct-SQL import surgery below — silently dropped any hostmaster table not on the list, so newly added module/field schema tables simply vanished from the migrated instance; the full dump gives the import target a faithful copy for renameaegirhost's rewrite and 5-pass queue. The same applies to xcopy export; xmass is unaffected (full-server replication, no hostmaster mysqldump).

On the target, import reduces to: re-import of the full hostmaster DB dump, a couple of frontend variable-set calls via drush (site_frontpage, plus hosting_client_send_welcome in the fix path — Aegir API, not direct SQL), then renameaegirhost --aegir-root. renameaegirhost handles the in-place hostname rename of the moved account — a full 5-pass Aegir task queue that rewrites Drush aliases, nginx vhost files under config/server_master/, and the Aegir DB content as its atomic rename step. This supersedes the legacy "migrated/merged instance" fixup layers, which were removed from import: the direct-SQL surgery (in-place sed of hostmaster drush aliases plus hardcoded-nid drush sqlq REPLACE/UPDATE/DELETE against hosting_context/hosting_package/node/ hosting_site/hosting_platform/users_roles, guarded by a one-shot post-merge-fix.pid) and the ghost/empty-platform delete loop, whose missing-sites/all test was true for every valid Composer D8+ platform and so deleted valid platforms from Aegir during import. xcopy import follows the same model; the intended remaining xoct/xcopy divergence is that xcopy does not enable the migration proxy.

Migrated/merged instances get no automatic surgery elsewhere either: the legacy fixups that used to rewrite imported instances on upgrade and nightly runs — the hardcoded-node-ID SQL rewrites against hostmaster's hosting_context/node/hosting_site tables, the greedy sed "Pre-Fix" rewrites of hostmaster.alias.drushrc.php, the nightly one-shot UPDATE hosting_context SET name=hostmaster SQL, and the nightly _fix_site_readonlymode reset — were all removed. They predate renameaegirhost and, on long-lived instances whose node IDs no longer matched the hardcoded assumptions, caused hostmaster drift rather than fixing it. After a cross-host migration, hostname/topology fixups are exclusively the job of renameaegirhost and the normal hosting machinery — do not expect BOA to auto-rewrite hostmaster contexts or aliases on the next upgrade or nightly run. The retired internal markers (post-merge-fix.pid, hmpathfix.pid, hosting_context.pid, readonlymode_fix.info) need no operator action.

DNS-proxy setup is not automatic: it is the explicit xoct proxy step run on the source, which converts the account's nginx vhosts to proxy templates forwarding to the target IP, removes the migration http-off, and emails the account owner. After DNS is repointed the proxy hop is no longer used.

Downtime per account: minutes to a few hours depending on site sizes. Sequential — one account at a time. xoct uses mydumper/myloader for the DB phase and is cross-version safe — it works between hosts running different Percona versions.

xboa is gone

xboa was the predecessor name. It is fully removed — there is no xboa binary and no xboa fetch entry (BOA fetches xoct, xcopy, and xmass; nothing under the old name). xoct is its functional successor with two improvements: the Aegir DB hostname replacement and post-import task queue are delegated to renameaegirhost (more thorough, 5-pass), and the hardcoded internal-account email exclusion was removed so xoct works correctly when driven by xmass. If muscle-memory still types xboa, an operator-created compatibility symlink is the documented workaround (BOA does not ship one):

ln -sfn /opt/local/bin/xoct /opt/local/bin/xboa

The symlink target must be the real binary at /opt/local/bin/xoct (xoct is fetched to /opt/local/bin only — it is not mirrored into /usr/local/bin); both directories are on PATH, so a bare xboa then resolves.

xcopy — duplicate an account, source stays live

xcopy shares xoct's dispatcher shape but is a non-destructive copy: it does not write http-off on the source, and it does not convert the source to a proxy. Use it to stand up a duplicate of an account on a second host (staging clone of a whole instance, pre-cutover rehearsal) while the original keeps serving.

xcopy pre-mig     <source-fqdn>                 # on SOURCE + TARGET
xcopy transfer    shared <target-ip>            # on SOURCE — DRY plan
xcopy transfer    shared <target-ip> --live     # on SOURCE
xcopy create      o1 <target-ip> [o2]           # on SOURCE
xcopy pretransfer o1 <target-ip> [o2]           # on SOURCE — DRY plan
xcopy pretransfer o1 <target-ip> [o2] --live    # on SOURCE
xcopy export      o1 <target-ip>                # on SOURCE (dumps, no http-off)
xcopy transfer    o1 <target-ip> [o2]           # on SOURCE — DRY plan
xcopy transfer    o1 <target-ip> [o2] --live    # on SOURCE
xcopy import      o1 <target-ip> [o2]           # on TARGET
xcopy noproxy     o1 <target-ip> [o2]           # on SOURCE (skip proxy conversion)
xcopy post-mig    <source-fqdn>                 # on SOURCE + TARGET
xcopy ssl-gen                                   # on TARGET (re-issue certs)

Dispatch verbs: pre-mig, export, create, import, pretransfer, transfer, noproxy, post-mig, ssl-gen. The noproxy verb is the deliberate counterpart to xoct proxy — it finalises the copy without proxying the source. ssl-gen re-issues Encryption on the target for the (now duplicate) account, which is required because the copied account, like any name-changed account, lands with Encryption disabled.

Because the source is never frozen, an xcopy produces a point-in-time duplicate that diverges from the still-live source the moment it completes — it is a copy, not a synchronised replica.

xmass — whole-server move

xmass moves an entire BOA host — every Octopus account, every Solr index, every config — from source to target. The DB transport is xtrabackup snapshot + MySQL GTID replication, not per-account mydumper cycles, so it scales to fleets of large databases where per-account dump/restore would be impractical.

The working steps (init, sync, status, cutover) all run on the source and take the target IP as their argument; replication and rsync flow source → target. post-mig is run on the target after DNS is updated, and the initial pre-mig is run on both source and target.

# Phase 0 — pre-mig, run on BOTH source then target
xmass pre-mig <source-fqdn>

# On SOURCE — install xtrabackup, enable GTID, snapshot,
# restore on target, start the replica
xmass init <target-ip> [--permanent-proxy]

# On SOURCE — rsync platforms/files/configs/Solr (repeat freely);
# DRY plan by default, re-run with --live to transfer
xmass sync <target-ip>                                 # DRY plan
xmass sync <target-ip> --live

# On SOURCE — replication lag + last-sync timestamp
xmass status <target-ip>

# On SOURCE — final cutover (block web, drain lag, promote target,
# renameaegirhost on target, convert source to proxy); the DRY run
# plans storage for all accounts + shared and stops before any
# destructive step
xmass cutover <target-ip> [--permanent-proxy]          # DRY plan
xmass cutover <target-ip> [--permanent-proxy] --live

# On TARGET — finalise after DNS update
xmass post-mig

Dispatch verbs: pre-mig, init, sync, status, cutover, post-mig. The sync and cutover verbs are gated by the DRY/--live model (see Storage-aware transfers and the DRY/--live gate below); one DRY plans all accounts plus shared. Pre-sync is incremental — once init runs, repeat sync over days or weeks until ready; the GTID replica keeps the target's databases current and sync carries the file-system deltas. Whole-server cutover downtime is typically 1–3 hours.

You do not pre-create matching Octopus accounts on the target. xmass restores the full xtrabackup snapshot, brings databases across via GTID replication, and enumerates the accounts dynamically from /data/disk/ — no Octopus instances are required on the target beforehand. xmass init auto-installs the matching percona-xtrabackup-* package and auto-enables GTID on both servers if not already active.

xmass cutover invokes xoct proxy per account during the cutover phase to convert the source vhosts to proxies, plus renameaegirhost on the target for the master and each Octopus account.

--permanent-proxy

--permanent-proxy marks the source as a permanent HTTP proxy after cutover rather than a temporary one pending DNS update. It is not cosmetic:

  • It keeps the migration-proxy realip/CSF trust on the target in place — the post-mig teardown becomes a deliberate no-op (migration_proxy_trust.sh teardown is skipped).
  • It drops a /data/conf/xmass_permanent_proxy.pid flag so notification wording reflects the permanent mode.
  • It prints the permanent mode in the cutover summary.

The flag can be set at init (_PERM_PROXY_FLAG) and overridden at cutover — --permanent-proxy on the cutover command wins over whatever was set at init.

Requirements

  • Identical Percona versions on source and target — GTID replication is version-strict. (This is the key difference from xoct/xcopy, which are cross-version safe via mydumper.)
  • BOA installed on the target at the same release as the source.
  • Root SSH key access source → target, set up by xmass pre-mig.
  • The source IP whitelisted in CSF on the target before init, so xtrabackup streaming and rsync can reach it.

Storage-aware transfers and the DRY/--live gate

DRY by default, --live to act

xoct/xcopy transfer and pretransfer, and xmass sync and cutover, default to a read-only DRY run: the tool resolves the target's attached mount, prints a [DRY-PLAN] line per store, pre-checks disk space, and finishes with [DRY] CLEAN or [DRY] NOT CLEAN, recording the result in a state file — /var/log/boa/xoct.migrate.<oct>_<tgt>.state, /var/log/boa/xcopy.migrate.<oct>_<tgt>.state, /var/log/boa/xmass.migrate.<tgt>.state (xmass keys per target only: one DRY plans all accounts plus shared). An explicit --live is required to perform the migration and is accepted only after a CLEAN dry run for the same account+target (per target for xmass); the clean-dry token is consumed on use, so one dry run can never arm two live runs. --dry/--test are explicit-DRY aliases on all three tools. These are flags, not verbs — only ---prefixed flags are recognised and stripped, so no positional argument is ever mistaken for one.

The gate covers only the file-transfer verbs. ssl-gen, export, import, create, proxy (noproxy on xcopy), pre-mig, and post-mig run ungated on xoct/xcopy; pre-mig, post-mig, init, and status run ungated on xmass. The DB/SQL steps — the mydumper/mysqldump export-import and the xtrabackup/GTID pipeline — are never gated. xmass cutover in DRY mode does the storage-plan pass only, for all accounts plus shared, and stops before any destructive step — no MySQL lock, no downtime.

Storage-aware placement

Store placement follows the target's disk reality, not the source's (_xoct_handle_store, with twins _xcopy_handle_store / _xmass_handle_store):

  • Target has a single attached /mnt mount → a store's contents are mirrored onto it (static/files → <mount>/files/<account>/static/files — using the destination account name, so a rename lands correctly; arch → <mount>/files/system/arch) and the on-target path re-pointed as a symlink — even if root has room.
  • No target mount → the store is de-referenced into a real directory on the target root.
  • A real-dir source stays a real dir on the target root, using the target mount only as a space fallback.

A catch-all sweep applies the same rule to any other /mnt-anchored symlink in the account tree, so none is left dangling on the target. Three conditions are a DENY — the dry run reports NOT CLEAN and --live is refused: a dangling source symlink (attached disk not mounted), a target with multiple /mnt mounts, or a store that fits nowhere. Site-level sites/*/files and sites/*/private symlinks resolve either way — they point at the account-level static/files path, which on the target is a real directory or a symlink onto the mount.

Relocated arch (migratefs)

Once migratefs has relocated /data/disk/arch onto attached storage it is a symlink, and all three tools transfer its contents: the shared sync routes arch through the storage-aware store handler, which resolves the symlink (readlink -f, scoped to this store — not a blanket rsync --copy-links) and materialises it on the target per the rule above — mirrored onto the target's mount at <mount>/files/system/arch with /data/disk/arch re-pointed, or de-referenced into a real /data/disk/arch on the target root. In LIVE mode an arch that cannot be placed aborts the shared sync with an error rather than proceeding without backups. Previously xoct/xcopy rsynced arch as a bare path and delivered a dangling symlink, while xmass skipped it entirely — silently transferring zero SQL dumps and cluster backups. A real (non-relocated) arch keeps the existing behaviour; a dangling arch symlink (attached disk unmounted at migration time) is a DENY in the dry run instead of a silent failure.

The shared sync also refuses an empty target: an empty ssh host resolves to localhost, so the remote mkdir/rm could have run against the source box and deleted its live arch symlink. The target mount is now resolved up-front, and the resolver exits non-zero on an empty target, an unreachable target, or a multi-mount target (xmass already died on an empty target).

Before migrating a host whose arch/static-files was relocated by migratefs, make sure xoct/xcopy/xmass are current on both hosts — BOA auto-updates them via _fetch_versioned; older versions copy or skip the bare symlink.

Solr indices and the space gate

Solr home data (/opt/solr4, /var/solr7/data, /var/solr9/data) — often tens of GB — is folded into the migration disk-space gate. Each tool runs a Solr space check before each Solr transfer: du -sk of the source vs available space at the target destination dir, measured by df on the nearest existing ancestor on the target — so it follows the target's own /var/solrN symlink onto its mount, or lands on root. If it does not fit, the tool prints DENY: <label>: <N>K does not fit at target <dest> — free space and retry, sets NOT CLEAN (so --live is refused until resolved), and skips that transfer. Unknown size or unknown target availability does not block — the measurement fails open. Solr placement is unchanged: a data-only rsync following the target's own Solr setup. Solr is deliberately not routed through the storage-aware store handler, which moves whole directories and would clobber the target's fresh Solr home. Previously Solr was only DRY-wrapped, not space-checked — a dry run would not warn, and a live run would pour the index onto a full target.

How web traffic is frozen — http-off

To stop writes on the source while databases are exported or drained, xoct and xmass write a per-account static/control/http-off.pid. The file's contents are a TTL in seconds — 3600 when set by xoct export, 7200 when set by xmass cutover. global.inc reads the marker and short-circuits every site in that account with a PHP-level 503.

This replaces the old readonlymode mechanism, which is unavailable via the system Drush 8 on Drupal 8+ and is bypassed by some commerce/API code paths. http-off is the automatic freeze; the xoct procedure still recommends an additional manual config_readonly / site_readonly toggle in global-extra.inc as belt-and-braces before the export window — treat http-off as the automatic mechanism, not a total replacement for read-only mode.

At each http-off toggle the nginx fastcgi speed cache is purged box-wide so already cached 200 (or 503) responses don't mask the change; the microcache repopulates within seconds on live sites. The pid is removed when the account is converted to a proxy at xoct proxy / xmass cutover time, so the now-proxied sites return to 200. xcopy never writes http-off — the source is never frozen.

The static/files transfer uses a symlink-safe two-pass helper (_xoct_transfer_static_symlink_safe, mirrored in xcopy as _xcopy_transfer_static_symlink_safe): pass 1 syncs everything under static/ with symlinks preserved (excluding static/files); pass 2 routes static/files through the storage-aware store handler (see Storage-aware transfers and the DRY/--live gate above), so on the target static/files may legitimately be a real directory on root or a symlink onto the target mount — either way neither the account-level static/files nor the per-site sites/*/files symlinks are left dereferenced into broken layouts.

Migration proxy — real-IP recovery and CSF trust

After cutover the source server is converted to a reverse proxy that relays traffic to the new host (xoct proxy per account, driven by xmass cutover), so visitors keep working until DNS is repointed.

The critical wiring is on the target: xmass runs _xmass_setup_migration_proxy_trust before relaying starts. It scps migration_proxy_trust.sh + migration_proxy_realip.sh from the source's /var/xdrago/ to the target's /var/xdrago/, chmod 0755s them, and runs migration_proxy_trust.sh trust <source-ips> [--permanent] on the target. That tool:

  • Writes set_real_ip_from <proxy-ip>; for the nginx realip layer (via migration_proxy_realip.sh), so the new host recovers the real client IP instead of seeing every request come from the proxy.
  • Hard-whitelists the proxy's egress IP in CSF — csf.allow on ports 80 and 443 plus csf.ignore (tagged # migration proxy).

Without this, every relayed request would appear to come from the proxy IP, so every per-client control on the new host (scan_nginx, geo bans, ip_access, the AI guards, CSF/lfd) would key on the proxy and a single ban would blackhole all migrated sites.

The trusted source IPs are derived from hostname -I on the source, filtered to routable IPv4 (loopback 127.* dropped, IPv6 dropped, private IPs kept since the source→target hop may run over a private network). Both validators reject the all-zeros host and any /0 prefix: trusting 0.0.0.0/0 as a realip source would honour the spoofable CF-Connecting-IP header from every peer, collapsing the realip trust boundary (nginx ignores host bits, so 1.2.3.4/0 is 0.0.0.0/0). The path is IPv4-only by design — CSF's whitelist is IPv4 and proxy_pass targets an IPv4 target IP, so an IPv6 proxy-peer line would be stripped and flap, and is dropped deliberately.

The proxy also refreshes its own Cloudflare realip (/var/xdrago/cloudflare_realip.sh) so that for CF-fronted sites it forwards the real visitor — not the CF edge IP — in the CF-Connecting-IP header it hands the new host.

Teardown removes the trust unless --permanent-proxy was passed. Note the asymmetry: xoct proxy converts an account's vhosts to proxy templates but does not itself call the trust scripts — the realip/CSF-trust wiring is xmass-driven. A standalone xoct move to a host that should recover real client IPs needs the trust wired separately.

Which tool when

One site/account, same host       → Aegir Migrate/Clone task  (see 02.site-cloning)
One account across hosts (move)   → xoct
One account across hosts (copy)   → xcopy   (source stays live)
Whole BOA host                    → xmass

xoct, xcopy, and xmass are the mature BOA cross-host tools; prefer them over any legacy manual import. The historical single-site manual remote_import / hosting_remote_import flow is no longer a functional BOA module and is treated as retired — see Discontinued features.

Related

  • Site cloning & in-host migration — the within-host Clone and Migrate tasks and the FastTrack/MyQuick accelerators.
  • Aliases & redirects — the alias/redirect model, relevant when a moved or renamed account changes its canonical hostname.
  • Database (MySQL/Percona) — mydumper/myloader internals and the Percona-version constraints that gate xmass.
  • Security & isolation — CSF, ip_access, and the realip layer the migration-proxy trust feeds into.
  • Abuse Guard — scan_nginx and the geo/AI bans that key on the recovered client IP after a migration proxy is trusted.
  • See the Reference appendix for the consolidated control-file and command tables.