Nightly automation — updatesymlinks

updatesymlinks is the scheduler / orchestrator of the files-symlinking subsystem: a root-only wrapper (/opt/local/bin/updatesymlinks) around autosymlink that adds everything a safe unattended run needs — Aegir task-queue pausing, single-instance locking, heavy-task and load guards, a per-night retry stamp, and email reporting. It sources /root/.barracuda.cnf and self-exits on a box that is not fully installed yet (missing /var/log/boa/reset_no_new_password.pid). What autosymlink itself converts, and how, is Tools reference's job; this page covers the automation around it. Source: aegir/tools/bin/updatesymlinks.

Sub-modes

The sub-mode argument is position-independent; --debug combines with any of them.

Both gates ship as NO in the cnf template (lib/settings/barracuda.sh.cnf:1031,1040), and the script itself defaults them to NO when absent from /root/.barracuda.cnf (updatesymlinks:65-66) — boxes whose cnf predates the variables stay opted out. Which system classes get them seeded to YES, and the rest of the variable story, live on Configuration.

The schedule — one always-present cron line

The root crontab ships a single entry (aegir/tools/system/cron/crontabs/root:29):

47 0-5,22,23 * * * /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /opt/local/bin/updatesymlinks --auto-fix >/dev/null 2>&1

• Hourly at :47 through the night — 22:47, 23:47, then 00:47–05:47. The :47 offset keeps it clear of the 6-hourly duplicity backups (:00) and the nightly backup/owl/upgrade cluster.
• Always present — the line is installed regardless of the toggles and self-exits unless at least one is YES, so enabling or disabling the automation is purely a .barracuda.cnf edit, never a cron edit.
• One line serves both opt-ins — with _AUTOSYMLINK_NIGHTLY=YES it runs the full pause + two-step apply; with only _ORPHAN_FILES_REPORT=YES it runs the read-only orphan report alone (no pause). So NIGHTLY=NO + ORPHAN=YES still reports every night.

Per-night stamp and hourly retry

Each hourly firing is an attempt, not a run. The night is identified by a noon-anchored date (date --date '12 hours ago'), which collapses the whole 22:00–05:59 window — evening plus the early hours past midnight — onto one id. The first un-blocked hour does the work and writes that id to /var/log/boa/autosymlink.nightok.stamp; every later hour that night compares the stamp and exits immediately, a cheap no-op. A blocked attempt (heavy task, or a provision task that would not drain) deliberately leaves the night un-stamped, so the next :47 window retries. Remove the stamp file to force a re-run before the window rolls over.

Heavy-task skip

Before doing anything, an attempt checks for work it must not race and, if any is found, exits and lets the next hour retry:

• a running autosymlink, autoupboa, barracuda, boa, or octopus process;
• the flags /run/octopus_install_run.pid, /run/boa_run.pid, /run/boa_wait.pid, /run/max_load.pid, /run/critical_load.pid (the last two are the load-control high/critical markers);
• any provision or duplicity process.

The same check is repeated after the queue pause below, so a heavy task that started during the grace window still cancels the apply. See Cron cadence & idle-load throttle for how the rest of the nightly machinery shares this window.

Queue pause and drain (apply path only)

The apply path must not convert a site's files mid-task, so before touching anything it holds the Aegir task queue still:

1. Pause — write its PID into /run/boa_queue_stop.pid (skipped if one is already present — the concurrent holder keeps it, and this run never owns it), the dedicated self-healing queue-stop file honoured by runner.sh (parent exit + per-child skip). It is designed to never freeze the queue for good: /run clears on reboot and clear.sh purges the file once the recorded owner PID is gone. It replaced an earlier .pause_tasks_maint.cnf-based pause, which was not self-healing. Mechanics and observation notes: Task queue → Pause / resume.
2. Grace — sleep _AUTOSYMLINK_PAUSE_GRACE seconds (default 240, overridable in /root/.barracuda.cnf; 0 skips the wait), so a runner.sh already mid-dispatch when the pause landed cannot slip a task into the window.
3. Drain — poll for provision processes, bounded at 12 × 5 s. If one is still active after grace + drain, the run bails without applying; the night stays un-stamped and the next hour retries.

On exit the stop file is removed only if this process created it and still owns it (the recorded PID is verified), so a pause window opened by a concurrent maintenance operation is never torn down. The same stop-file mechanism is used by the nightly backups relocation in night/10-account.sh — see Backups on a static filesystem. The orphan-report path deliberately never touches the pause file, so a concurrent apply pass keeps its own window intact.

Re-entrancy is guarded by the shared lock.inc single-instance lock where deployed, with a legacy pgrep count fallback (more than two instances exits, logged to /var/log/boa/too.many.log).

The two-step apply and email

Once quiescent, the run executes autosymlink --batch-if-clean — dry-run first, apply only if the dry-run came back clean, in one invocation — then always follows with autosymlink --report, folding the orphan check into the same night. Outcomes are read back from /var/log/boa/autosymlink.state (written by autosymlink: _LAST_ACTION, _LAST_APPLY_COUNT) and reported by email to _MY_EMAIL only when something happened:

Each notice is two messages: an Actions Report (the permanent action log) and a Details Report (the full run output). What an orphan is, and what the auto-fix does with one, is covered in Orphans & archiving.

Logs

--debug — why did nothing happen?

Every early exit above is silent by design (this is a cron job). Add --debug (or -d, any position) to any invocation to have it print the reason on stdout — with zero behaviour change:

updatesymlinks --auto-fix --debug
#  updatesymlinks[debug]: not acting — both opt-ins are off (…); enable one in /root/.barracuda.cnf
#  updatesymlinks[debug]: not acting — already completed for this night (stamp …); remove that file to force a re-run
#  updatesymlinks[debug]: not acting — a heavy task is running, retry next hour: duplicity(1)

When a run does proceed, the same flag traces each stage (pause, grace, drain, batch outcome, stamp), including the case where it ran and simply found nothing to convert. This is the first tool to reach when the nightly automation "does nothing".

Related

• Overview — the store layout, event table, and safety model the nightly run operates on.
• Tools reference — autosymlink itself: --batch-if-clean, --report, and the manual verify commands.
• Configuration — _AUTOSYMLINK_NIGHTLY, _ORPHAN_FILES_REPORT, _AUTOSYMLINK_PAUSE_GRACE defaults by system class, and the subsystem's control files.
• Orphans & archiving — what the folded-in orphan report detects and what the auto-fix archives.
• Backups on a static filesystem — the other nightly mover sharing the /run/boa_queue_stop.pid pause.
• Task queue — how runner.sh honours the queue-stop file and why it is self-healing.
• Reference appendix — where the variables above are indexed.