SSL operations — LE renewal, custom certs

BOA issues and renews Let's Encrypt certificates automatically through the bundled dehydrated client driven from the nightly owl.sh run. This page covers the on-disk cert tree, the actual renewal cadence, renewal-failure reporting, and the procedure for replacing an LE cert with a custom OV/EV/wildcard cert behind a don't-overwrite marker.

Cert tree on disk

Per-site LE material lives under the frontend's tools/le/certs/<site>/:

<aegir-root>/tools/le/certs/<site>/
  privkey.pem      ← symlink (LE-managed)
  fullchain.pem    ← symlink (LE-managed)
  chain.pem        ← symlink (LE-managed)

<aegir-root> is /var/aegir on the Master or /data/disk/<USER> on an Octopus Satellite. The don't-overwrite markers live in a sibling tools/le/.ctrl/ directory.

Automatic LE renewal

Renewal runs from the daily cron, not weekly. The crontab fires the owl.sh orchestrator once per day at 15 4 * * * — bash /var/xdrago/owl.sh under nice/ionice (cron/crontabs/root:24). owl.sh replaced the retired daily.sh; a host not yet through that transition may still name daily.sh in its root crontab, so crontab -l tells you which agent a given box runs. The nightly dehydrated --cron --ipv4 --preferred-chain 'ISRG Root X1' invocation lives in two night workers under /var/xdrago/night/:

  • 10-account.sh — _le_hm_ssl_check_update() (:394) renews each account's Hostmaster/frontend cert, running dehydrated as the account user via su (:435-441);
  • 20-sites.sh — _le_ssl_check_update() (:1742) renews every LE site cert.

dehydrated --cron re-issues any cert with less than 30 days to expiry; certs with more runway are left untouched, so the daily pass stays cheap.

A separate, roughly-monthly path force-rebuilds certs: each worker computes _DOM=$(date +%e) and a random _RDM=$((RANDOM%25+6)), and adds --force when _DOM equals _RDM (or when the per-account static/control/force-ssl-certs-rebuild.info control file is present). This applies to both the account frontend cert (10-account.sh:420-433) and every per-site cert (20-sites.sh:1791-1803). The --force is further gated per domain: it is added only when no log/ctrl/site.<domain>.cert-x1-rebuilt.info marker exists; the marker is created on force and purged after 14 days (_PURGE_CTRL=14, 10-account.sh:657-661), so a forced re-issue cannot recur within ~14 days.

Do not confuse this with _SSL_FORCE_REINSTALL=YES in /root/.barracuda.cnf — that is not a certificate operation. It is a one-shot toggle that forces an OpenSSL source rebuild on the next barracuda run (sets _SSL_LEGACY_INSTALL_REQUIRED / _SSL_MODERN_INSTALL_REQUIRED=YES) and cascades _SSH_FORCE_REINSTALL=YES, _NGX_FORCE_REINSTALL=YES and _PHP_FORCE_REINSTALL=YES (lib/functions/system.sh.inc:5043-5049); it is scrubbed from the live cnf after use (system.sh.inc:1592-1596) — set it, run barracuda, and it is consumed. To force certificate re-issue, use the force-ssl-certs-rebuild.info control file above. See Barracuda cnf for the full *_FORCE_REINSTALL family.

LE issuance behind HTTP Basic Auth

Issuance and renewal work on sites protected with HTTP Basic Auth. The hosting_le_vhost injector emits the ACME challenge location with an explicit exemption (hosting_le_vhost.drush.inc:85-90):

location ^~ /.well-known/acme-challenge {
  allow all;
  alias <aegir-root>/tools/le/.acme-challenges;
  try_files $uri 404;
  auth_basic off;
}

The auth_basic off; (:89) exempts HTTP-01 validation from the site-wide Basic Auth, so password-protected staging/demo sites can enable and renew LE certificates without lifting the password. Before this shipped (bundled with 5.10.3; hosting_le issue #1790, commit a679a7b, contributed by Naurisr), the 401 on the challenge path blocked issuance and renewal.

Renewal-failure reporting

Post-5.10.3, each per-account night worker parses its own account log (/var/log/boa/daily/acct-<user>-<NOW>.log) at the end of its run (_le_account_report, 10-account.sh:475, invoked at :306) and extracts every failed renewal — the failing site, the certificate names, and the ACME error mapped to a plain-language cause (DNS NXDOMAIN, domain pointing elsewhere, key-auth mismatch, connection timeout). Two mails come out of that parse:

  • Operator report — the server admin (_MY_EMAIL from /root/.barracuda.cnf, exported to the workers as _ADMIN_EMAIL) gets the full per-account detail every night unless _INCIDENT_REPORT is OFF (gate at 10-account.sh:504-505; values normalized in owl.sh:76-90 to OFF/ALL/MINI/CRIT, default MINI; legacy NO maps to OFF, YES to MINI). A catch-all also carries non-validation ACME/backend errors — urn:*acme:error tokens including rateLimited / serverInternal, Traceback, Drush command terminated abnormally (10-account.sh:478-495).
  • Client notice — the affected account's _CLIENT_EMAIL (from its .octopus.cnf) gets an actionable notice — fix the DNS, disable Encryption, or drop the dead alias — throttled to once per 7 days per failing site (_throttle=7 at 10-account.sh:494; marker log/ctrl/le-notify.<site>.info, purged after 14 days at :663-664). The notice sets Reply-To to the account owner, falling back to the server admin, so replies do not bounce off root@<host> (10-account.sh:560-570).

Client notices are on by default: disable with _LE_CLIENT_NOTIFY=NO globally in /root/.barracuda.cnf (shipped default YES, barracuda.sh.cnf:1278) or per account in /root/.<user>.octopus.cnf — the per-account value wins, and only a literal NO disables (_le_client_notify_on, 10-account.sh:458-463). Because each worker reads only its own account's log and its own _CLIENT_EMAIL, a notice cannot be cross-attributed to another account's client. The _LE_CLIENT_NOTIFY config-table entry lives on Barracuda cnf.

LE-skip keywords

provision_hosting_le.drush.inc skips LE issuance (and self-signs instead) for hostnames whose label matches .temp./.tmp./.temporary., .test./.testing., .dev./.devel., or .nodns. (the .dev/.test skip is suppressed when the dev-ctrl flag is set). Note this roster does not include stage/staging — a myapp.staging.example.org gets a real LE cert. (The stale stage|staging entry survives only in the retiring hosting_le README, not in the code.)

Replacing an LE cert with a custom cert

Use a custom cert only when LE cannot issue what you need: a paid wildcard or multi-org SAN cert, an OV/EV cert required for compliance, or an internal-only site that cannot answer LE HTTP challenges. Otherwise stay on LE.

The procedure replaces the three LE-managed symlinks with regular files and adds a per-site marker so the daily renewal pass leaves them alone.

1. Create the don't-overwrite marker

touch <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid

Provision reads this marker (dont-overwrite-<site.uri>.pid under tools/le/.ctrl) and treats the cert as immutable — without it, the next Verify or daily renewal re-issues an LE cert over your files.

2. Replace the three PEM files

cd <aegir-root>/tools/le/certs/foo.example.com/

rm privkey.pem
# place your private key as privkey.pem (a regular file, not a symlink)

rm fullchain.pem
# place leaf cert FIRST, then every intermediate to (but not including) the root

rm chain.pem
# place ONLY the intermediates (no leaf, no root)

fullchain.pem is the leaf plus the intermediate chain; chain.pem is the same intermediates without the leaf prepended. The filenames must be exactly privkey.pem / fullchain.pem / chain.pem.

3. Ownership and permissions

chown aegir:aegir privkey.pem fullchain.pem chain.pem
chmod 600 privkey.pem
chmod 644 fullchain.pem chain.pem

4. Re-emit the vhost

drush @<site-alias> provision-verify

Verify regenerates the vhost referencing the cert files and reloads Nginx.

Confirming the custom cert is live

openssl s_client -connect foo.example.com:443 -servername foo.example.com < /dev/null 2>/dev/null | \
  openssl x509 -noout -issuer -subject -dates

issuer= should name your CA (e.g. O=DigiCert Inc), not O=Let's Encrypt.

Renewing a custom cert

BOA never touches a cert behind a dont-overwrite-<site>.pid marker, so custom-cert renewal is entirely manual — track the expiry yourself. To renew, overwrite the three PEM files in place and run drush @<site-alias> provision-verify (or service nginx reload). Leave the marker in place.

Reverting to Let's Encrypt

  1. Remove the marker: 
    rm <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid
  2. Optionally clear the cert files (BOA re-issues anyway): 
    rm <aegir-root>/tools/le/certs/foo.example.com/{privkey,fullchain,chain}.pem
  3. Run drush @<site-alias> provision-verify. BOA re-requests an LE cert and restores the symlinks once issuance succeeds.

Disabling Encryption cleans up the LE cert dir

When a site's Encryption is disabled and no dont-overwrite-<site>.pid marker exists, the following Verify tidies up the LE material: it runs dehydrated --cleanup and then symlinks -dr over the site's cert directory to strip the dangling LE-managed symlinks. That is the whole disable path — it does not remove the per-site .ctrl/ control files (forced-renewal-02-<site>.pid, demo-<site>.pid, live-<site>.pid, <site>.pid); those markers persist on disk after Encryption is disabled (provision_hosting_le.drush.inc:97-109). The dont-overwrite-<site>.pid immutable marker gates the whole cleanup: when it is present the disable branch is skipped entirely, so a manually installed certificate stays protected. Clone and migrate operations auto-disable Encryption on the target name; see Site cloning ("TLS / Encryption auto-disable").

Wildcard custom certs

A true wildcard (*.example.com) covers every subdomain, but BOA's per-site cert directory model does not natively express a wildcard — place the wildcard PEM files in each covered site's certs/<site>/ directory, each behind its own don't-overwrite marker. A host-level shared-cert arrangement via a custom include (Custom rewrites & location blocks) is possible but outside BOA's managed cert path.

Migrating off the legacy dedicated-IP SSL model

For sites still on the deprecated dedicated-IP xoct ssl-gen custom-cert model, the move to LE goes:

  1. Disable HTTPS redirects on the affected sites.
  2. Repoint DNS at the default instance/server IP (if you were on dedicated IPs).
  3. Remove any leftover HTTP/HTTPS proxy vhosts from /var/aegir/config/server_master/nginx/pre.d/ and reload Nginx.
  4. touch ~/static/control/ssl-live-mode.info and wait five minutes.
  5. Enable SSL for those sites.

Related