Solr core lifecycle (manage_solr_config.sh)

/var/xdrago/manage_solr_config.sh is the runtime Solr-core reconciliation daemon (~1415 LOC). It walks every site, reads each boa_site_control.ini, syncs the core set, optimises indices, archives orphans, and runs a read-only health check. This is the script behind the "changes take a few minutes to take effect" behaviour: the INI-driven reconciliation runs on every four-minute pass; the heavier maintenance is self-throttled inside that invocation.

Scheduled runs

Invoked directly from the root crontab every four minutes — not via minute.sh:

*/4 * * * * /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/manage_solr_config.sh >/dev/null 2>&1

The whole pass is skipped during a BOA install/upgrade: at startup it scans for a running autoinit/automini/barracuda/boa/octopus process and the /run/octopus_install_run.pid, /run/boa_run.pid, /run/boa_wait.pid lock files, and sets _protectedRun=TRUE to exit without touching anything.

Force an immediate pass as root:

bash /var/xdrago/manage_solr_config.sh

The forced run still honours the 6 h / 12 h throttles; remove the relevant sentinel first to force the maintenance (below).

What runs in each pass

Pass start (every 4 min, root crontab)
  ↓ install/upgrade running → exit (_protectedRun)
For each Octopus instance under /data/disk/:
  For each site:
    Read boa_site_control.ini → target Solr-core state
    Compare to on-disk core
      core needed but missing      → create
      core unneeded (INI removed)  → delete within 15 min (solr delete, rm solr.php)
      integration-module changed   → re-deploy config family
      custom_config=NO & update=YES → refresh schema/solrconfig
    Sync sites/<domain>/solr.php
  ↓
Orphan cleanup (_cleanup_orphan_cores)  — throttled 6 h, after site reconcile
  ↓
Per-core health check (_check_solr_core_health) — every pass, read-only
  ↓
Index optimise (_run_optimize_if_due)   — throttled 12 h

The load gate _O_LOAD_MAX (_CPU_TASK_RATIO * 100, default _CPU_TASK_RATIO=3.1) lets a busy host skip per-site work, so a core change can be deferred a pass under high load.

Core deletion (INI removal)

Deletion is explicit, not age-based: once the site INI no longer names a recognised solr_integration_module, a subsequent pass issues solr delete against every version's data dir for the computed core ID (and its _Old_/_Legacy_ aliases) and removes sites/<domain>/solr.php. BOA cleanly deletes the core within 15 minutes of the line being commented out (a four-minute pass reconciles it, subject to the load gate). The associated Drupal module stays in the codebase; disable it manually if unneeded.

Orphan-core cleanup (3-tier, 6 h)

Orphan cleanup is fully automated — you do not normally archive cores by hand. On each pass (throttled to once per 6 h via /var/backups/solr/.orphan_cleanup_last_run.pid, skipped during install/upgrade), _cleanup_orphan_cores classifies every oct.* core on disk and retires those nothing references. It runs after per-site reconciliation so a just-recreated managed core is on disk before classification.

Cores are never deleted. A qualifying core is first unloaded from Solr's registry — action=UNLOAD&deleteIndex=false&deleteDataDir=false&deleteInstanceDir=false — then the directory is moved to /var/backups/solrN/<YYYYMMDD-HHMMSS>-<core>/.

Classification

The active-core set is built from the host's nginx vhosts and Aegir drush aliases, but _build_active_core_set registers a vhost only for an enabled, non-hostmaster site: it skips any disabled/parked vhost (one carrying the "Do not reveal Aegir front-end URL here" marker) and any hostmaster (aegir/distro) vhost before recording it. A disabled, parked, or hostmaster site therefore registers no vhost, so its core is treated as having no vhost and lands in Tier 1. Each core is sorted into one tier — Tier 2 requires vhost and alias, everything else falls to Tier 1:

Staleness is measured on data/index/ mtime (Lucene segment-commit time), falling back to data/ mtime; a core with no data/ directory was never indexed and counts as immediately stale.

Actively-managed cores are age-immune

A core whose site has solr_integration_module set is never archived on age alone, even if its index looks stale — _check_sites_list actively manages it, and archiving would let the next pass recreate it empty and lose the index. These log ORPHAN-SKIP.

Log markers (one per core, in the pass log)

Recovering an archived core

The core was unloaded before the move, so Solr has no registry record — re-add with CREATE (not RELOAD), which re-registers the existing directory without touching index files:

port=9099                          # 9077 for Solr 7
core="oct.o1.example.com"
ts="20260418-222802"               # timestamp prefix from the backup dir name
bkp="/var/backups/solr9/${ts}-${core}"
dest="/var/solr9/data/${core}"     # /var/solr7/data/ for Solr 7

mv "${bkp}" "${dest}"
chown -R solr9:solr9 "${dest}"     # solr7:solr7 for Solr 7
curl "http://127.0.0.1:${port}/solr/admin/cores?action=CREATE&name=${core}&instanceDir=${dest}"

Per-core health check

_check_solr_core_health runs every pass (not throttled), once per running instance (solr7 on 9077, solr9 on 9099, gated on /etc/init.d/solrN). It queries the admin STATUS API read-only via python3 and makes no changes — it is the diagnostic feeding the optimise/orphan decisions. It logs:

• HEALTH-ERROR — no response from the Solr instance on its port (daemon down or unreachable), or the STATUS JSON failed to parse. The most operationally useful marker — it means the health check never got a reading.
• HEALTH-INIT-FAIL — a core reporting initFailures (failed to load).
• HEALTH-WARN — segment count > 50, deleted docs > 20% of maxDoc, or index size > 500 MB.
• HEALTH-INFO — N cores registered (or no cores registered), the per-instance roll-up line.
• HEALTH-SKIP — python3 not installed, check skipped.

Throttle sentinels and forced maintenance

There is no per-site state cache (older docs referenced /var/xdrago/.solr-state/<sitename>.state; that path does not exist). The real state is the throttle sentinels:

Force an immediate orphan sweep or optimise on the next run by removing its sentinel:

rm -f /var/backups/solr/.orphan_cleanup_last_run.pid   # force orphan cleanup
rm -f /var/backups/solr/.optimize_last_run.pid         # force index optimise
bash /var/xdrago/manage_solr_config.sh

Logs

Each pass writes its full stdout/stderr (the === section markers and all ORPHAN-* / HEALTH-* lines) to a timestamped file:

/var/backups/solr/log/solr-<YYMMDD-HHMMSS>.log

Files older than a day are pruned at the start of each pass. The cron line discards cron-level output to /dev/null, so the per-pass file is the only persisted record — there is no /var/log/boa/solr_management.log or /var/xdrago/monitor/log/solr-*.log (older docs cited those; no code path writes them).

tail -f "$(ls -t /var/backups/solr/log/solr-*.log | head -1)"

Custom-config interaction

With solr_custom_config = YES, the daemon skips re-deploying schema.xml/solrconfig.xml; operator config in sites/<domain>/files/solr/ stays in place; all other lifecycle behaviour (deletion on INI removal, solr.php refresh, optimisation) still runs. To deploy a Drupal-generated config without permanent lock-in, set solr_custom_config = NO for one pass, then restore YES (see Overview and tuning).

Failure modes

"Create core" never completes — Solr daemon down. service solr9 status; tail -50 /var/solr9/logs/solr.log; restart, and the next pass retries.

Core not deleted after INI removal — deletion fires within 15 minutes once the INI no longer names a recognised module. If still present after that, force a run and read the newest pass log; under load over _O_LOAD_MAX the site may not have been reached yet.

Orphan core not archived yet — cleanup is deliberately patient: a core is archived only once past its tier threshold (14 d unreferenced, 60 d vhost+alias), the sweep runs at most every 6 h, and solr_integration_module-managed cores are age-immune. Read the newest pass log — the ORPHAN-FRESH/ORPHAN-SKIP/ ORPHAN-CANDIDATE lines state exactly why each core was kept or archived. Force a sweep by removing the throttle sentinel. Archived cores are moved to /var/backups/solrN/, never deleted, so a wrongly-archived core can always be restored.

Source location

Deployed: /var/xdrago/manage_solr_config.sh (read-only). Source: aegir/tools/system/manage_solr_config.sh — modify there to upstream a fix.

Related

• Overview and tuning — the INI variables this daemon enforces and the optimisation internals.
• Solr 9 modules — the module set the daemon keeps deployed.
• Reference appendix — Solr site-INI variables.
• Discontinued features — the Solr-4-era manual core-management flow this daemon replaced.