Building & testing BOA changes

There is no CI that runs your change for you and no staging fleet you push to. Proving a BOA change holds is manual and box-shaped, and it runs in four gates, cheapest first: lint (parse + shellcheck + php -l), the edgetest harness (one command that pass/fails the edge-policy stack), a disposable VM (the only place environment-dependent breakage shows), and a serial-bump smoke check (did the box you changed actually re-fetch the new tool). This page is the test discipline; the worked drush-filter verification (an end-to-end Gate-3/4 run of a security-sensitive backend change) is the other leaf in this topic.

The lint gates below are summarised here as when to run them; the exact forbidden-syntax list, the runtime-divergence traps, and the ShellCheck annotation convention live on Code style & conventions. The _fetch_versioned stamp/serial machinery the last gate exercises lives on The serial & fetch pipeline. This page ties them to a test loop.

Gate 1 — lint, before anything reaches a box

Lint is the only gate that runs on your laptop, so it runs on every touched file before a commit. It catches parse errors and the forbidden-syntax class; it catches nothing about runtime behaviour (that is Gates 2–4).

Bash — every touched .sh, .sh.inc, .sh.txt:

bash -n path/to/file.sh          # parse gate
shellcheck path/to/file.sh       # clean, or annotated (see code-style leaf)

Intentional violations are annotated at the line, never suppressed globally — the tree's convention is a # shellcheck disable=SCxxxx # <why> comment, e.g. the two on _fetch_versioned itself (BOA.sh.txt:1071,1091) that document _pthLog/_tRee/_xSrl/_crlGet as BOA-provided globals and the intentional _crlGet flag-string split.

PHP — every touched Provision, Hosting, hosting_*, or Drush-fork file must parse across the 5.6-through-8.5 span, because the same backend file runs under a 5.6 CLI on legacy boxes and an 8.x CLI on current ones:

php74 -l path/to/file.inc         # nearest-old floor available locally
php85 -l path/to/file.inc         # the ceiling

The trap is that php74 -l proves parse-safety only on 7.4+. Syntax that fatals on 5.6 — ??, arrow fn(), typed properties, <=>, match, nullsafe ?-> — is accepted by 7.4/8.x, so a local lint that passes does not clear the 5.6 floor. There is no local php5.6; a real php5.6 -l on a legacy or transitional box is owed before shipping backend PHP, and for credential/DSN/config/value-parsing paths the runtime result on 5.6 must be reasoned through, not just the parse. The full forbidden list and the runtime-divergence traps (== juggling, parse_url, count(NULL), is_numeric('0x1A')) are on the Code style leaf.

Lint clears syntax. It cannot see a su without - that changes HOME, a realpath that resolves differently under the real Drupal-root cwd, or a ==-juggle that flips on 5.6. Those are Gate 3.

Gate 2 — edgetest, the edge-policy harness

edgetest (aegir/tools/bin/edgetest) is BOA's one automated behaviour test: a single-command pass/fail checker for the whole edge-policy stack — AI-bot UA classification, Cloudflare realip, the csf→nginx web-ban mirror, and the per-site AI/IP access controls. It is deployed like any other BOA tool, via _fetch_versioned inside _update_agents (serial-gated in BOA.sh.txt), landing at /opt/local/bin/edgetest and symlinked into /usr/local/bin/edgetest — so on any SKYNET-active box it is just edgetest on the PATH. It prints a coloured PASS/FAIL/WARN/SKIP line per check plus a summary, and exits 0 iff every critical check passed (non-zero otherwise) — so it drops straight into a test script.

What it exercises

Two modes, selected by flag:

Read-only phases (both modes where applicable), each a _phase_* function:

• presence — the four generator tools present under /var/xdrago/ (ip_access, ai_policy, nginx_deny, cloudflare_realip), their cron entries, nginx configtest OK, nginx running.
• realip configuration — CF ranges file non-empty, real_ip_header CF-Connecting-IP and set_real_ip_from actually in the running config (nginx -T), not just on disk.
• AI matrix — the UA classification table: GPTBot/Google-Extended (forged)/Perplexity-User (evasive) and secret paths (/.env, /.git/config) must block (444); OAI-SearchBot/ChatGPT-User/ Google-Agent/OAI-AdsBot/a real browser must be allowed. In local mode it first confirms nginx -T actually has a server_name for --site, else a --resolve to 127.0.0.1 just hits the default server and the matrix is meaningless.
• rate limiting — a 25-request OAI-SearchBot burst should show some pass and some throttled-with-444 (the search zone is 1 r/s, limit_req_status 444), while a browser burst must be 0/25 throttled (browsers are never charged to an AI zone).
• fragments / ban wiring / regression — per-site control fragment dirs, the geo $remote_addr $is_banned map loaded in the running config, and spot-checks (netstat SSH anti-lockout source, IPv4-only csf.allow).

The classifier is deliberate about what is not a policy result: a 444 (guard/throttle) reads to curl as code 000 and is scored a block; a 5xx (backend/upstream error) and a 403 (ip_access deny) are reported WARN/inconclusive, never as a pass or fail. Scheme is auto-detected — it probes http, and switches to https only when the site enforces it (an http request 301/302s to https), so a test VM with a no-real-SSL self-signed proxy does not 5xx the run; --http/--https force it.

--full — the state-changing proofs (test VM only)

--full adds the mutating phases that actually prove enforcement end to end: the realip+ban proof (trust localhost, csf -td a sentinel IP, mirror it in via nginx_deny.sh, then request as the banned IP through a CF-Connecting-IP header and confirm the block bites), the per-site AI toggles (train-allow, evasive-allow opt-in then prune), the ip_access input-validation + anti-lockout proof (a bad octet 192.168.1.300 is skipped, not fatal; `127.0.0.1

• deny allpresent;configteststill OK), and a generator idempotence check (a secondai_policy.sh` run must be a no-op — the change-gate holds).

--full requires root and refuses to run otherwise. Every artifact it creates self-cleans: an _emergency_cleanup function is armed with trap … EXIT (edgetest:98,114) so a csf -tr of the sentinel IP, removal of the # edge-policy-test-tagged control lines, and a regenerate+reload happen even on Ctrl-C. It is still a mutating run against real csf and control files — run it on a disposable VM, never production. The read-only default and --remote create no artifacts and never touch csf or a control file (even a no-match sed -i would rewrite a file and bump its change-gate).

How to run it

# read-only, on the box that hosts the site
edgetest --site example.com --oct o1

# + state-changing proofs, as root, on a disposable VM
edgetest --site example.com --oct o1 --full

# probe a site hosted on ANOTHER box, from this box's IP (real DNS)
edgetest --site example.com --remote

edgetest --help          # options (parsed from the script header)

Run --remote from a whitelisted box (expect allowed) and a non-whitelisted one (expect a 403 ip_access deny) to prove per-site IP access from both vantages. The manual end-to-end checklist edgetest automates — including the two things it deliberately does not (the realip rewrite as seen from a real external client, and the configtest rollback backstop) — is docs/AI-POLICY-TESTING.md in the BOA tree.

Extending it

Add a coverage case by writing a _phase_<name> function that emits results through the existing _pass/_fail/_warn/_skip helpers (they keep the running tallies the summary and exit code read), then wiring it into the run block near the end of the script. Follow the established shape:

• Probe over HTTP with _http "<ua>" "<path>" (it returns "<code> <curl_exit>" and honours the detected scheme + local/remote resolve); classify the result with _blocked / _unreachable / _server_error / _denied rather than bare status comparisons, so a 444-as-000 and a proxied 5xx are scored the way the rest of the harness scores them.
• A mutating phase must guard on [[ "${_FULL}" -eq 1 ]], be root-gated at the call site, and register its teardown in _emergency_cleanup so a Ctrl-C mid-run still leaves the box clean.
• Bumping the tool ships the change: edgetest's own fetch serial is at BOA.sh.txt:1422 and follows the same down-counting rule as every other tool (see the serial pipeline).

Gate 3 — the disposable-VM gate

The rule the tree enforces on itself: prove a change on a throwaway VM built from a public dev branch before it reaches a production branch. Changes land in the tree "unit-tested and ShellCheck-clean but explicitly not yet VM-tested at commit time", and the operator docs for the riskier reapers say to exercise them "in dry-run and on a disposable VM before enabling on production." The public release model makes the from where precise: build the VM from 5.x-dev (or -base/-edge), never from a 5.x-pro*/5.x-lts* branch — a tag on pro/lts triggers SKYNET self-update across the production fleet, so an unproven change must never reach those branches.

The gate is a real, throwaway box and not a laptop because the failures it catches are environmental and cross-version, exactly the class lint and edgetest-on-your-workstation cannot see:

• a su without - that changes HOME; a realpath that resolves differently under the real Drupal-root cwd; an ICU/intl or Percona soname mismatch; a permission or ownership bit only set by the real installer.
• the 5.6 end of the span: with no local php5.6, a runtime claim about 5.6 is only ever confirmed on a box that runs the 5.6 CLI — a disposable legacy/transitional VM is where that confirmation happens.
• the real task path: for an Aegir change, run the actual hosting task, not just the CLI it wraps — the Provision/Hosting bootstrap, drush context and queue behaviour only exist inside a task.

A minimal loop:

1. land the change where a test box installs it the production way:
   - branch-borne change  -> push to a public dev tree the box installs from
     (Aegir source is branch-borne only under _DL_MODE=GIT; the default
     _DL_MODE=BATCH box rebuilds from the staged bundle, not your branch)
   - tarball-borne change -> bump the pin + stage the tarball on the mirror
     (the classic Drush 8 fork is pin-borne in every mode)
2. build a disposable box: full Barracuda install, then one Octopus instance,
   from the public dev branch
3. run the box's own upgrade path so it fetches your change as production would
4. exercise the behaviour by hand:
   - edge-policy change -> edgetest --full   (Gate 2)
   - task-path change   -> run the real Aegir task
   - 5.6 runtime claim  -> re-run on a box with the 5.6 CLI
5. tear the box down

Retired build/test mechanics that used to live in old trees are on Discontinued features — check there before documenting anything you find only in an old branch.

Gate 4 — the serial-bump smoke check

A change to a serial-gated BOA tool only reaches a box if the tool's fNN serial was bumped in the same commit — otherwise the box's stamp still satisfies the gate and it keeps the old copy silently. So the last thing to verify is not that the code is right but that a box actually re-fetches it.

The mechanism (full detail on the serial pipeline): a bump is a decrement — serials count DOWN (f99 → f98), because _fetch_versioned clears the tool's old stamp glob and writes a fresh one on every successful fetch, so any changed serial invalidates the box's gate. A tool fetched by both _update_agents and _update_boa_tools (the ~30 shared core tools) must have the serial bumped in both call sites together — a divergent serial makes each run wipe the other's stamp and refetch every cycle.

To smoke-test the re-fetch on a disposable box:

# 1. before the update, note the tool's current stamp (empty if never fetched)
ls /var/log/boa/<tool>.ctrl.*.pid

# 2. run the box's update path (the production fetch route)

# 3. confirm a NEW stamp at the bumped serial exists, and the tool changed
ls /var/log/boa/<tool>.ctrl.*.pid          # serial in the name = the new fNN
md5sum /opt/local/bin/<tool>               # differs from the pre-update copy

The stamp is /var/log/boa/<tool>.ctrl.<serial>.<tree>.<xSrl>.pid (_pthLog=/var/log/boa, BOA.sh.txt:74,1072); the <serial> token in the filename is the fNN the box last fetched at. If the stamp still shows the old serial after an update, the bump did not ship — the tree's copy and the box disagree. Two failure signatures to recognise:

• stamp unchanged, tool unchanged — you edited the tool but forgot to bump its serial; the box's gate is still satisfied and it kept the old copy.
• re-fetches every single run (stamp rewritten each cycle) — a shared tool whose serial was bumped in only one of its two call sites; fix by matching the serial in both.

For a tool that carries a run-once function gate (_bTs, the 5103vNN marker that counts up within a release), a behaviour change to the function — not just the fetched file — must bump _bTs in the same commit, or the function short-circuits on its existing marker. That marker and the DOWN-vs-UP serial distinction are on the serial pipeline leaf.

Related

• Verifying the drush extension filter — the other leaf in this topic: a worked Gate-3/4 verification of a security-sensitive backend change (the *.drush.inc deny-filter), run as each identity on a real box, and the _DRUSH_VERSION tarball-pin bump that lands it.
• Code style & conventions — the full Gate-1 lint detail: the bash and PHP lint gates, the 5.6 forbidden-syntax list, and the runtime-divergence traps.
• The serial & fetch pipeline — _fetch_versioned, the DOWN-counting fNN serials, _bTs, and how a tagged release propagates: the mechanism Gate 4 smoke-tests.
• Variables and Commands — the consolidated _VAR and CLI tables, including the tool serials, _tRee/_xSrl, and the upgrade verbs a test box drives. </content> </invoke>