Edge map/geo mechanics & the never-remove rule

Every request BOA serves is filtered by a wall of Nginx map/geo variables declared once in the box's http { } block and read per request by the shared guard chain. This page is the maintainer's contract for that wall: how the maps render into the master nginx.conf, why the declare-here / enforce-there split exists, why a deployed map variable can never be removed or renamed, how the per-class policy maps compose, and where the rendered files land so you can verify a change on a box.

The vhost generator covers the render machinery — which Provision_Config class writes which template at which scope. This page is the map/geo catalogue and the invariants a change to server.tpl.php must keep.

Two files, two scopes

The wall is split across exactly two server-scope templates, both rendered once per box (not per site):

vhost_include.tpl.php is included by every per-site server { } block (vhost.tpl.php renders include nginx_vhost_common.conf). So one file declares the variables at http scope, a second reads them at server/location scope, and every deployed vhost pulls in the second by reference. A map variable is therefore a shared ABI between the box-wide http config and every vhost on the box — that is the whole reason the never-remove rule exists.

The declare-then-enforce idiom

An Nginx map/geo is a variable definition; the if ($var) that acts on it lives elsewhere. BOA leans on this deliberately: a variable referenced by an if is only legal if the http block defined it. Two consequences a maintainer must internalise:

1. A variable must be defined before it is used, in a config Nginx loads as a whole. nginx -t (and every reload/restart) parses the entire tree; an if ($is_banned) in a vhost with no map $remote_addr $is_banned anywhere in http is a hard unknown "is_banned" variable parse error that fails the whole config — not just that vhost.

2. The definition can be data-driven without changing the ABI. The two population maps use an include so the variable is always defined even when its data is empty:

   geo $remote_addr $is_banned {
    default 0;
    include /data/conf/nginx_banned_ips.c*;   # server.tpl.php:473-476
   }
   map $host $boa_i18n_guard {
    default 1;
    include /data/conf/boa_i18n_guard.map*;   # server.tpl.php:924-927
   }

   The BOA monitor/firewall layer writes nginx_banned_ips.conf; an operator opt-out writes boa_i18n_guard.map. The glob (*/.c*) means an absent file is simply an empty map — $is_banned stays defined and defaults to 0, $boa_i18n_guard stays defined and defaults to 1. Never collapse this into a bare if that reads a file directly; the indirection is what keeps the variable defined across the render window described next.

Most maps are the simpler static form: a single-signal map $key $flag in server.tpl.php read by one if in vhost_include.tpl.php, no operator knob and no data file. The foreign-CMS probe guard is the canonical shape — a map $uri $is_cms_probe (server.tpl.php:458-462) matching wp-*, administrator, and phpmyadmin as whole path segments ((?:^|/)…(?:[./?]|$), so /wp-content-strategy and /site-administrator do not match), enforced by a single if ($is_cms_probe) { return 444; } (vhost_include.tpl.php:210) before the try_files → @drupal → /index.php fall-through pays a full Drupal bootstrap on a path that cannot exist on a Hostmaster docroot. adminer is deliberately excluded (BOA ships Adminer) and generic auth words (login, admin, user) are not matched — they collide with real URL namespaces. It has no opt-out, but it is still a deployed map variable: the never-remove rule below applies to it exactly as to $is_banned.

Why a deployed map variable is never removed or renamed

This is the single most dangerous edit in server.tpl.php. BOA runs one host Nginx. A barracuda upgrade re-renders the master http config before it re-renders the per-Octopus-instance vhosts. Verify the ordering in the code:

• boa-private/lib/functions/master.sh.inc:449 verifies @server_master (or @server_localhost) first — this re-renders the server-scope files, i.e. the new nginx.conf with the new maps.
• Only after that does it run hosting-dispatch + hosting-tasks (master.sh.inc:453-467), which is what re-verifies each Octopus instance and re-renders its per-site vhosts.

So there is a window where the new http block is already live while the instance vhosts are still the previous render. If the new http block dropped (or renamed) a map that an old vhost still references:

• The next nginx -t fails unknown "<var>" variable.
• nginx -s reload refuses the new config and keeps the old workers — but a full nginx restart (which the upgrade chain does) has no old workers to fall back on, so Nginx does not come up.
• Every instance on the box is down until each vhost is individually re-rendered to stop referencing the missing variable.

This is not hypothetical. It happened when the single broad $is_ai_crawler map was replaced by per-class maps: old vhosts still did if ($is_ai_crawler) { return 444; }, and the fix was to restore $is_ai_crawler verbatim as a do-not-remove shim. That shim is still in the tree with the reason inline:

### Backward-compatibility shim — DO NOT REMOVE.  $is_ai_crawler was the old
### single broad AI map.  It is kept DEFINED (original broad match) because a
### barracuda upgrade re-renders this master http config BEFORE the Octopus
### instance vhosts ...
map $http_user_agent $is_ai_crawler {   # server.tpl.php:304-318 (map decl :314)
  default  '';
  ~*Ai2Bot|Amazon|Anthropic|...  is_ai_crawler;
}

The rule, operationally

The http block must always be a superset of what any deployed vhost references. When changing maps in server.tpl.php:

1. *ADD new `$is_` maps.** Never repurpose an existing name.
2. If a new map supersedes an old one, keep the old variable DEFINED with its original definition (so not-yet-re-rendered vhosts keep their prior behaviour) and a do not remove comment. Enforcement can move; the definition stays.
3. Before deleting any map variable, grep every vhost template — current AND the shape a previously-shipped Provision would have rendered — and confirm zero references. Retire a definition only once nothing on any box can reference it.
4. The same caution applies to anything consumed across the staggered render: limit_*_zone names, fastcgi_params, include targets. A vhost that limit_conn boa_i18n_anon requires the limit_conn_zone … zone=boa_i18n_anon to exist in http.

This is why the enforcement side never invents a variable inline. Adding a new guard is a two-file edit: declare the map in server.tpl.php, then read it in vhost_include.tpl.php. Do them in the same change so nginx -t never sees an undefined variable on any render.

Carry-verbatim between the paired templates

server.tpl.php and vhost_include.tpl.php are a matched pair, and so are the SSL and subdir variants that reuse the same include. Several invariants only hold if both halves agree — treat these as carry-verbatim:

• *Every `if ($is_)invhost_include.tpl.phpneeds itsmapin server.tpl.php`.** The reverse is allowed (a declared-but-unused map is inert), but an enforced-but-undeclared variable is a fleet-down parse error.
• The AI rate-limit key rosters must track their class maps. Each $ai_*_limit_key map (server.tpl.php:395-420) must list the same UA tokens as the matching $is_ai_* class map. A token in the class map but not the key map is un-rate-limited; a token in the key map but not the class map is silently un-blockable. The comment at server.tpl.php:391-393 states this explicitly.
• subdir.tpl.php does not include the common file — it inlines its own full body. A guard added to vhost_include.tpl.php does not reach subdir vhosts; mirror any shared-guard change into subdir.tpl.php by hand (see the vhost generator subdir note). This is the same duplicate-and-mirror discipline as the paired master/satellite BOA scripts: intentional duplication, kept in sync manually, never DRY-ed.
• Comments carry the why. The DO NOT REMOVE banner on $is_ai_crawler, the "roster must track the class map" note, the "keyed per vendor not per IP" explanation on the AI limit keys — these are load-bearing. Preserve them verbatim when editing nearby; they are the only record of a production incident on the line you are about to change.

How the per-class policy maps compose

The AI policy is not one map — it is a small pipeline of composable maps, each keyed on a single request attribute, combined by a final map keyed on the concatenation of the intermediates. Two patterns recur.

Mutually-exclusive class maps + per-vendor key maps

The AI stance sorts every AI agent into one class by a distinctive UA token (never a bare vendor name — GPTBot is training, ChatGPT-User is a user fetch, bare ChatGPT matches nothing):

The three allowed classes are not blocked by a global if; a per-site BOA ai_policy fragment carries if ($is_ai_*) { return 444; } directly when an operator opts a site out. The two opt-in blocks (training, evasive) hinge on an allow flag ($ai_train_allow / $ai_evasive_allow) defaulted to 0 before the ai_policy include and set to 1 only from inside the fragment, so an absent fragment leaves the block on (vhost_include.tpl.php:230-236, :246-252).

Rate limiting composes through a second map per allowed class — the per-vendor key maps $ai_search_limit_key / $ai_user_limit_key / $ai_utility_limit_key (server.tpl.php:395-420). An empty key is not counted by its zone, so each limit_req_zone (server.tpl.php:109-111) counts only its own class and leaves all other traffic untouched. The key is a constant per vendor (not per client IP) because a single assistant prompt fans out across many source IPs — a per-IP cap never bites the aggregate.

Multi-signal composition (the search-flood and i18n gates)

The DDoS gates combine several weak signals into one strong verdict by keying a final map on the concatenated outputs of intermediate maps. The boa_i18n_anon concurrency cap is the clearest example (server.tpl.php:924-955):

map $host $boa_i18n_guard { default 1; include /data/conf/boa_i18n_guard.map*; }
map $request_uri $boa_i18n_path { default 0; ~*^/[a-z][a-z](-[a-z]+)?/ 1; ... }
map $cache_uid $boa_is_anon { default 0; "" 1; }
map "$boa_i18n_guard$boa_i18n_path$boa_is_anon" $boa_i18n_anon_key {
  default ""; "111" $host;
}

Three predicates — vhost opted in, request is under a two-letter language prefix, request is anonymous — concatenate to a 3-char string; the final map emits $host only for "111" and "" otherwise. The empty key is invisible to limit_conn_zone $boa_i18n_anon_key zone=boa_i18n_anon (server.tpl.php:120), so only the targeted class is counted. Enforcement is one line at the FPM chokepoint: limit_conn boa_i18n_anon 24 with limit_conn_status 444 (vhost_include.tpl.php:1744-1765), tunable via the nginx_i18n_anon_conn Drush option.

The search-flood gates use the same shape: $has_fulltext_search + $has_no_referrer → $block_search_no_referrer (server.tpl.php:504-524), and the three-signal $has_fulltext_search$has_root_only_referer$has_any_facet → $block_search_root_referer (:614). Because $cache_uid (the session-cookie map) feeds $boa_is_anon, authenticated editors are never caught by these gates — that predicate is reused, not recomputed.

The compose rule for a new gate: add the weak-signal maps at http scope, add the combining map keyed on their concatenation, then a single if ($block_*) in vhost_include.tpl.php. Keep every intermediate defined (the never-remove rule applies to them too — a deployed vhost that reads $block_search_no_referrer fails if a later render drops it).

Where the rendered files live — for verification

A template edit changes nothing until a Verify re-renders the config classes. To confirm your change landed, read the rendered files on the box, not the tree:

The master http config is written by Provision_Config_Nginx_Server (template server.tpl.php) to <config_path>/nginx.conf — the application_name is nginx, and for @server_master config_path is /var/aegir/config/server_master (http/Provision/Config/Http/Server.php:21-29, http/Provision/Service/http/public.php:52-57). The include dirs pre.d/, platform.d/, vhost.d/, post.d/ all derive from the same <config_path>/nginx app dir (public.php:54-58).

Verification checklist for a map change:

1. grep the map name in the rendered nginx.conf on a master and an Octopus instance — both must show the new definition after Verify.
2. grep the enforcement in the rendered nginx_vhost_common.conf — the if/limit_conn must be present and reference only defined variables.
3. Run nginx -t on the box. A green test means the http block is a superset of every vhost's references.
4. Confirm the BOA sed patch layer did not undo the edit — the templates on a box live under /var/aegir/.drush/sys/provision/http/Provision/Config/Nginx/ and are patched in place by boa-private/lib/functions/nginx.sh.inc during the upgrade chain. A template change that a sed there rewrites is silently reverted on the next upgrade; see the vhost generator patch-layer section.

Related

• Vhost generator (Provision templates) — the Provision_Config render cycle, the template set, the on-disk targets, and the nginx.sh.inc sed patch layer that transforms these templates on a box.
• settings.php wiring — the Drupal side of the same generator; the vhost injects DB creds these settings read back.
• Solr installer internals — the Barracuda-side search backends the same vhosts talk to.
• Web & search stack internals — topic chapter — the four-leaf render picture this page zooms into.
• Aegir backend APIs — the provision_nginx_server_config hook, the extension-safe way to inject http/server config without editing a template.
• Variables reference and Commands reference — the consolidated _VAR and CLI tables.
• Discontinued features — retired map variables and superseded edge mechanics.