Edge policy — AI bot policy, real client IP, secret-path deny

BOA applies a set of edge policies at the Nginx layer before a request reaches PHP-FPM: per-class AI bot handling, real-client-IP recovery behind Cloudflare, a universal secret/config-path deny, and per-vendor rate limits. These are rendered into the master http{} config (the map/geo/limit_req_zone definitions) by server.tpl.php, and enforced per request by the guard chain in Inc/vhost_include.tpl.php.

This page is the policy layer — who is classified, allowed, rate-limited or dropped, and how the realip plumbing makes those decisions key on the true visitor. The abuse/IDS layer (the $is_banned geo, the asset/content-chain flood maps, the print/no-referer gate, the .php catch-all, the bot and search-amplification blocks, the $is_cms_probe foreign-CMS admin-probe map (server.tpl.php:458-463), the boa_i18n_* anonymous-localized concurrency maps, and the scan_nginx log scorer) is a separate subsystem documented in Abuse guard.

AI bot policy — per-class maps

BOA classifies AI crawlers, indexers and assistant fetchers into mutually-exclusive classes by user-agent, and applies a default action per class. Classification is always by a distinctive UA token, never a bare vendor name — so GPTBot (training) and ChatGPT-User (assistant fetch) land in different classes, and a bare ChatGPT matches nothing.

The stance: block the worst offenders unconditionally, sort every real AI agent into a class, and make each class flippable per site. Training and the evasive user-fetch class are opt-in (off by default); search / user / utility are opt-out (on by default). Source: server.tpl.php:323 ($is_ai_training), :333 ($is_ai_search), :346 ($is_ai_user), :359 ($is_ai_evasive), :368 ($is_ai_utility), :378 ($is_ai_forged).

Default inversion — training is BLOCK by default

This is a behaviour change from the earlier policy. The old single broad $is_ai_crawler map allowed AI by default; the per-class policy blocks training by default and only allows search, user and utility traffic (each rate-limited). Operators who want AI training crawlers on a given site must opt in explicitly (see per-site control).

The $is_ai_crawler backward-compat shim

The old broad $is_ai_crawler map is kept defined as a do-not-remove backward-compatibility shim (server.tpl.php:303-318, which carries an explicit DO NOT REMOVE banner). The reason is a render-order trap: a barracuda upgrade re-renders this master http{} config before the per-Octopus instance vhosts. An instance vhost rendered by the previous Provision still references $is_ai_crawler, so an http config that no longer defined it would make nginx -t fail with unknown "is_ai_crawler" variable — taking every instance on the host down until each is individually re-rendered. Shared http-block map variables must stay a superset across versions: never remove or rename one a deployed vhost may use.

Evasive class — best-effort only

$is_ai_evasive exists for agents that are nominally user-triggered but ignore robots.txt and, when blocked, drop their declared UA and rotate IPs/ASNs to slip past a UA rule. Perplexity-User is the current member — Cloudflare de-listed it as a verified bot for exactly this. It is blocked by default, cleared per site by the evasive-allow flag. Because the block keys on the UA, it is best-effort only: once the agent abandons its UA it looks like an ordinary browser and the (fail-open) maps let it through. The real backstop for the rotating/undeclared traffic is the IDS/CSF layer (see Abuse guard), not this policy. Source: server.tpl.php:352-363.

Note the honest counterpart Google-Agent is classed as an honest user-fetcher in $is_ai_user (matched as Google-?Agent, server.tpl.php:348), not as evasive.

Forged opt-out tokens → universal 444

Google-Extended and Applebot-Extended are robots.txt-only opt-out directives — a real crawler never sends them as a live user-agent. Their presence in a UA is therefore proof of forgery and is hard-blocked (444) globally, regardless of UA class, with zero false positives. Source: server.tpl.php:378-382; the guard fires at vhost_include.tpl.php:218 (after the secret-path deny and the $is_cms_probe guard).

The $is_crawler / AI boundary

The scrapers / bad-bots hard-block map $is_crawler (444 at vhost_include.tpl.php:257) is separate from the AI maps. The former AI substrings (Amazon, bytedance, CCBot, externalagent, openai, perplexity) were deliberately removed from $is_crawler because bare openai / perplexity matched the openai.com / perplexity.ai vendor URLs embedded inside OAI-SearchBot / ChatGPT-User / PerplexityBot / Perplexity-User user-agents, and so wrongly blocked allowed AI traffic.

AI is now governed solely by the per-class $is_ai_* maps. Do not re-add AI tokens to $is_crawler — per the never-pollute shared-map rule. Source: the removal-rationale comment and cleaned roster at server.tpl.php:478-492. ($is_crawler itself — the generic scraper block — is part of the abuse-guard layer; see Abuse guard.)

Per-vendor rate limits

The three allowed AI classes are throttled by limit_req zones, but keyed per vendor rather than per client IP. The keys come from map definitions that emit a constant string per UA token (one constant per vendor) and an empty key for everything else — $ai_search_limit_key, $ai_user_limit_key, $ai_utility_limit_key (server.tpl.php:395-421). An empty key is not counted by its zone, so no non-AI traffic is ever throttled by these zones.

Source: zones at server.tpl.php:109-111. Keying per vendor (not per IP) is the correct primitive for an assistant whose single prompt fans out across many source IPs: a per-IP cap never bites the aggregate (each IP stays under the limit), whereas one shared key per vendor caps that vendor's total rate across every IP and vhost, and gives each vendor its own bucket so no one vendor can starve another's allowance.

Each $ai_*_limit_key roster *must track its matching `$isaiclass map** — a token in one but not the other is silently un-rate-limited (or, for the class map, un-blockable). The two are kept adjacent inserver.tpl.php` for that reason.

Per-site control — policy.txt

The global defaults are flipped per site via a per-Octopus control file:

/data/disk/<oct>/static/control/ai/policy.txt

read by aegir/tools/system/ai_policy.sh. Each record is one line:

<site>  [train-allow] [evasive-allow] [search-block] [user-block] [utility-block]

ai_policy.sh generates a per-site fragment into the instance's own config/includes/ai_policy/<site>.conf — exactly the path the per-satellite vhost pulls via include $server->include_path/ai_policy/<uri>.conf* (the .conf* suffix anchors the glob to this site's own fragment; a bare <uri>* would also match a longer site whose name extends it, e.g. example.com vs example.com.au). The flag → directive mapping (ai_policy.sh:87-91):

A site with no record keeps the global defaults. ai_policy.sh is a single global script that loops over every Octopus instance with an activated policy.txt. The control file is also catalogued in Control files & INI as a per-site control file.

The guard chain (per request)

The relevant subset of vhost_include.tpl.php runs top-to-bottom early in request processing. Training and evasive are blocked by a composite guard (set the block to the class, clear it if the per-site allow flag is set, drop if still set); search / user / utility have no global guard because they are allowed by default, so a per-site block is carried directly by the policy.txt fragment.

if ($is_secret_path)  { return 444; }   # secret/config-path probes     (vhost_include.tpl.php:198)
if ($is_cms_probe)    { return 444; }   # foreign-CMS admin-path probes (vhost_include.tpl.php:210) — abuse-guard layer
if ($is_ai_forged)    { return 444; }   # forged opt-out tokens         (:218)

set $ai_train_block $is_ai_training;     # training composite           (:230)
if ($ai_train_allow)  { set $ai_train_block ''; }   #  per-site opt-in clears it
if ($ai_train_block)  { return 444; }    #  otherwise training blocked   (:234)

set $ai_evasive_block $is_ai_evasive;    # evasive composite            (:246)
if ($ai_evasive_allow){ set $ai_evasive_block ''; } #  per-site opt-in clears it
if ($ai_evasive_block){ return 444; }    #  otherwise evasive blocked    (:250)

$ai_train_allow / $ai_evasive_allow are defaulted to 0 before the ai_policy include in the vhost template, so without a fragment both classes stay blocked. The rate limit for the three allowed classes is applied separately in location / via limit_req zone=ai_search|ai_user|ai_utility. (The $is_banned, $is_crawler and $is_cms_probe guards that bracket/interleave this excerpt belong to the abuse-guard layer — see Abuse guard.)

Universal secret / config-path deny

map $uri $is_secret_path (server.tpl.php:429-437) hard-blocks (444) probes for credential and config paths that never exist on a hosted Drupal/Aegir site, regardless of UA. The guard fires at vhost_include.tpl.php:198, before the AI guards.

It matches on the decoded, normalised $uri (not the raw request line) and is anchored to path segments, so a literal .env cannot match an arbitrary substring, and $uri normalisation defends against percent-encoded evasion (e.g. /%2eenv → /.env). The roster:

config.json and key.json are listed because Drupal/Aegir never serve them at web root, so a request for one is always a probe. This deny spans the edge-policy and abuse-guard subsystems; the abuse framing is in Abuse guard.

Real client IP (Cloudflare realip)

Every guard above keys on $remote_addr. Behind Cloudflare the raw TCP peer is a CF edge, not the visitor — so the master http{} config recovers the real client (server.tpl.php:131-133):

real_ip_header    CF-Connecting-IP;
real_ip_recursive on;
include /data/conf/nginx_cloudflare_real_ip.c*;   # set_real_ip_from <CF ranges>

The include is a wildcard glob (.c*) so nginx -t passes before the ranges file exists. With no trusted ranges declared, the CF-Connecting-IP header is ignored and $remote_addr is left unchanged — there is no spoofing risk, and direct (non-CF) traffic never triggers realip.

The trusted ranges are maintained by aegir/tools/system/cloudflare_realip.sh, which fetches Cloudflare's published IPv4 + IPv6 ranges into /data/conf/nginx_cloudflare_real_ip.conf. It is deployed to /var/xdrago/ and runs daily via crontab (04:45) plus once at install directly from BOA.sh.txt — so a fresh or updated box does not wait for the daily cron. It is idempotent: it only rewrites the include and reloads Nginx when the fetched ranges actually change, and validates every CIDR (octet range, prefix length) before writing so a malformed upstream response can never break the host's configtest.

Effect: with realip active, $remote_addr becomes the real visitor and $realip_remote_addr the CF edge — so rate-limit keys, bans and access logs all bite the real client even for CF-proxied sites. Realip-window caveat: until cloudflare_realip.sh populates the ranges file, $remote_addr is unchanged (the CF edge), so enforcement keys on the edge during that window.

PHP REMOTE_ADDR pin

fastcgi_param REMOTE_ADDR is set to $realip_remote_addr — the original TCP peer (the CF edge) — not $remote_addr (server.tpl.php:156). This is deliberate: it keeps Drupal's global.inc own reverse-proxy real-client resolution correct (PHP still sees the edge as the proxy and resolves the client itself), while Nginx-level enforcement keys on the realip-rewritten $remote_addr. This is a behaviour change from the pre-realip era, where REMOTE_ADDR was the raw peer.

A related tool, migration_proxy_realip.sh, extends the same wildcard include with a sibling .cmig member to trust an xmass/xoct migration proxy (the old host forwarding CF-Connecting-IP) during a site migration; see Migration & cloning.

Shared Nginx reload lock

Five BOA generators write into the host Nginx config and run configtest + reload independently:

To stop their configtest+reload cycles colliding on the same host Nginx, all five take a shared advisory lock /run/boa_nginx_config.lock — flock -w 30 on fd 9; if the lock is not acquired within 30s the run is skipped and retried on the next tick (ai_policy.sh:26,33-37). Each generator follows the same discipline: change-gate → atomic write → configtest → reload, with rollback to the last-good config on failure.

Source: all five scripts under aegir/tools/system/; fetch + at-install realip run at BOA.sh.txt:1489-1496; cron cadence in aegir/tools/system/cron/crontabs/root:8-12.

Related

• Abuse guard — the IDS/abuse layer: $is_banned geo, flood chains, scan_nginx scorer, the CSF ban pipeline. This page is the edge policy; that section is the abuse detection/enforcement.
• Config templates — the master Nginx config these maps render into, and the never-remove-map-vars rule.
• Custom rewrites & location blocks — operator includes that layer over (and can short-circuit) the guard chain.
• Control files & INI — policy.txt and the other per-site control files.
• Migration & cloning — the migration-proxy realip extension.