scan_nginx scoring engine

scan_nginx.sh is the detection half of the Abuse Guard — a post-hoc log analyser. On each tick it reads a window of recent access.log lines, scores every real client IP across five independent ban-issuing detectors (plus two alert-only Tier-B paths), and writes the IPs that cross a threshold to /var/xdrago/monitor/log/web.log, the entry point of the ban pipeline. It never sits in the request path and cannot block a request in flight: its job is to decide who is banned; enforcement is real-time in the request guards once the ban reaches the $is_banned geo.

Run model

nginx_guard.sh branches on the load-pause pidfiles. On the normal path — neither /run/max_load.pid nor /run/critical_load.pid exists — it launches the scorer 10× per minute (5 s apart), each launch a single scan_nginx pass, reloading Nginx first if access.log is missing or empty. When a load-pause is active the IDS is not skipped: the guard runs one bounded scan_nginx pass (a single launch — no 10× fan-out, no sleep loop) so the worst aggressors are CSF-banned while Nginx is stopped and the box does not re-saturate on resume. Nothing in that pass touches Nginx: _block_ip appends offenders to web.log/archive (plus the instant csf -td 900 on 80/443 when /etc/boa/.instant.csf.block.cnf exists), and guest-fire converts web.log into CSF bans on its own schedule, independently of Nginx — so the paused-path pass can neither lift the pause nor add the normal path's web-tier load. minute.sh re-invokes the guard each minute, giving one more ban pass per paused minute; scan_nginx itself has no load-pid self-gate, so the pass actually runs. Pause semantics are on the load control page.

nginx_guard.sh tick (normal: 10 passes/min, 5 s apart · load-pause: 1 bounded pass)
   ↓
re-entrancy guard (shared lock.inc; legacy pgrep fallback exits if >2 instances)
   ↓
load config: built-in defaults, then source /root/.barracuda.cnf (replaces any value)
   ↓
read access.log from the last byte offset (incremental)
   ↓
for each line:  resolve real IP → exemption gate → 5 detectors + Tier-B tallies
   ↓
_handle_blocking / _handle_ddos_blocking / _handle_ua_burst_blocking
_handle_path_flood_blocking / _handle_http10_auth_flood
_handle_i18n_flood / _check_fpm_saturation   (Tier B — alert only)
   ↓
write offenders to web.log + scan_nginx.archive.log

• Window. Each run reads new bytes since the last position recorded in /var/log/scan_nginx_lastpos. On a first run, or if the log was rotated or truncated (current size < saved offset), the offset resets and the last _NGINX_DOS_LINES lines (default 1999) are taken as a baseline.
• Post-hoc. Because the window is whatever has accumulated since the last tick, a fast burst (dozens of requests in seconds) is scored after it completes. The fast-ban weights below shorten the score needed, not the cron latency — see Operations + tuning for the latency model.
• Config precedence. Built-in defaults are assigned first, then /root/.barracuda.cnf is sourced and replaces any value it sets — a plain assignment, not a merge. Full list in the Configuration reference.

Resolving the real client IP

The log records the IP chain as "$proxy_add_x_forwarded_for" (the X-Forwarded-For values with $remote_addr appended last). scan_nginx scores only the last token of the chain — the value Nginx's realip module rewrote $remote_addr to (the CF-Connecting-IP on Cloudflare vhosts, or the direct peer otherwise). The earlier X-Forwarded-For entries are client-supplied and spoofable, so they are never scored or banned.

• The real client may be a valid, public IPv4 or IPv6 address. An IPv4 last token bans via CSF as before; an IPv6 last token — which can only be a realip-recovered client from a trusted proxy (BOA disables IPv6 server-side and pins realip to the Cloudflare ranges) — is scored the same way and banned via the nginx-native IPv6 path (nginx_deny6.sh → the $is_banned geo), since CSF is IPv4-only. Opt out per box with _NGINX_V6_BAN_DETECT=NO.
• Private ranges are skipped for both families (10./127./169.254./192.168./ 172.16–31. and ::1/::/fe80::/10/fc00::/7).
• The forwarded-for proxy array is deliberately left empty — upstream proxies are never ban candidates.

Realip dependency — a bad trust chain scores the wrong IP. If the realip trust chain is wrong (e.g. a customer's own reverse proxy is not in set_real_ip_from), every request through it is attributed to the proxy's address. The Abuse Guard then scores that one IP for all the traffic behind it and can ban the whole site. Getting realip right (via cloudflare_realip.sh / migration_proxy_realip.sh and the rendered set_real_ip_from ranges) is a prerequisite, not an optional extra. Symptom: a single upstream/CDN IP racking up a huge score for traffic that is obviously many distinct clients. See Operations.

The exemption gate

Immediately after the IP is resolved, and before any detector runs, each line is tested against _NGINX_DOS_IGNORE_PATHS, a space-separated list of URI prefixes that must never be scored:

_is_ignored_request "${_line}" && continue

Because the test runs at loop scope (a continue), an exempt line is skipped by every detector at once — the continue fires before any of the six per-line trackers run, so the line feeds none of them. This exists because machine/API endpoints authenticate per request at the application layer (HMAC, OAuth tokens), not by IP — so IP-counting them is always wrong. A webhook provider (Shopify, QuickBooks, Stripe…) delivers in bursts from a rotating pool; an API client is simply high-volume. Either pattern otherwise reads as a per-IP or shared-UA flood and self-bans a legitimate caller. Default list:

/shopify/webhook /quickbooks/webhook /stripe/webhook
/paypal/webhook /github/webhook /gitlab/webhook
/graphql /public-api /oauth2

The match is deliberately strict and laundering-proof:

• It parses the real $request URI out of the line — not a substring search over the whole line — so a token smuggled into a User-Agent, Referer, or query string cannot forge an exemption.
• It requires a full METHOD /path HTTP/x shape, strips the query string, and refuses any URI containing .. or a % escape (an encoded traversal like /shopify/webhook/%2e%2e/wp-login.php is scored, not exempted).
• Each entry matches as an exact path or a sub-path prefix (/graphql exempts /graphql/api/endpoint).
• It is status-agnostic — a 444 on an exempt path is also skipped, which is what stops a banned webhook IP's retries from re-feeding its own score.

The defaults ship in the script (not only the per-box override) so the exemption survives a /root/.barracuda.cnf regeneration. An override replaces the whole list — include everything the box needs; an empty value disables the feature.

The function-IFS gotcha. The script runs under a global IFS=$'\n\t' (no space), so any function that word-splits a space-separated value must set a function-local whitespace IFS first or the split collapses to one token and silently matches nothing. _is_ignored_request sets local … IFS=$' \t\n'; the two aggregate handlers use the _SAVE_IFS="${IFS}"; IFS=' '; …; IFS="${_SAVE_IFS}" save/restore idiom. A space-split helper tested interactively passes but fails in-script — when one "matches nothing", suspect the inherited IFS before re-auditing the matching logic. See Operations.

Detector 1 — per-IP scoring

_process_ip accumulates a weighted counter per real IP. The weights derive from _NGINX_DOS_LIMIT (default 399):

The first row is the single heaviest weight: a line matching the _NGINX_DOS_STOP regex adds the entire _NGINX_DOS_LIMIT at once, saturating the score. The default is a SQLi blind-timing fingerprint set — WAITFOR.DELAY|DECLARE.*@x|/\*\*/|%27.*%29.*%3B|0x[0-9a-f]{6} — matched unanchored against the whole log line and evaluated in both DoS modes.

An IP is written to web.log when its score crosses the limit and it made at least _NGINX_MIN_BLOCK_REQS (default 3) raw requests in the window. The raw-request floor stops a single heavily-weighted hit from banning a one-request IP.

Under the hood. The threshold compared in _handle_blocking is _MININUMBER = (_NGINX_DOS_LIMIT + 1) / 2 for the first gate, and the IP is only banned when its score also exceeds _CRITNUMBER (normally _NGINX_DOS_LIMIT). Two session shields raise _CRITNUMBER to a sentinel so the IP is effectively never banned: an IP with an active SSH session (_is_logged_in) → 9999, and the box's own _MYIP → 9998.

A second tier of +5 increments fires only when _NGINX_DOS_MODE=1 (the default is mode 2, where they are inactive):

The optional [xx/] prefix matches a two-letter language segment (e.g. /de/search). On a default (mode-2) box these add nothing.

Worked examples, with defaults:

• Webshell scanner. A .php probe that 404s scores ~133 per hit; three hits → 399 ≥ limit → blocked in ~3 requests instead of ~140. A single stale .php link from a real visitor scores 133 once — below the limit and below the 3-request floor — so it never bans. Heavy weight that does not saturate is false-positive-safe.
• Ordinary 404 noise. A normal missing page scores ~10; it takes ~40 such hits to reach the limit, so incidental 404s don't ban.

The .php-probe weight is path-anchored (the .php must be in the request path, not the query string, Referer, or UA) and excludes legitimate entry points (index, update, install, cron, xmlrpc, authorize, restore, rebuild, boost_stats, rtoc, js). The catch-all that makes a non-entry .php return 404 on a BOA docroot is the Nginx rule documented in request guards.

Detector 2 — shared-UA aggregate (DDoS)

Distributed floods often share one machine User-Agent across many rotating IPs. _track_ua_ip builds, per UA, the set of distinct real IPs and the total request count. After the loop, _handle_ddos_blocking declares a UA an attack fingerprint when either threshold is crossed:

When a UA is flagged, every contributing IP that made at least _NGINX_DDOS_IP_MIN_REQS (default 20) requests with it is blocked. The per-IP minimum keeps incidental single hits from a shared egress (one Cloudflare PoP) out of the block list. UAs of 10 characters or fewer are ignored to avoid matching empty or trivial agents. This is the detector that, before the path-exemption existed, banned an entire webhook provider's rotating pool on its shared Shopify-Captain-Hook / GuzzleHttp UA — which is why the exemption gate runs at loop scope and covers this detector too.

Why the defaults are this high. These values were retuned upward from 20 IPs / 200 reqs / 3-req block after the low originals banned real visitors. Each scan scores only ~5 s of appended lines, and within that window the single most common mobile-browser UA on a high-traffic site is shared by well over 20 distinct IPs (and 200 requests) — while a genuine distributed botnet randomises its UA per IP, so one UA shared by many IPs is the signature of a popular browser, not a bot. The old block floor of 3 was exactly a human search session (results page + per-keystroke autocomplete + AJAX views + result clicks); 20 sits clear of any single session while still catching one IP hammering a shared UA. Genuinely abusive single IPs remain covered by the per-IP scorer and the path-flood detector; the redirect-heavy distributed fleets this loosening would miss are covered by Detector 5.

Detector 3 — path-flood aggregate

The hardest pattern is a botnet sending one request per IP: no single address ever trips a per-IP limit. _track_path_flood counts both 200 and 444 traffic to each watched expensive-path prefix across all IPs, and _handle_path_flood_blocking blocks every qualifying participant once a prefix is declared under flood:

The flood declaration counts both 200 and 444 (so distributed bots the real-time guards already 444 are still caught and reported in aggregate). The per-IP listing / csf -td gate counts only 200 responses — backend-reaching requests that actually cost a Solr / PHP-FPM cycle. An IP whose hits to the watched prefix were all 444 is already free-blocked by Nginx at zero backend cost, so it is not individually banned: a csf -td on it would be redundant, would bloat web.log during distributed 444 floods, and risks false-positiving CGNAT / shared-egress clients. A 200 slower than _NGINX_PATH_FLOOD_SLOW_SECS earns an extra per-IP increment. Watched prefixes come from _NGINX_PATH_FLOOD_WATCH (Solr / Search-API / facet endpoints by default); this detector was built for search-amplification attacks that bypass simple 444 rules by adding a Referer.

Why the defaults are this high. Retuned upward from 5 IPs / 15 reqs / 10-hit gate. The prior declaration thresholds sat below real peaks: 5 distinct IPs declared a flood on any busy public search page within the ~5 s window, and 15 responses is ~3 req/s of search traffic — exceeded by any popular search page at peak, where 100 (~20 req/s site-wide) sits above legitimate interactive use and well below a real Solr / Search-API amplification flood. Declaration alone never bans — the per-IP gate decides who is blocked — but a low value wastes work and widens the blast radius on a legitimate peak. The per-IP gate itself rose from 10 to 20 because a shared CGNAT / Apple-Private-Relay egress aggregating many real users can exceed 10 backend 200s in a window; it is kept deliberately modest because under the default _NGINX_DOS_MODE=2 a backend 200 scores only +1 in the per-IP scorer, so a moderately heavy search scraper is caught primarily here — raising it much further would let it through.

Detector 4 — HTTP/1.0 auth-path spam (cross-run window)

A credential/registration-spam botnet POSTs to Drupal auth paths (/user/register, /user/password) over HTTP/1.0 while forging a modern-browser User-Agent. HTTP/1.0 is the clean transport-layer tell — no browser built in ~15 years speaks HTTP/1.0 to a public HTTPS host — and the botnet this detector was built for paces a slow trickle across a small CIDR block (the source describes ~1 slow request per IP every few minutes, a /29 observed), so neither the per-IP scorer's raw-request floor (with its per-run counter reset) nor the search-oriented path-flood watch list ever catches it.

Tally. A line is counted only when its $request field parses positionally as METHOD /path HTTP/1.0 — the case pattern [A-Z]*" /"*" HTTP/1.0" both validates the request-line shape and selects HTTP/1.0 in one step (HTTP/1.1 and HTTP/2.0 never match) — and the query-stripped URI matches _NGINX_HTTP10_AUTH_PATHS (default ^/([a-z]{2}/)?user/(register|password)(/|$), the optional two-letter prefix mirroring i18n paths). URIs containing .. are refused. This is the same positional, traversal-rejecting $request parse as the exemption gate, so a token smuggled into a User-Agent, Referer, or query string cannot fake either signal. A malformed _NGINX_HTTP10_AUTH_PATHS override is caught at startup (the ERE is test-evaluated) and reverted to the default with a warning — a typo cannot silently disable the detector; malformed numeric overrides revert the same way.

Window and ban. Per-IP tallies persist across runs in /var/xdrago/monitor/log/http10_auth.window; buckets older than _NGINX_HTTP10_AUTH_WINDOW (default 600 s) are pruned. The cross-run window is the point — the bot is too slow for any single ~5 s scan window. An observed IP is banned when its own windowed count reaches _NGINX_HTTP10_AUTH_IP_THRESHOLD (default 3) or its /24 aggregate reaches _NGINX_HTTP10_AUTH_CIDR_THRESHOLD (default 6). The /24 path escalates a slow distributed block cleanly, but only IPs actually seen sending the signal are ever banned — never an unseen address in the /24.

Guards. Private, already-banned, locally-allowed and CSF-whitelisted IPs are skipped at tally time, so they never accumulate window state; active-SSH (_is_logged_in) sessions and the box's own _MYIP are skipped at ban time. Bans go through _block_ip, which re-checks the csf.allow whitelist and feeds the normal web.log → guest-fire → guest-water pipeline (plus the instant csf -td 900 on 80/443 when /etc/boa/.instant.csf.block.cnf exists). The handler runs after the per-IP, DDoS-UA, UA-burst and path-flood passes so _BANNED_IPS is fully populated. On by default (_NGINX_HTTP10_AUTH_DETECT=YES).

Safety precondition — who speaks HTTP/1.0 to this Nginx? $server_protocol is the protocol on the connection to this Nginx, not the realip-recovered client's. All five BOA proxy templates set proxy_http_version 1.1 (proxy.conf, ssl_proxy.conf, pln_proxy.conf, https_proxy_le.conf, and the legacy nginx_wild_ssl.conf), so visitors arriving through BOA's own proxy / PX0 tier are designed to log HTTP/1.1 or HTTP/2 at origin. A box behind a non-BOA front proxy or CDN that downgrades to HTTP/1.0 at origin — or one whose deployed proxy vhosts have not yet been regenerated from the current templates — must opt out with _NGINX_HTTP10_AUTH_DETECT=NO; first confirm in the access log that real clients show HTTP/1.1/2 and only the bot shows HTTP/1.0. See Operations.

Detector 5 — distributed UA-burst scanner fleet

Detector 2 and the per-IP scorer both exclude 301 redirects and both need high per-IP or per-UA volume — exactly the gap a redirect-heavy distributed auth-probe fleet slips through (the shape this was built for: dozens of cloud IPs sharing one forged UA, a few requests each, most of them 3xx/4xx to non-existent paths). _track_ua_burst is therefore fed every line including 301s, in its own arrays so it never perturbs Detector 2; like Detector 2 it only considers UAs longer than 10 characters.

A UA is declared a fleet only when all three gates pass:

"Bad" is a deliberate status shortlist: 301/302/307/308/400/403/404/410 only. 444 is excluded (already blocked — counting it would double-count), 401 excluded (auth challenge), 2xx/304 excluded (success/cache), 5xx excluded (server fault, not the client's doing). The bad-status ratio is the false-positive keystone: a popular browser UA shared by many legitimate IPs is ~all 200/304 and never trips — which is why IP_MIN can sit far below _NGINX_DDOS_UA_IP_THRESHOLD (12 vs 100) without re-opening the shared-UA over-ban class Detector 2's retune closed.

When a fleet trips, only IPs that themselves sent at least _NGINX_UA_BURST_IP_MIN_BAD (default 3) bad probes under that UA are blocked — a legitimate visitor sharing the UA sent 200s (zero bad) and is never caught. The whitelist (_is_whitelisted_ip / _ALLOWED_IPS), logged-in and self-IP (_MYIP) guards mirror the DDoS pass, and blocking goes through _block_ip. The pass runs between the DDoS and path-flood passes. On by default; _NGINX_UA_BURST_DETECT=NO disables it (the per-line tracker is gated on a single flag, so the hot loop pays nothing when off), and malformed numeric overrides revert to the defaults. Source guidance: tune tighter, never looser, on real reports.

This detector complements the $is_cms_probe edge map on the edge policy page: the edge map sheds foreign-CMS path probes per request at zero backend cost, but deliberately does not match generic auth words (login, signin, admin, user, account) because they collide with real customer URL namespaces and Drupal's own paths — that distributed generic-auth-word tail is banned here, in aggregate.

Tier B — detect, alert, snapshot (no per-IP bans)

Two further paths detect and record but never ban an IP — banning is futile against a source spread across thousands of addresses at one or two requests each. The Tier-A guardrail (the boa_i18n_anon per-vhost concurrency cap on the request guards page) does the real-time capping; floodreport on the Operations page reads everything these paths write. Both are on by default and both tallies are gated on a single flag each, so the hot loop pays nothing when they are off; the exemption gate and the monitoring skips run first at loop scope, so webhook paths, Site24x7 and files.* traffic are never counted.

Distributed i18n-flood detector (_NGINX_I18N_FLOOD_DETECT=YES). Tallies localized requests per vhost — a non-IP key, because a distributed source never trips per-IP limits. A request is "localized" when its path starts with a two-letter language prefix, optionally with a script/region suffix (/pt-br/, /zh-hans/), or carries the D7 ?q=<lang>/ form — the same class the Tier-A guardrail caps. Each run's tallies merge into a cross-run sliding window of _NGINX_I18N_FLOOD_WINDOW (default 120 s) persisted in window.state under /var/xdrago/monitor/log/i18n_flood/. A vhost trips on either:

On a trip it appends to /var/xdrago/monitor/log/i18n_flood.log and writes a forensic snapshot (top vhosts, client IPs, User-Agents, status mix, language prefixes, mean/max request_time) under i18n_flood/, with a per-vhost cooldown of _NGINX_I18N_FLOOD_COOLDOWN (default 300 s) via marker-file mtime, so a multi-minute burst yields a handful of records, not one per scan. The stress gate is what separates a flood from a popularity spike: the attack shape this was built for drives each uncached localized page through an expensive synchronous backend, while a cache-warm recon burst (high volume, ~0 % stress) stays below the gate and never trips. The tally key is the vhost; the client field parsed for snapshots is the realip'd $remote_addr, so snapshots rank real clients, never the CF/PX0 edge.

FPM-saturation trigger (_NGINX_FPM_SAT_DETECT=YES). _check_fpm_saturation byte-offset-tails every file matching _NGINX_FPM_ERR_GLOB (default /var/log/php/php*-fpm-error.log) for new lines matching _NGINX_FPM_SAT_PATTERN (reached max_children setting) — the per-pool ceiling event PHP-FPM logs the instant a pool cannot spawn a worker. The minute-cadence php.sh monitor (auto-healing) tails the same logs for the same string with the same offset technique, but emits only a plain raise-pm.max_children capacity NOTE; this trigger pairs the alert with the attack forensics. Offsets persist in i18n_flood/fpm_maxchildren.pos and re-baseline to 0 when a file shrank. On a hit it appends an FPM-SATURATION line to i18n_flood.log and writes a box-wide (*) snapshot in the same format as above.

Whitelisting

Three layers protect known-good addresses from every detector:

• CSF allow list. _is_whitelisted_ip refuses to block any IP the firewall already allows (exact host or CIDR). It is a keystone guard covering every call path into _block_ip — including the bulk DDoS and path-flood passes — so a bulk ban can't drop a CDN PoP or a search-engine crawler. The allow is honoured on every port regardless of the port scope of the csf.allow entry (an s= record means "trusted source").
• /root/.local.IP.list. Loaded into _ALLOWED_IPS; these IPs are never scored.
• In-run cache. _BANNED_IPS (seeded from the current web.log) prevents an IP being written twice in one pass.

To exempt a monitoring service fleet-wide, add its published IPs to the CSF allow list rather than exempting a path — see Operations.

There is no successful-login skip in scan_nginx. The "don't ban an IP that just logged in" logic lives in the SSH/login monitor hackcheck.sh, which builds an _accepted set from grep -F 'Accepted ' auth.log and never bans those IPs. scan_nginx's web-side login handling is the inverse — it adds weight on /user/login floods. Do not expect a recent web login to protect an IP here. The one session shield scan_nginx has (_is_logged_in, threshold raised to 9999) keys off active SSH sessions (netstat ESTABLISHED on port 22), not any web or auth.log login. For a durable, protocol-independent exemption use csf.allow or /root/.local.IP.list.

Output

When a detector decides to block, _block_ip:

1. appends IP # [xSCORE] TIMESTAMP to /var/xdrago/monitor/log/web.log and to /var/xdrago/monitor/log/scan_nginx.archive.log (skipping IPs already in the in-run cache);
2. if /etc/boa/.instant.csf.block.cnf exists, also issues csf -td 900 on ports 80 and 443 immediately, shaving one hop off the pipeline.

web.log feeds the temporary-ban applier; scan_nginx.archive.log feeds the persistent escalator. Continue to the ban pipeline.

Related

• The ban pipeline — the three scripts that turn web.log and scan_nginx.archive.log into a CSF ban and the $is_banned geo.
• Request guards — the real-time map/geo wall that enforces the bans this scorer produces.
• Configuration reference — every _NGINX_* default the scorer reads.
• Operations + tuning — reading state, the latency model, the realip dependency and the function-IFS gotcha.