Abuse Guard code internals

This is the maintainer view of scan_nginx.sh — the ~1975-line post-hoc log analyser that is the detection half of the Abuse Guard. It is for people extending it: adding a detector, retuning a default, or tracing why a helper that passed offline banned nothing in production. It documents the mechanisms, not the operator knobs; the effective thresholds and the reader-facing tuning model live on the Operating side, and every _NGINX_* default is in the variables reference.

Everything here is verified against scan_nginx.sh and its launcher nginx_guard.sh at HEAD, plus the paired Provision request-guard templates.

The shape of a detector

scan_nginx is a single-pass batch job with a fixed skeleton, and every detector — existing or new — plugs into the same six stages:

  1. Defaults assigned as plain script-level _NGINX_* variables (scan_nginx.sh:64-375).
  2. Config override: /root/.barracuda.cnf is sourced once (:381-386), replacing any value it sets — a plain assignment, never a merge.
  3. Validation: each numeric/regex override is range-checked and reverted to the built-in default on a bad value (:397-457), so userland config can never break arithmetic or silently disable a detector.
  4. Per-line tally: the main loop reads the log incrementally and calls the detector's _track_* helper on each surviving line, accumulating into declare -A associative arrays (:471-531, loop :1757-1931).
  5. Post-loop handoff: after the loop closes, a _handle_* / _check_* function evaluates the accumulated arrays and calls _block_ip for offenders (:1945-1972).
  6. Output: _block_ip writes the IP to web.log (:663-694); the external ban pipeline turns that into a CSF ban on its own schedule.

The detectors are deliberately independent — separate arrays, separate handlers, separate config blocks — so one can be added, retuned or disabled without touching another. Five detectors issue bans (per-IP weighted scorer, shared-UA DDoS, UA-burst fleet, path-flood, HTTP/1.0 auth-spam) and two Tier-B paths detect/alert/snapshot without banning (distributed-i18n-flood, FPM-saturation).

Incremental byte-offset log reading

The window each pass scores is whatever appended since the last pass, tracked by byte offset in /var/log/scan_nginx_lastpos (:1733-1755):

_OFFSET_FILE="/var/log/scan_nginx_lastpos"
_last_offset=0
[[ -f "${_OFFSET_FILE}" ]] && _last_offset=$(< "${_OFFSET_FILE}")
_current_size=$(stat -c %s "${_log_file}")
# rotate/truncate guard: file shrank below saved offset -> start over
(( _current_size < _last_offset )) && _last_offset=0

if (( _last_offset == 0 )); then
  exec 3< <(tail -n "${_NGINX_DOS_LINES}" "${_log_file}")     # baseline
else
  exec 3< <(tail -c +$(( _last_offset + 1 )) "${_log_file}")  # new bytes only
fi
  • Baseline vs delta. On the first run, or after a rotate/truncate reset, the last _NGINX_DOS_LINES (default 1999) lines are taken as a one-shot baseline. Every subsequent run reads only the new bytes since the saved offset — tail -c +N, not tail -n.
  • Rotate/truncate safety. _current_size < _last_offset means the file was rotated or truncated; the offset resets to 0 and the next read re-baselines. Without this guard a rotated log would seek past EOF and score nothing.
  • Offset write. After the loop closes the descriptor (exec 3<&-), the current end-of-file size is written back as the next run's start (:1936-1939). The write is unconditional on the file existing, so a pass that read nothing still advances the offset.
  • Why this shapes tuning. nginx_guard.sh launches the scorer ~10× per minute, so a normal delta is only ~5 s of appended lines. Aggregate thresholds (shared-UA, path-flood) are sized against that window, not a per-minute or per-hour rate — a value that looks high per-hour is often one popular browser's 5 s of traffic. This is the single most common source of over-ban regressions; see Detector defaults are window-relative.

The same byte-offset-tail idiom is reused for the FPM error logs (_check_fpm_saturation, :1599-1634) with its own per-file position map — see Byte-offset tailing beyond access.log.

Cross-run sliding-window state files

A per-run window is blind to a slow attacker — one request per IP every few minutes never accumulates inside a single 5 s delta. Two ban detectors and one Tier-B path therefore persist state across runs in plain pipe-delimited files, all under /var/xdrago/monitor/log/:

State file Owner Window var (default) Bucket record
http10_auth.window _handle_http10_auth_flood _NGINX_HTTP10_AUTH_WINDOW (600 s) IP\|epoch\|count
i18n_flood/window.state _handle_i18n_flood _NGINX_I18N_FLOOD_WINDOW (120 s) vhost\|epoch\|req\|slow\|err\|c444
i18n_flood/fpm_maxchildren.pos _check_fpm_saturation (byte offsets, no time window) file\|size

All three share one prune-merge-write shape (_handle_http10_auth_flood, :1681-1727; _handle_i18n_flood, :1542-1594):

_now="$(date +%s)"; _cut=$(( _now - _WINDOW ))
_tmp="${_state}.$$"; : > "${_tmp}"                  # per-PID temp, race-safe
# 1. carry forward unexpired buckets
while IFS='|' read -r _key _e _c; do
  [[ -n "${_key}" && "${_e}" =~ ^[0-9]+$ ]] || continue
  (( _e > _cut )) || continue                        # prune expired
  printf '%s|%s|%s\n' "${_key}" "${_e}" "${_c}" >> "${_tmp}"
  _W["${_key}"]=$(( ${_W["${_key}"]:-0} + _c ))      # rebuild windowed total
done < "${_state}"
# 2. append this run's tallies with the current epoch
for _key in "${!_RUN[@]}"; do
  printf '%s|%s|%s\n' "${_key}" "${_now}" "${_RUN[$_key]}" >> "${_tmp}"
  _W["${_key}"]=$(( ${_W["${_key}"]:-0} + _RUN[$_key] ))
done
mv -f "${_tmp}" "${_state}"                          # atomic replace
# 3. evaluate _W against the threshold, ban / alert

Design invariants to preserve when you extend this:

  • Prune on read. Expiry is enforced every pass by dropping buckets older than _cut; there is no separate reaper. A run that reads the window is what ages it out.
  • Write to a PID-suffixed temp, then mv -f. Ten overlapping passes per minute mean two scorers can touch the file at once; the .$$ temp plus atomic rename is the only thing keeping the window from interleaving. A new windowed detector must copy this, not append in place.
  • Every field is re-validated on read (_e =~ ^[0-9]+$, :-0 defaults). The file is userland-adjacent (an operator can cat/edit it); a corrupt row is skipped, never fatal.
  • The window is a ban-decision input, not a ban ledger. web.log / scan_nginx.archive.log are the ban record; the window only decides whether the threshold is crossed this pass.

For the HTTP/1.0 detector the windowed per-IP totals are additionally summed per /24 (_net="${_ip%.*}", :1707-1711) so a slow trickle spread across a CIDR block trips _NGINX_HTTP10_AUTH_CIDR_THRESHOLD even when no single IP reaches _NGINX_HTTP10_AUTH_IP_THRESHOLD — but only IPs actually seen in the window are ever banned, never an unseen address in the /24.

Byte-offset tailing beyond access.log

_check_fpm_saturation (:1599-1634) reuses the offset technique against a glob of files rather than one log. It keeps a file|size position map (fpm_maxchildren.pos), and for each file matching _NGINX_FPM_ERR_GLOB (default /var/log/php/php*-fpm-error.log) it counts only the _NGINX_FPM_SAT_PATTERN (reached max_children setting) lines appended since last run:

for _f in ${_NGINX_FPM_ERR_GLOB}; do          # unquoted: glob expands
  _sz="$(stat -c %s "${_f}")"
  _off="${_POS["${_f}"]:-0}"
  (( _sz < _off )) && _off=0                    # per-file rotate/truncate guard
  (( _sz > _off )) && _new="$(tail -c +$(( _off + 1 )) "${_f}" \
                              | grep -c -- "${_NGINX_FPM_SAT_PATTERN}")"
  printf '%s|%s\n' "${_f}" "${_sz}" >> "${_tmp}"
done
mv -f "${_tmp}" "${_posfile}"

BOA's minutely php.sh monitor tails the same logs for the same string with the same offset technique, but emits only a plain raise-pm.max_children capacity NOTE; this trigger pairs the alert with attack forensics (a box-wide * snapshot). If you add a third reader of these logs, give it its own position file — sharing one offset map between two readers double-consumes lines.

Resolving the real client IP — never the spoofable XFF

Score the real client, never a forwarded-for token. The log records the IP chain as $proxy_add_x_forwarded_for (the X-Forwarded-For values with $remote_addr appended last). scan_nginx acts on only the last token — the value Nginx's realip module rewrote $remote_addr to (:1790-1809):

_last_raw="${_ip_array[$(( ${#_ip_array[@]} - 1 ))]:-}"
if _validate_ip "${_last_raw}" \
  && [[ ! "${_last_raw}" =~ ^(10\.|127\.|169\.254\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\.) ]]; then
  _REAL_IP="${_last_raw}"
else
  _REAL_IP=""
fi
_PROXIES_ARRAY=()   # earlier chain entries are NEVER ban candidates
  • The earlier X-Forwarded-For entries are client-supplied and spoofable (a client can prepend a forged public IP), so the proxy array is left deliberately empty — upstream tokens are never scored.
  • The real client must be a valid, public IPv4 or IPv6 address; private ranges are skipped for both families. An IPv4 last token is scored and banned via CSF as before. An IPv6 last token — which can only be a realip-recovered client from a trusted proxy (BOA disables IPv6 server-side and pins realip to the Cloudflare ranges, so no direct v6 peer or spoofable chain entry can appear here) — rides the same family-agnostic detectors and is banned via the nginx-native IPv6 path (_block_ip → web6.tempban → nginx_deny6.sh → the $is_banned geo), because CSF is IPv4-only. Gated by _NGINX_V6_BAN_DETECT (default YES; set NO to restore the earlier IPv4-only behaviour).
  • The realip trust chain is a hard prerequisite. If a customer's own reverse proxy is not in set_real_ip_from, every request through it logs the proxy's address as the last token, and the scorer bans that one IP for all the traffic behind it — the whole site. A new detector that keys on IP must consume _REAL_IP from this resolution, never re-parse the chain itself. Tier-B snapshots parse the realip'd client field for the same reason: they rank real clients, never the CF/PX0 edge.

Exemption gate — ordering before detectors

The exemption gate runs at loop scope, before any detector's tally, so a continue skips every detector at once (:1836):

_is_ignored_request "${_line}" && continue     # exempts all detectors

Ordering is load-bearing. The webhook/API over-ban class (banning a provider's rotating pool on its shared UA) is a DDoS-UA failure, not a per-IP one — a per-IP-only skip would miss it. Because the gate precedes _process_ip, _track_ua_ip, _track_ua_burst, _track_path_flood, _track_i18n_flood and _track_http10_auth, one gate covers them all. The files.* host skips (:1821-1828) sit at the same scope for the same reason.

_is_ignored_request (:760-790) is laundering-proof by construction, and its parse is the template every request-line matcher in the script copies:

_after="${_line#*\"*\"}"      # drop leading "IP-chain" quoted field
_req="${_after#*\"}"; _req="${_req%%\"*}"        # _req = METHOD URI PROTO
case "${_req}" in [A-Z]*" /"*" HTTP/"[0-9]*) : ;; *) return 1 ;; esac
_uri="${_req#* }"; _uri="${_uri%% *}"; _uri="${_uri%%\?*}"   # bare path
[[ "${_uri}" == /* && "${_uri}" != *".."* && "${_uri}" != *%* ]] || return 1
for _p in ${_NGINX_DOS_IGNORE_PATHS}; do
  [[ "${_uri}" == "${_p}" || "${_uri}" == "${_p}"/* ]] && return 0
done
  • It parses the real $request URI positionally — not a substring search over the whole line — so a token smuggled into a User-Agent, Referer or query string cannot forge an exemption.
  • It requires a full METHOD /path HTTP/x shape, strips the query string, and refuses any URI containing .. or a % escape (an encoded traversal like /shopify/webhook/%2e%2e/wp-login.php is scored, not exempted).
  • It is status-agnostic — a 444 on an exempt path is skipped too, which is what stops a banned webhook IP's retries from re-feeding its own score.

_track_http10_auth (:1648-1671) reuses the identical positional, traversal-rejecting parse to select HTTP/1.0 and match the auth-path ERE — so neither signal can be faked from an earlier field. Copy this parse for any new request-line matcher; do not fall back to a whole-line regex.

The function-local IFS gotcha

The script sets a global IFS=$'\n\t' at the top (:48) — no space. Any helper that word-splits a space-separated value therefore splits nothing under the inherited IFS: the value collapses to one token and the loop matches nothing, silently. The symptom is a detector that passes every interactive test yet exempts/bans nothing in the running script.

Two idioms fix it, both present in the source; use one in every space-splitting helper you add:

  • Function-local declaration — set IFS in the local line so it is scoped to the function and restored on return. _is_ignored_request (:766) and _track_http10_auth (:1649):

    local _line="$1" _after _req _uri _p IFS=$' \t\n'

  • Save/restore around the split — for the aggregate handlers that split mid-function (_handle_ddos_blocking :1050-1054, _track_ua_burst :1162-1165, _handle_path_flood_blocking :1300-1303; and the watch-list parse :1443-1445):

    local _SAVE_IFS="${IFS}"
IFS=' '
# ... split _NGINX_*_LIST ...
IFS="${_SAVE_IFS}"

When a space-split helper "matches nothing", suspect the inherited IFS before re-auditing the matching logic — the classic works-in-test, fails-live signature. The main-loop IFS=',' and IFS='|' reads are already function/subshell-scoped via read, so they don't leak; the trap is specifically a bare for x in $list over a space-separated variable.

Post-hoc latency — the scorer never blocks in-request

scan_nginx is a post-hoc analyser: it runs on a cadence, off the request path, and cannot block a request in flight. Its job is to decide who is banned; enforcement is real-time, downstream, once the ban reaches the $is_banned geo the request guards read.

The consequences shape every design choice here:

  • A burst is scored after it completes — the fast-ban weights (a _NGINX_DOS_STOP match adds the whole _NGINX_DOS_LIMIT at once) shorten the score needed to cross the threshold, not the ~5 s cron latency to the next pass.
  • Truly distributed attacks that arrive faster than the pass cadence are the reason the aggregate and cross-run detectors exist — they catch in aggregate what no single fast pass can.
  • The one exception to "never touches the firewall in-line" is the optional /etc/boa/.instant.csf.block.cnf marker, which makes _block_ip also issue an immediate csf -td 900 on 80/443 (:689-692), shaving one hop off the pipeline — but that still fires from the post-hoc pass, not the request.
  • A new detector must therefore stay O(lines) and cheap: it runs inside a loop that fires ten times a minute. Per-line trackers are gated on a single _*_ON flag (_I18N_ON, _H10_ON, _UAB_ON, :1456-1472) precisely so the hot loop pays nothing when the detector is off. Wire your tracker behind such a flag.

The load-pause pass — the IDS no longer goes dark

nginx_guard.sh (nginx_guard.sh:81-108) branches on the load-pause pidfiles. On the normal path — neither /run/max_load.pid nor /run/critical_load.pid exists — it launches the scorer 10× per minute (5 s apart), reloading Nginx first if access.log is missing or empty:

if [ ! -e "/run/max_load.pid" ] && [ ! -e "/run/critical_load.pid" ]; then
  [ -s /var/log/nginx/access.log ] || service nginx reload
  for _iteration in {1..10}; do
    nohup ${_monPath}/scan_nginx.sh > /dev/null 2>&1 &
    sleep 5
  done
else
  # load-pause: ONE bounded pass, no 10x fan-out, no sleep loop
  nohup ${_monPath}/scan_nginx.sh > /dev/null 2>&1 &
fi

The else branch is the point: previously the entire launch was gated behind "no load-pause", so the IDS stopped scanning exactly when a flood was strong enough to trip the watchdog's Nginx stop — offenders stayed unbanned and re-saturated the box on resume. Now a single bounded pass runs while paused. It can neither lift the pause nor add the normal path's web-tier load, because _block_ip bans via CSF and appends to web.log without reloading Nginx, and scan_nginx itself has no load-pid self-gate, so the pass actually runs. minute.sh re-invokes the guard each paused minute, giving one more ban pass per minute.

How a new detector plugs in

Concretely, to add a banning detector, mirror the HTTP/1.0 or UA-burst detector:

  1. Config defaults — add _NGINX_<NAME>_* script-level variables in the defaults block (:64-375). If the detector is on by default use a _NGINX_<NAME>_DETECT="YES" string flag with a documented per-box opt-out.
  2. cnf override + validation — the single source "${_CONFIG_FILE}" already loads any override. Add fail-closed validation (:436-457): numeric knobs =~ ^[1-9][0-9]*$ || _default; a regex knob test-evaluated with [[ "probe" =~ ${_RE} ]] 2>/dev/null; (( $? > 1 )) && revert. A malformed override must revert to the default with a warning, never silently disable the detector.
  3. Arrays + gate flag — declare -A your tally arrays alongside the others (:471-531); set _<NAME>_ON=1 from the DETECT flag (:1456-1472) so the per-line tracker is one branch when off.
  4. Per-line tracker — call _track_<name> inside the main loop, after the exemption gate and realip resolution, gated on _<NAME>_ON. Consume _REAL_IP; parse the request line with the positional exemption-gate parse, never a whole-line regex.
  5. Threshold vars + handoff — add a _handle_<name> after the loop (:1945-1972). It runs after the per-IP, DDoS, UA-burst and path-flood passes so _BANNED_IPS is fully populated and you avoid double-banning. Ban through _block_ip — it is the keystone guard: it re-checks _is_whitelisted_ip (the csf.allow map) on every call and seeds the in-run _BANNED_IPS cache so an IP is never written twice.
  6. web.log ban handoff — _block_ip appends IP # [xSCORE] TIMESTAMP to web.log and scan_nginx.archive.log (:679-680); the external web.log → guest-fire → guest-water pipeline converts that into the CSF ban and the $is_banned geo on its own schedule. Your detector's job ends at _block_ip.

Also honour the three whitelist layers on every ban path: _is_whitelisted_ip (CSF allow, in _block_ip), _ALLOWED_IPS (/root/.local.IP.list, skipped at scoring), and the session shields (_is_logged_in active-SSH, and the box's own _MYIP). A Tier-B detect-only path (no per-IP ban) skips step 5's _block_ip call and instead writes a log line plus a _i18n_snapshot forensic record.

Detector defaults are window-relative

Aggregate thresholds are sized against the ~5 s delta window, not a human's sense of "a lot of requests". The retune that closed the am095/kwestiasmaku over-ban class raised the shared-UA and path-flood defaults sharply (:84-99, :154-194) because the most common mobile-browser UA on a busy site is shared by far more than the old 20-IP floor within a single window — a popular browser, not a botnet (a real distributed botnet randomises its UA per IP). The load-bearing rule for anyone touching these numbers, straight from the source comments: tune tighter, never looser, and only on a real report. The UA-burst detector's bad-status ratio (_NGINX_UA_BURST_BAD_PCT) is the false-positive keystone that lets its IP floor sit far below the DDoS-UA one without re-opening the over-ban class — a legitimate shared browser UA is ~all 200/304 and never trips.

Do not quote specific default values in code comments or living docs — they move on retune. State the rule (window-relative, tighter-only) and let the variables reference carry the current numbers.

Paired-template carry-verbatim caveats (request-guard maps)

The scorer bans; the request guards enforce — and those live in the Provision Nginx templates, not in bash. Several detectors here are two halves of one mechanism split across scan_nginx.sh and the templates, and the template side follows a strict declare-then-enforce discipline you must not break when you touch it.

  • Two-stage map idiom. A guard variable is declared in the http block by server.tpl.php (server.tpl.php:458-462 for $is_cms_probe; :924-952 for the $boa_i18n_guard / $boa_i18n_path / $boa_is_anon → $boa_i18n_anon_key chain) and enforced per request in vhost_include.tpl.php. server.tpl.php re-renders the http block before the vhosts, so every deployed vhost can reference the map. Never remove or rename a deployed map variable — dropping one turns the next nginx -t into a fleet-wide reload failure. Add, never replace; the idiom exists precisely so nginx -t never sees an undefined variable.
  • $is_cms_probe (server.tpl.php:458-462) matches WordPress/Joomla/phpMyAdmin admin paths as whole path segments only (/wp-content-strategy and /site-administrator deliberately do not match), and vhost_include.tpl.php returns 444 when it is set — before Drupal bootstraps. That 444 is then scored by scan_nginx's per-IP 444 counter, so the edge guard both sheds load and makes the offender visible to the firewall layer. adminer is excluded (BOA ships Adminer); generic auth words (login, signin, admin, user, account) are excluded because they collide with real customer namespaces — that distributed generic-auth-word tail is banned in aggregate by the UA-burst detector instead.
  • The Tier-A i18n cap (limit_conn_zone $boa_i18n_anon_key, server.tpl.php:120; enforced at location = /index.php) is the real-time companion to the Tier-B i18n-flood detector: the 444s the cap sheds are the detector's earliest trip signal (_NGINX_I18N_FLOOD_C444_THRESHOLD). The cap keys on a constant per vhost, not per IP, so it never appears in csf -t / csf -g — it caps blast radius rather than chasing rotating IPs.
  • Carry any shared change verbatim into the self-contained templates. The subdir vhost template does not include the shared common file — it inlines its own body — so a guard-chain fix in vhost_include.tpl.php does not reach it. Mirror the change there too, verbatim, or subdir vhosts silently diverge. Likewise the SSL vhost path reuses the same include, so it inherits the guard chain automatically — but the subdir copy is the one that bites.
  • BOA sed-patches the shipped templates on disk during the Barracuda/Octopus upgrade chain. An edit to a .tpl.php in the tree only lands if it survives (or is applied by) those patches — a template change the seds would undo is silently reverted on the next upgrade. When a map or guard you added does not appear on a box, check the patch set, then confirm the vhost was actually re-rendered by a Verify.

Related