Operations + tuning

This page is the operator/maintainer side of the Abuse Guard: how to read its live state, how to purge or cycle a ban, how to whitelist an IP / a path / a monitoring service, how to turn on debug output, and the hard caveats you need to understand before you trust — or extend — the system.

For the detector logic see scan_nginx scoring engine; for the data flow into CSF and the geo file see the ban pipeline; for the variable defaults see the Configuration reference.

Reading live state

Everything the Abuse Guard does is reflected in a handful of files. None of them is a database — they are plain text and safe to cat/grep.

Practical reads:

# Who is currently temp-banned, and on which ports
csf -t

# Is a specific IP banned, and why (temp + permanent + allow)?
csf -g 203.0.113.7

# What did the scorer flag most recently?
tail -n 50 /var/xdrago/monitor/log/web.log

# Which IPs are repeat offenders (escalation candidates)?
cut -d'#' -f1 /var/xdrago/monitor/log/scan_nginx.archive.log \
  | sort | uniq -c | sort -rn | head

# How many entries are in the live geo right now?
grep -c . /data/conf/nginx_banned_ips.conf

The geo file is a derived mirror: nginx_deny.sh rebuilds it on every pass from the current csf.tempban (WEB ports) plus the csf.deny lines tagged Brute force Web Server. It only rewrites and reloads Nginx when the set actually changed, and on a failed reload it reverts to its last-good backup (/data/conf/.nginx_banned_ips.last_good.conf). Editing the geo file by hand is pointless — the next pass overwrites it. Drive bans through CSF, not the geo.

floodreport — summarise i18n-flood activity

For the Tier-B files above you rarely need to grep by hand: floodreport (aegir/tools/bin/floodreport, deployed into PATH) is the on-demand reporter in the fpmreport/loadreport family — read-only, POSIX awk + sort only, writes nothing and changes nothing. It reads the detector's own output (i18n_flood.log plus the per-trip snapshots) and reports per-vhost event counts (guardrail-shedding / volume+stress / FPM-saturation), a timeline of notable events with sheds summarised, a merged attack profile from the snapshots (top User-Agents, language prefixes, client networks, status mix, peak request_time), and a per-vhost verdict — cap holding, FPM saturation, or mass-503 (the BOA load-protection path).

floodreport               # this node, all retained i18n-flood data
floodreport --days 7      # window: last 7 days (relative to the newest event)
floodreport --hours 6     # window: last 6 hours
floodreport --data DIR    # analyse an off-node copy of the logs (like fpmreport)
floodreport --json        # one machine-readable finding per line

If a box has _NGINX_I18N_FLOOD_DETECT=NO (see the Configuration reference) the detector produces no data and floodreport reports nothing to analyse.

Purging or cycling a ban

Because the geo is a mirror of CSF and is fully regenerated each pass, you never edit the geo to clear a ban — you change CSF and let it propagate. Bans also self-expire: a temp ban is csf -td … 900 (15 minutes) and disappears on its own; the geo follows on the next nginx_deny.sh pass.

# Remove a single temporary ban now (does NOT touch csf.deny)
csf -tr 203.0.113.7

# Remove a permanent deny (water's "Brute force Web Server" escalation)
csf -dr 203.0.113.7

# Flush ALL temporary bans (last resort; the scorer will re-ban real abuse)
csf -tf

After any of these the geo still shows the old entry until the next scan tick regenerates it. To force the geo back in sync immediately:

bash /var/xdrago/nginx_deny.sh

Truncating archive.log by hand only resets repeat-offender history — it does not lift any existing ban (those live in CSF and the geo). For the clean-slate case, hand-ordering those steps is superseded by clearwebbans below.

clearwebbans — one-shot recovery from web-IDS false positives

clearwebbans (aegir/tools/bin/clearwebbans, deployed into PATH) does the whole WEB-scoped clean slate in one idempotent run, in this order:

1. removes permanent csf.deny Brute force Web Server escalations (csf -dr per IP);
2. removes temporary bans on ports 80/443 (csf -tr per IP, taken from the csf.tempban rows whose port field is 80/443 — per the script's own note, on a heavily-polluted box this loop takes a while);
3. wipes web.log and both archive generations — scan_nginx.archive.log (guest-water's escalation source) and the rotated scan_nginx.archive.x3.log — so guest-fire cannot re-apply the bans and guest-water cannot re-escalate from history;
4. resets /var/log/scan_nginx_lastpos to the current end of access.log, so the next scan sees only new traffic and never re-flags the lines that produced the false positives;
5. re-asserts the synproxy rules if active, then regenerates the nginx geo-ban set via /var/xdrago/nginx_deny.sh so $is_banned clears at once (nginx_deny.sh self-handles its lock, configtest and reload).

SSH and FTP bans are deliberately untouched. --dry-run (-n) reports the counts plus up to ten sample IPs per set and changes nothing — it also skips the lock, so reporting never blocks a running cleanup; real runs are flock-guarded at /run/clearwebbans.lock (-h/--help prints usage). IP tokens from every source are strictly validated as IPv4 hosts before they reach csf.

When to use it: after loosening the IDS limits, or any false-positive event — including Detector 4 / UA-burst false positives — to release everyone the web detectors banned. Best run once the updated scan_nginx limits are already deployed, so the released IPs are not immediately re-banned under the old thresholds. Keep the manual csf -tr/csf -dr commands above for single-IP cases; clearwebbans is the mass-release valve.

Whitelisting

There are three independent ways to make the Abuse Guard never act on an address or a request, depending on what you are protecting.

Whitelist an IP — use the CSF allow list

The keystone guard is _is_whitelisted_ip: scan_nginx parses /etc/csf/csf.allow once at startup into an exact-host map plus a CIDR index, and every call path into a block (_block_ip, the shared-UA handler, the path-flood handler) checks it first. An allowed IP is never scored or banned, and the allow is honoured on every port regardless of the port scope of its csf.allow entry — an s= record means "trusted source" and gates the monitor's block decision on 80/443 too.

# Permanently trust an IP (or CIDR) fleet-wide
csf -a 198.51.100.0/24 "office network"

This is the right tool for a monitoring service: add its published IP ranges to csf.allow rather than exempting a path. A path exemption opens that URI for everyone; a CSF allow exempts only the named source.

A second, scorer-local layer is /root/.local.IP.list — IPs listed there are loaded into _ALLOWED_IPS and skipped before scoring. It is a convenience for local/infrastructure addresses; for anything that must also be trusted by the firewall stages, use csf.allow.

Note. The successful-login skip is not in scan_nginx. The "don't ban an IP that just logged in" logic lives in the SSH/login monitor hackcheck.sh, which builds an _accepted set from grep -F 'Accepted ' auth.log and never bans those IPs. scan_nginx's web-side login handling is the inverse: it adds weight on /user/login POST/GET floods. Do not expect a recent web login to protect an IP from the Abuse Guard. scan_nginx does carry one session-based shield of its own — _is_logged_in raises an IP's block threshold to a sentinel 9999 (the box's own _MYIP to 9998) so it is effectively never banned — but it keys off active SSH sessions (netstat ESTABLISHED on port 22), not any web or auth.log login. For a durable, protocol-independent exemption use a csf.allow entry or /root/.local.IP.list.

Whitelist a path — _NGINX_DOS_IGNORE_PATHS

When the thing you must protect is an endpoint that authenticates per request (a webhook receiver, an API root), exempt the path rather than chasing the caller's rotating IPs. Add it to _NGINX_DOS_IGNORE_PATHS in /root/.barracuda.cnf:

# Override REPLACES the default list — include the shipped defaults you still need
_NGINX_DOS_IGNORE_PATHS="/shopify/webhook /quickbooks/webhook /stripe/webhook \
/paypal/webhook /github/webhook /gitlab/webhook /graphql /public-api /oauth2 \
/my/custom/webhook"

The match is strict (real $request URI only, query string stripped, ../% -escape refused) and exempts the line from all three detectors at once. The full semantics are on scan_nginx scoring engine. Mind the IFS gotcha below if you script around this value.

When to opt out of the HTTP/1.0 auth-spam ban

One exemption is a per-box detector opt-out rather than a per-address whitelist. Detector 4 — the HTTP/1.0 auth-spam ban (see scan_nginx scoring engine) — is on by default and keys on $server_protocol: the protocol of the connection to this Nginx, not the realip-recovered client's. BOA's own proxy layer is safe — all five proxy templates (proxy.conf, ssl_proxy.conf, pln_proxy.conf, https_proxy_le.conf, nginx_wild_ssl.conf) set proxy_http_version 1.1, so an updated BOA front proxy / PX0 tier no longer downgrades to HTTP/1.0 at origin and real proxied visitors log HTTP/1.1 / HTTP/2. The residual false-positive sources are a non-BOA front proxy or CDN that talks HTTP/1.0 to origin, or a box not yet updated to the HTTP/1.1 proxy confs — opt out there, with _NGINX_HTTP10_AUTH_DETECT=NO in /root/.barracuda.cnf.

Before trusting the default behind any proxy, confirm the transport tell is clean (the confirmation recipe from the source comment):

# Real clients must show HTTP/1.1 or HTTP/2.0 on the auth paths;
# only the bot traffic should be on HTTP/1.0
grep 'user/register' /var/log/nginx/access.log

If the detector has already false-positived, clearwebbans (above) releases every web-IDS ban in one run.

Enabling debug output

The scorer's verbose debug logs are governed by _NGINX_DOS_LOG (SILENT | NORMAL | VERBOSE). The script's built-in fallback is VERBOSE, but autoupboa seeds _NGINX_DOS_LOG="SILENT" into /root/.barracuda.cnf (which the scorer sources, overriding the fallback), so the effective stock default is SILENT — _verbose_log writes nothing until you raise the level or set a debug marker. Two /etc/boa/ markers control the rest (both relocated from /root under C-003 — see the control files & INI topic):

The verbose log fans out by category: /var/log/scan_nginx_debug.log (general), /var/log/scan_nginx_flood_debug.log (counter increments), /var/log/scan_nginx_admin_debug.log and /var/log/scan_nginx_other_debug.log (ignored-URI traces). The category split and the Counter/admin/other traces only appear when _NGINX_DOS_LOG is VERBOSE; at NORMAL (or at the autoupboa-seeded SILENT with the .debug.monitor.log.cnf marker set) everything lands in the general log.

To watch a real classification decision, create the marker and run the scorer by hand (it takes no flags — it is launched by nginx_guard.sh, ten short overlapping passes per minute, gated by the single-instance lock):

touch /etc/boa/.debug.monitor.log.cnf
bash /var/xdrago/monitor/check/scan_nginx.sh
# At the autoupboa-seeded SILENT level the marker re-enables logging, but the
# flood/admin/other category split needs _NGINX_DOS_LOG=VERBOSE, so everything
# lands in the general log — inspect that:
tail -f /var/log/scan_nginx_debug.log
# (for the per-category split, set _NGINX_DOS_LOG=VERBOSE in
#  /root/.barracuda.cnf and watch scan_nginx_flood_debug.log instead)
rm -f /etc/boa/.debug.monitor.log.cnf   # remember to remove the marker afterwards

Self-healing watchdogs

The Abuse Guard's worker scripts can hang under a genuine flood (a web.log with thousands of IPs makes a single guest-fire pass take minutes). Several independent layers keep the pipeline from wedging — and they live in different scripts, not in scan_nginx:

1. In-process fire watchdog — _kill_stuck_fire (guest-fire.sh). At the top of every run it reads /run/fire.pid; if the recorded PID is alive and the pidfile's mtime is older than _FIRE_TIMEOUT (180 s, 3× the normal ~50 s run), it kill -9s the stuck process, removes the pidfile, and logs to /var/log/boa/fire_stuck.log. It then records its own PID so the next invocation can time it.
2. External fire watchdog — _kill_stuck_fire_external (autoupboa). Run during the weekly self-upgrade as a safety net independent of the pidfile. It enumerates pgrep -f guest-fire.sh, computes each PID's elapsed time from /proc/PID/stat field 22 versus /proc/uptime, and kill -9s any older than _FIRE_WATCHDOG_TIMEOUT (180 s), logging to the same /var/log/boa/fire_stuck.log. (The release notes and change log sometimes attribute both _kill_stuck_fire* functions to scan_nginx — that is wrong; they live in guest-fire.sh and autoupboa.)
3. Scorer single-instance lock — _manage_single_lock (scan_nginx.sh). scan_nginx does not have a fire-style timeout; it self-protects by refusing to pile up. It sources /opt/local/{bin,lib}/lock.inc and calls _single_instance_lock if the shared lock library is present; otherwise it falls back to a legacy pgrep guard that exit 0s when more than 2 instances are running, logging to /var/log/boa/too.many.log.

Independently, minute.sh's _csf_flood_guard kills runaway guest-fire.sh swarms (and CSF storms) when no protected run is in progress (/run/boa_run.pid absent):

• more than 9 fire processes → csf -tf + csf -df + pkill -9 -f fire.sh, logged to /var/log/boa/fire-purge.kill.log;
• more than 7 → csf -tf + pkill -9 -f fire.sh, logged to /var/log/boa/fire-count.kill.log;
• more than 4 csf processes → pkill -9 -f csf + csf -tf + csf -df, logged to /var/log/boa/csf-count.kill.log.

If you find one of these logs growing, the box is under sustained flood, not misconfigured — the watchdog firing is the system working.

Maintainer caveats

These are the things that surprise people who debug or extend the Abuse Guard.

Detection is post-hoc — it cannot block mid-burst

scan_nginx runs on a cron cadence and scores log lines that have already happened. A burst that completes inside one tick (dozens of requests in seconds) is served before the IP is ever written to web.log; what the scorer produces is a ban for the next visit, enforced in real time by the request guards. The fast-ban weights shorten the score needed, not the cron latency. If you need in-flight blocking, that is the real-time guard layer's job, not the scorer's. Do not "fix" the latency by shrinking the tick — you will just collide with the single-instance lock.

The realip dependency — a bad trust chain scores the wrong IP

scan_nginx scores only the last token of the forwarded chain — the value Nginx's realip module rewrote $remote_addr to. That is only correct if the realip trust chain is correct. If a customer's own reverse proxy is not in set_real_ip_from, every request through it is attributed to the proxy's address, and the Abuse Guard will score that one IP for all the traffic behind it — and can ban the whole site. Getting realip right (via cloudflare_realip.sh / migration_proxy_realip.sh and the rendered set_real_ip_from ranges) is a prerequisite for correct scoring, not an optional extra. Symptom to recognise: a single upstream/CDN IP racking up a huge score for traffic that is obviously many distinct clients.

The function-IFS gotcha — works-in-test, fails-live

scan_nginx sets a global IFS=$'\n\t' near the top of the script (no space). Any function that word-splits a space-separated config value must set a function-local whitespace IFS first, or the split collapses to a single token, matches nothing, and silently does the wrong thing — with no error. This already bit the ignore-path loop: _is_ignored_request now sets local … IFS=$' \t\n' before iterating _NGINX_DOS_IGNORE_PATHS. The two aggregate handlers (_handle_ddos_blocking, _handle_path_flood_blocking) use the save/restore idiom instead (_SAVE_IFS="${IFS}"; IFS=' '; …; IFS="${_SAVE_IFS}").

The trap is that a helper tested interactively passes (your shell's IFS contains a space) but fails inside the script (the global IFS does not). When a space-split helper "matches nothing" in-script, suspect the inherited IFS before re-auditing the matching logic. The rule for anyone extending the script is: word-split a space-separated value only after setting a function-local space IFS.

Related

• scan_nginx scoring engine — the detector logic and the real-client / exemption mechanics referenced above.
• The ban pipeline — what consumes web.log and produces the CSF bans you read here.
• Request guards — the real-time enforcement these operations ultimately drive.
• Configuration reference — the knob defaults behind the tuning here.
• Operator troubleshooting & recovery — wider blocked-IP recovery and load-incident triage.