Ask HN: Protecting your Sites/Services from Unwanted Traffic?

So... If PoW (proof-of-work) schemes like Anubis ultimately don't' work in practise, because Bots/Crawlers are increasingly using headless browsers, are able to solve captures, proofs, etc;

What options do we have realistically to filter out or block unwanted traffic?

What clever schemes can we come up that don't rely on centralised serices like Cloudflare?

Comments

mstaoruJul 26, 2026, 10:34 AM
Browsing is absolutely intolerable these days, with almost every website having a block screen, "we're checking your browser...", stuff like that. Lately I just close these websites immediately.
rubylimeteaJul 26, 2026, 11:53 AM
Cloudflare: "We reduced bot traffic by 80%"

Me: "Bro you reduced all traffic by 90%, my customers are pissed"

wanderingpixelJul 26, 2026, 9:06 AM
It depends on what you define as unwanted traffic.

Personally I do not have an issue with bots as long as they behave and are not straight up malicious, so I rely on a combination of rate limiting, a fine-tuned OWASP CRS ruleset and an aggressive Fail2ban enforcement (hit 2 triggers and you get a 24 hour ban, 2 bans and you get banned for 30 days).

My sites also make extensive use of static elements and caching.

prologicJul 26, 2026, 12:13 PM
How do you deal with DDoS attacks on your infra / sites? Especially ones from residential proxies or vpn networks? These are the hardest to defend against as they often are relentless and no amount of proof-of-work, captchas or otherwise seems to stop them from coming. I don't think it's something you can easily block at any firewall level, the sheer volume of IP(s) and Subnets is too much.
wanderingpixelJul 27, 2026, 5:12 AM
I limit slow requests with mod_reqtimeout and have e.g. reduced request body limits as I don't accept large uploads.

A real sustained DDoS would probably still require upstream handling by larger infrastructure than I run.

LeftiumJul 26, 2026, 1:47 PM
You can hide data in random static (for a novel captcha). Very obvious to most humans (if it becomes popular, the bots could probably be designed solve it; just requires diffing the images):

- https://fx.leftium.com/static

- https://www.youtube.com/shorts/tPfzNzy24to

---

I noticed most of the traffic to my hobby sites was bot traffic (probing for common vulnerabilities). So I added a robots.txt for well-behaved bots and rules for the bad ones. It works pretty well.

robots.txt: https://github.com/Leftium/robots-txt

Sample cloudflare rules:

    (
      http.request.uri.path contains "/wp-login"
      or http.request.uri.path contains "/wp-admin"
      or http.request.uri.path contains "/wp-content"
      or http.request.uri.path contains "/wp-includes"
      or http.request.uri.path contains "/xmlrpc.php"
      or http.request.uri.path contains "/wordpress/"
      or http.request.uri.path contains "/wp-json/"
      or http.request.uri.path contains "/wp-config"
      or http.request.uri.path contains "/wp/"
      or http.request.uri.path.extension eq "php"
      or http.request.uri.path.extension eq "aspx"
      or http.request.uri.path.extension eq "asp"
      or http.request.uri.path contains ".env"
      or http.request.uri.path contains "sftp-config"
      or http.request.uri.path contains ".vscode/"
      or http.request.uri.path contains "credentials"
      or http.request.uri.path contains "stripe"
      or http.request.uri.path contains "secrets"
      or http.request.uri.path contains "config.json"
      or http.request.uri.path contains "config.yml"
      or http.request.uri.path contains "config.yaml"
      or http.request.uri.path contains "config.py"
      or http.request.uri.path contains "config.inc"
      or http.request.uri.path contains "config.rb"
      or http.request.uri.path contains "settings.json"
      or http.request.uri.path contains "settings.yml"
      or http.request.uri.path contains "settings.py"
      or http.request.uri.path contains "application.yml"
      or http.request.uri.path contains "application.properties"
      or http.request.uri.path contains "parameters.yml"
      or http.request.uri.path contains "appsettings.json"
      or http.request.uri.path contains "docker-compose"
      or http.request.uri.path contains "/.git/"
      or http.request.uri.path contains "/.circleci/"
      or http.request.uri.path contains "/.aws/"
      or http.request.uri.path contains "/.bitbucket/"
      or http.request.uri.path contains "/.travis.yml"
      or http.request.uri.path contains "serverless.yml"
      or http.request.uri.path contains "debug.log"
      or http.request.uri.path contains "error.log"
      or http.request.uri.path contains "/storage/logs/"
      or http.request.uri.path contains "_profiler"
      or http.request.uri.path contains "phpinfo"
      or http.request.uri.path contains "/debug/"
      or http.request.uri.path contains "/horizon/"
      or http.request.uri.path contains "/manage/"
      or http.request.uri.path contains "fileman"
      or http.request.uri.path contains "assetmanager"
    )
kassnerJul 26, 2026, 7:04 PM
In a similar tone, I made Caddy module that temporarily blocks an IP address once it tries to access specific URLs: https://github.com/kassner/caddy-trapdoor It’s great to catch those bursts of requests, nowadays I see a single IP probing hundreds of paths in a couple of seconds. This catches majority of them for me.
prologicJul 26, 2026, 2:16 PM
I'm actually almost tempted to build a reverse proxy that detects this type of "scanning" behaviour and just drops it to the floor.
dooltaJul 26, 2026, 10:56 AM
[flagged]
lpsatworkJul 26, 2026, 10:27 AM
[flagged]
xchiliJul 26, 2026, 5:51 AM
[dead]
monter7484Jul 26, 2026, 1:07 PM
[dead]