Cross posted from: reddthat.com/post/66285192
We use NGINX’s 444 response A LOT.
In coordination with careful rate-limiting, it’s been a dramatic improvement.
The worst of the bits don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.
poVoq@slrpnk.net 5 days ago
Not only wikis sadly. Anything that has public facing deep links that trigger extensive database operations are being hammered by these bots and few servers can take the load.