Comment on Aggressive AI scrapers are making it kinda suck to run wikis
thatsnothowyoudoit@lemmy.ca 5 days ago
We use NGINX’s 444 response A LOT.
In coordination with careful rate-limiting, it’s been a dramatic improvement.
The worst of the bits don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.
grrgyle@slrpnk.net 5 days ago
Hmm interesting. I wasn’t aware of this one