Comment on Aggressive AI scrapers are making it kinda suck to run wikis

thatsnothowyoudoit@lemmy.ca ⁨5⁩ ⁨days⁩ ago

We use NGINX’s 444 response A LOT.

In coordination with careful rate-limiting, it’s been a dramatic improvement.

The worst of the bits don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.

source
Sort:hotnewtop