Submitted 2 weeks ago by walden@sub.wetshaving.social to wetshaving@sub.wetshaving.social
LLM scrapers and bots are a continuing challenge. Lemmy is not behind Cloudflare and custom filtering rules using NGINX only gets you so far.
I’m trying the setting called “Private Instance”.
It’s my understanding that all this does is require a user to be logged in to use the instance. This will hopefully cut down or stop the constant scraping. Preliminary signs are that it helps a ton.
Do you want to give Anubis a try? That has helped our instances avoid the bots
Another vote here for anubis! It cut our traffic down by some 75% or so.
I’ve heard mostly good things about Anubis, but I’ve also heard it can be hard to get it right. I’ve never looked into it so I’m not sure what people are referring to… maybe the balance between blocking lots of bots but still letting humans through?
This Lemmy instance is mostly sticking around for posterity, so hopefully Private Instance cuts down on the B.S. so I don’t have to think about it any further!
I had some challenges setting it up which were mostly related to using load balancing for Lemmy-UI. In general there isn’t much of a problem letting humans through so long as you only apply it to the front end UI not to the API.
We have a rule to allow anyone with a jwt set which means logged in users are not challenged.
Finding an instance that blocks content from logged out users would likely put off new users, but if you’re not looking for new users then requiring login seems like an easy solution to the problem.
dbzer0 is using haproxy with haphash successfully to mitigate.
https://github.com/dgl/haphash
He might be willing to share the uri/paths to approve in your proxy config to deploy haphash or anubis etc.
walden@sub.wetshaving.social 1 week ago
@otter@lemmy.ca @hal@pifed.nz thanks for the encouragement to try Anubis.
The instance is now running using the Ansible playbook, with Anubis enabled. I changed the server back to non-private (which didn’t help, anyway).
I’ve been interested in switching to running Lemmy via the Ansible playbook for a long time, but now that I’ve done it I miss “my way”! I’ll either have to go back to “my way” or fork the playbook and learn how to make ansible do things how I like it.
Mostly, I need to figure out how to have everything go through Nginx Proxy Manager instead of the built in nginx container. I like having everything on an
nginxdocker network so they can use hostnames to find each other instead of IP addresses.