Submitted 9 months ago by squashkin@exploding-heads.com to freeforum@exploding-heads.com
Site has been unusable for me intermittently for like a week or something, do we have some ideas for how we could mitigate the DDOS issue?
Are we looking for something like Cloudfare but not Cloudfare?
Increased server capacity?
Backup instances to move to when one gets overloaded?
(Cross-posted from exploding-heads.com/comment/274147)
Donations maybe?
We don’t know what the problem is. DDOSs are causing one of two things. Hardware overloading or network overloading. If the problem is the first one we just donate for better hardware. If it is the second one Kapow would need to upgrade his network connection. Here I am assuming he is self hosting it.
If he is running a VPS then he would need to upgrade his VPS plan to more expensive one.
I talke with Kapow briefly about it. Honestly hard to reply because when I click post we are under attack. I get board waiting and go do something else. We’ve been looking into migrating to nostr. I would say the nostr ecosystem isn’t ready yet. zapddit.com with nos2x-fox on desktop plus Amethyst for phone isn’t terrible. flycat.club is quirky and fun but really slow with communities. I get lots of javascript framework garbage errors running it locally. I like it’s old looking interface though.
I suggest we get a relay up or two and try to get people to make nostr accounts so we can communicate when Exploding-heads is down. We can put our public keys in our profile for easy visibility. and maybe a list of relays we like to use.
I can totally host a relay on like wss://relay.wolfballs.com. I would prefer to keep it to only this tribe of people if possible.
I can also look into hosting a mostr bridge to get all these post pushed to nostr relays. That SHOULD already work. These both would take very little time commitment on my part.
Nostr is nice because if our relays are down you can just send to a bunch of other ones. People can still find you.
Really hard to ddos everyone.
When election season comes around I imagine ddos attacks will pick up. I think these attacks aren’t really politically motivated and likely people just playing and learning what is effective against lemmy instances. Also possibly a uptick in bots scanning lemmy instances because they are more popular. AI training bots and otherwise.
Honestly I think cloud fare would work fine for EH. I doubt they would take it down for speech issues.
If we need to pay for it we can do some crowd funding. I sent kapow a little etherium (I’ve been using this site lately so worth it) and if like 5 people did it could fund a few months it can’t be that expensive.
Migration to nostr would be good, but the current state of it isn’t perfect and is not ready to use with communities and use it like we are using EH rn.
The cause of the ddos might actually be not powerful enough hardware because EH is hosted on linode which offers 40gbps incoming traffic on all their servers. No matter dedicated or shared.
I don’t support the usage of Cloudflare. We give too much control over all the data to it. It also doesn’t protect agains not powerful enough hardware.
Migration to nostr would be good, but the current state of it isn’t perfect and is not ready to use with communities and use it like we are using EH rn.
Totally agree. Should be more like a backup right now. It’s probably atleast a year away maybe two from being a decent competitor. It will happen though.
I don’t support the usage of Cloudflare. We give too much control over all the data to it. It also doesn’t protect agains not powerful enough hardware.
Fair enough.
admin@exploding-heads.com 9 months ago
We already sit behind the Akami DDOS prevention system. I do not think cloudflare would be any more effective.
When I look at the server - I am seeing the same behavior I saw when the big Lemmy instances started upgrading to v18. And these recent problems started when they started upgrading to release candidates for v18.3 - So I think this is a Lemmy federation problem not DDOS attacks.
We have plenty of bandwidth - do not even coming close to using it all. When we have these problems I am seeing spikes in CPU usage (normally 10% but spikes to 150% when we see these issues) and Disk I/O.
We plan upgrading the version of Lemmy we are running this weekend to see if that makes a difference.
Owner_of_donky@exploding-heads.com 9 months ago
You are a good admin
Freedom@exploding-heads.com 9 months ago
Glad to hear man. Thanks for the hard work