It’s loading way faster than it was this morning.
Comment on Performance issues?
Sal@mander.xyz 2 weeks ago
I can see that since October 13 the CPU use spiked to 100%…
I just reset the server to check if that would fix it quickly. It don’t think it did.
I can see the processes that are taking up a lot of CPU (parallell database queries it seems), but I still don’t understand what is causing these and why they have spiked since Oct 13. I will need to investigate.
Bonus@mander.xyz 2 weeks ago
Sal@mander.xyz 2 weeks ago
Hmm, thanks!
I just made another change. I am not sure why the database suddenly started taking up so many resources, so I have reduced the amount of resources available to it. It did reduce CPU use to 80% as opposed to 100%, but it is still somewhat laggy for me.
Fourth@mander.xyz 1 week ago
Ty for your efforts
Sal@mander.xyz 1 week ago
No problem! Happy to contribute
Bonus@mander.xyz 2 weeks ago
Awesome, best of luck to you.
BTW, this brought me here today and had me noticing how slow it was loading:
Is there currently a fedaration issue between piefed.social and mander.xyz or am I doing something wrong? piefed.social/post/1369461#comment_8420089
Sal@mander.xyz 2 weeks ago
Thanks! It seems like the lag may have been due to a scraper
Sal@mander.xyz 2 weeks ago
So far, I have been able to ‘control’ the CPU use by setting limits to the process that pulls stuff from the database (pool size, CPU, memory).
This does release some of the CPU for other tasks, but I think that that what creates the lag might actually be the clogged database queries. So, constraining those resources might not solve the lag problem.
Image
admiralpatrick@lemmy.world 2 weeks ago
Check your web access logs for these 3 IPs:
And see if they’re making repeated un-scoped (no
pageparamater) requests to/api/v3/comment/list. If they are, block them in your firewall.Those used to hit my instance constantly with requests like
/api/v3/comment/list?sort=Old&page=16514(yes, page 16,414). When I blocked those IPs making those requests, problem solved.Sal@mander.xyz 1 week ago
Thanks!
I don’t see those specific IPs, nor 16514. But now I see what scrapers tend to look like in the logs :)
I am now pretty sure that the cause was scraper-like activity coming from the Mlmym front-end that I am serving over an onion site. I am not sure if it randomly started mis-behaving or if a tor scraper was using it.
After blocking this, federation was restored, performance increased, and CPU use came down:
Image
acockworkorange@mander.xyz 2 weeks ago
Did you upgrade Lemmy on the 13th or something? Could be a new bug.
Also, did site traffic increase at all? Maybe there’s an AI scraper messing up.
Sal@mander.xyz 2 weeks ago
I did not update or change anything in the past few days.
But, now that you mentioned an AI scrapper I looked into the logs and noticed some heavy requests to the API from a specific IP.
Requests look consistent to scraping - just consistently and continuously issuing GET requests to different API endpoints.
I have started denying their requests and it is the first thing that seems to have actually helped!
I don’t want to speak too early but I think you may have identified the cause. Thanks!
Sal@mander.xyz 2 weeks ago
I just realized that it is not a ‘scraper’, the requests came from the server that I am using to provide an interface to the site as an Onion site. The amount of requests was suspiciously high so maybe a bot is scraping through Tor. I will leave it off for a few days and see if I can turn it back on later.