Experiencing intermittent 503 errors from lemmy world using from photon frontend.
These urls return 503: lemmy.world/api/v3/site? lemmy.world/api/v3/user/mention?limit=20&page=1&u… lemmy.world/api/v3/private_message/list?limit=20&… lemmy.world/api/v3/user/replies?limit=20&page=1&u…
error body:
{“error”:“Offline”}
MrKaplan@lemmy.world 1 day ago
Hi,
sorry for the late response to this. The immediate issue is resolved, and we’re looking to extend our monitoring in the coming days to have better insights on the underlying issue, as well as alerts before it would get to the point of throwing errors.
For some reason Lemmy has been having significantly more open file handles (including e.g. connections) than usual, leading to a denial of service. At this point it’s unknown whether this is caused by a bug in Lemmy, an attack, or something else.
laurenceOfSuburbia@lemmy.world 1 day ago
Hey thank you for looking into it. Maybe some kernel tuning needed?
MrKaplan@lemmy.world 1 day ago
Unlikely, we haven’t changed these limits recently and we haven’t exceeded the limits in the past. While it’s possible to change the limits, it’s not clear yet why they are being exceeded currently.