Yay, nice to see mander back up and running.
May I ask, do we know what happened?
Also, should we maybe have a matrix chat or something as a backup communication channel?
I second that, sounds like a good idea.
This place is back online!
Submitted 3 weeks ago by BlastboomStrice@mander.xyz to mander@mander.xyz
Comments
Gloomy@mander.xyz 3 weeks ago
max_dryzen@mander.xyz 3 weeks ago
Thank you Salamander I love this instance
Zoop@beehaw.org 3 weeks ago
I’m glad y’all are back! I hope you stick around!
Successful_Try543@feddit.org 3 weeks ago
BlastboomStrice@lemmy.dbzer0.com 3 weeks ago
Lol, we didnt, hopefully we’ll back back online soon
Shdwdrgn@mander.xyz 3 weeks ago
Woot!
Fourth@mander.xyz 3 weeks ago
🌾🤘😤🤘🌾
M0oP0o@mander.xyz 3 weeks ago
Yay, I can post again.
There is some federation issues still how ever, this very post should have a few more comments in it.
nettle@mander.xyz 3 weeks ago
And I can post images again!
(I’m guesing the outing was due to server changes that allowed this)
M0oP0o@lemmy.world 3 weeks ago
Still down for me…
Colloidal@programming.dev 3 weeks ago
Is it? Not for me, at least.
Sal@mander.xyz 3 weeks ago
AAAHHHH 😭 Such a relief. This was stressful.
The websites makes use of Contabo’s object storage service for storing and serving images. Earlier during the month, I got an e-mail saying that there would be a 12-hour downtime window due to a migration. Alright, no problem.
A bit before this users reported problems with uploads. I figured this was related to the migration.
After the “migration”, images still caused problems due to some “administrative rule” preventing uploads. I sent messages daily to Contabo. At first they were responding that “their engineers were looking into it, that this was related to the migration, and that they had no time estimate”. OK, I can be patient…
After a few days I lost some of my patience and pressed them more often, until they started sending me generic messages and eventually saying that “the problems were already fixed” and suggesting it may be some configuration error on my side.
This is where it got stressful, because my knowledge of the implementation of the images back-end is superficial. So… That meant I would need to study the details and either fix it myself or show proof that it was their fault.
These past few days I have tried to do some reading and tested configuration changes, but I am the kind of person that puts too much on their plate and I have been quite busy, so this has been an additional stress point.
Today, images suddenly started working (until it came down again due to some logs piling up). This tells me that it was not a configuration issue and it was with high probability something on Contabo’s side that they fixed today. Uploads are working as well now. Contabo support never shared any information about the specifics of what went wrong, so I don’t know.
This doesn’t mean that I don’t need to study the details, but at least now the pressure is lower. My plan now is to do some studying on how images are stored and served and once I have a good understanding I can migrate the instance to a dedicated server. I will also look into techniques for making a server more resilient when things do go wrong.
BlastboomStrice@mander.xyz 3 weeks ago
Oi, thanks for all of your hard work Sal!
I was thinking, I’m just interacting with lemmy and when the server goes down I almost never expect it to not go up again. But it doesn’t run on magic, for the server to go back up online and work properly, you work a lot (depending on the severity of the issue) and like today it can get really stressfull:/ What I might take for granted, it really isn’t on your end.
So thanks again for making this place alive again, good luck learning more about the details of image storage (hopefully there might be people who can help on lemmy) and I also see that pictures work properly now, congratulations:))
Redfox8@mander.xyz 3 weeks ago
Good work and thanks for the update and info :) a good reminder that there’s people out there looking after our communities whilst we doomscroll away! ;)
fossilesque@mander.xyz 2 weeks ago
Image
amos@mander.xyz 2 weeks ago
Have you thought of creating a mastodon or bluesky or whatever account just for these cases? It would have been nice to know that you were aware that the server mander was not working. We did not know if something had happened to you, or if you had decided to close mander or whatever.
Also, have you thought of adding an additional admin?
Mac@mander.xyz 3 weeks ago
We appreciate the effort you are putting in and you are doing a great job.
Thank you. :)
Aviandelight@lemmy.world 2 weeks ago
Thanks Sal! Glad to know that even though this was stressful that you’re okay.