Lemmy works by sending a message between servers, waiting for an OK, sending another message, waiting for an OK, etc.
That means if servers are on opposite sides of the world and it takes 0.2 seconds to send a message and get an acknowledgement, you have a hard limit of 5 messages that can be sent per second, even if they’re both on 100 gigabit links.
Lemmy.world was sending aussie.zone about 16,000 messages per hour, which is about 4.5 per second. So around 220 ms per message. And it wasn’t keeping up.
Now lemmy.world has… it looks like two separate connections sending messages to aussie.zone, so it can have two messages in flight at a time.
T156@lemmy.world 3 days ago
Basically, Lemmy works by having a whole.bunch of different servers chitchat to each other. Hence how I can post a reply to you, even though you’re on Aussie.zone, and I’m on Lemmy.world.
The problem was that because Lemmy.world was big, and there were a lot of people in Aussie.zone who had feeds over there, Lemmy.world couldn’t fire stuff off quickly quickly enough.
Analogically, you’re basically getting a telegram to your house every day, saying what’s on the news. Except that there are a lot of people, so the telegraph operator can’t write and send the telegrams out quickly enough, and the message started backing up.
Lemmy.world basically enabled multi-threaded sending, which works like hiring more telegraph operators. While they could now keep up, there was still a huge amount of backlog that they had to send.
It’s only until this post that they finally caught up.