Status update July 4th
Just wanted to let you know where we are with Lemmy.world.
Issues
As you might have noticed, things still won’t work as desired… we see several issues:
Performance
- Loading is mostly OK, but sometimes things take forever
- We (and you) see many 502 errors, resulting in empty pages etc.
- System load: The server is roughly at 60% cpu usage and around 25GB RAM usage. (That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%)
Bugs
- Replying to a DM doesn’t seem to work. When hitting reply, you get a box with the original message which you can edit and save (which does nothing)
- 2FA seems to be a problem for many people. It doesn’t always work as expected.
Troubleshooting
We have many people helping us, with (site) moderation, sysadmin, troubleshooting, advise etc. There currently are 25 people in our Discord, including admins of other servers. In the Sysadmin channel we are with 8 people. We do troubleshooting sessions with these, and sometimes others. One of the Lemmy devs, @[email protected] is also helping with current issues.
So, all is not yet running smoothly as we hoped, but with all this help we’ll surely get there! Also thank you all for the donations, this helps giving the possibility to use the hardware and tools needed to keep Lemmy.world running!
I’m just excited to be back in the Wild West again – all of the big players had bumps, at least this one is working to fix them.
And giving updates!
I’d rather have to deal with hiccups and bumps along the way, because the community only grows more each time.
I still remember early reddit days of 502 it went through, 504 try once more.
Early? Haha, reddit still shits the bed regularly
Yee-haw, mother fuckers.
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A pretty bad one at that…
But… but… Rust…
Rust makes holes and that’s how leaks happen
Underrated comment
Rust protects you from segfaulting and trying to access deallocated memory, but doesn’t protect you from just deciding to keep everything in memory. That’s a design choice. The original developers probably didn’t expect such a deluge of users.
Leaking memory is safe
I’m calling it - if there’s actually a memory leak in the Rust code, it’s gonna be the in memory queues because the DB’s iops can’t cope with the number of users.
On some of the latest release candidates I think so.
Lemmy has a memory leak? Or, should I say, a “lemmory leak”?
A lemmory meek, obviously!
Wait isn’t lemmy written in rust how do you create a memory leak in rust? Unsafe mode?
let mut memory_leak = Vec::<u8>::new(); loop { memory_leak.push(0); }That’s not a memory leak though. That’s just hording memory. Leaked memory is inaccessible.
In the example it’s inaccessible.
It’s not. The vec is still accessible and if it goes out of scope rust automatically clears the memory.
But it will never fall out of scope because of the loop
Exactly.
Exactly but it’s still accessible since it’s in scope.
Really appreciate all the time and effort you all put in especially while Lemmy is growing so fast. Couldn’t happen without you!
As a game dev for bigwigs I know all too well about memory leaks, and so very much appreciate your patch notes, updates, and transparency. You’re doing great with such fast exponential growth
💙 Thanks for your hard work!
Amen. So happy to be done with Reddit.
What was that? We’re going to need more and better hardware soon, and you have a Patreon and a paypal on the sidebar?
Yeah, that sounds pretty reasonable, we can work with that.
There is a sidebar?
On web browser, yep.
I’m thoroughly confused by most of what is going on. I’m using kbin on mobile. The interface isn’t terrible and it fills the random surfing void that I am used to.
Just a heads up, you’re using kbin which is a different “website” than lemmy. You are also seeing a thread that originated in lemmy. That means both kbin and lemmy can see each other’s content, but the website UI is different between them so if you read about some UI element and can’t find it in kbin that might be one of the reasons.
That being said, kbin also has a sidebar with useful information. In the mobile site you can find it at the very bottom on the page, since it doesn’t fit to the side in such a small screen.
Omg I never scrolled that far down lmao, now I can go block 196 and the other shit posting magazines
Yesss! Lemmy was difficult to use until I did that. Then eventually I had a long enough list of communities/magazines that I could browse subscribed only and that really supposed the issue. I’m busy not a fan of random porn and bigoted memes y’know?
Every community has a sidebar. If you’re at the page for the community it should be at the top of the website. “sidebar+”
If you’re using a 3rd party app it’s different for each one in terms of how to see it.
Just signed up for the 8€/month tier, hopefully my blue checkmark will arrive soon!
Could I get a discord invite? I’m an ex sysadmin with a. Lot of free time
@ruud > That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
Hmm, makes me curious if there is a Lemmy memory leak, or simply that the load wants to stabilize above of the RAM you have? I hope contributions can help you with another 32 GB RAM? Thank you for your work! 🍻
We have 128GB of RAM. It just skyrockets after a while!
@ruud Oh damn. This spontaneously sounds crazy but I’m admittedly a novice at servers on this scale.
I haven’t checked into the code yet, but I imagine you can map out what all is in memory and force more aggressive garbage collection to find some middle ground.
Rust is not a GC language.
As somebody who flocked to Voat during the height of the Ellen Pao controversy and remembered the site being rendered unusable for whole days at a time from the Reddit Hug of Death, I’m remarkably surprised at how well Lemmy.world has held up. I thought the fediverse would have truly crumbled from this exodus.
I remember when Voat came out and the slight exedous that brought. I made an account and everything but it never properly took off. I checked on it two or three years later and it was just filled with alt-right/racist/transphobic garbage. Sad it never took off as a reddit alternative, reddit likely would have greatly benefited from a proper alternative, not sad it closed down after I saw what it ended up.
So far the fediverse feels really different tho, very explicitly anti that type of shit. I’m sure it will pop up, they always do, but maybe now people know how to deal with it. Block it, defederate, deplatform.
FYI, it has popped up
explodingheadsis a great example but many servers including lemmy.world became proactive in defederating from the instance
Lemmy is a distributed system that can’t quite be perma-killed, kind of like ye olde usenet. Anyone can create a new instance at will.
The work you’re doing is greatly appreciated! It’s like you invited half the internet into your house. I feel like I should’ve brought a cake or something
You should consider setting up a small donation to the site, if you can afford it. If you throw $5 a month at them, you’re not only covering your own hosting fees, but covering for some of the folks who can’t afford to as well. And maybe Ruud gets to buy some beans.
It appears I have no idea how to imbed an image, but good call. Subscribed.
It’s a ton better this afternoon! Thank you!!
I’ve noticed that too!
Of course these performance issues are a bit annoying, but I gotta say that I love these updates and explanations here. Great communication, keep it up, please!
That is, if we restart Lemmy every 30 minutes. Else memory will go to 100%
who’d have thought memory leaks would be possible in Rust 🤯
(sorry not sorry Rust devs)
c/programmingcirclejerk when?
Ackchually, leaking memory is totally possible in Rust! But so is hogging way too much of it because you’re hanging on to too many things in memory at once🤷
Huge respect for what you’ve built here, but it might be worth reaching out to the lemm.ee admin. I only know enough DevOps and cloud hosting to be dangerous, not helpful. But his instance seems stable and scalable. He might be able to offer some insight into the issues here
Yes he’s one of the other admins in our Discord, he’s very helpful!
Keep up the good work!
I created an account in lemm.ee until the issues are fixed. Then I will happily go back to my lemmy.world account.
Lemm.ee is also a good choice!
Thank you so much for your hard work and for fixing everything tirelessly, so that we can waste some time with posting beans and stuff lol.
Seriously, you’re doing a great job <3
Great distraction from the collapse
