The tech giants make enough money that they could keep on growing forever, from my understanding.
But the fediverse? Sure the main instances that get enough funding are going to be okay, but what about the single-user instances 10 years from now on when there’s a lot more content to download? Won’t they go bankrupt just by trying to annex the big instances?
And I have the impression that the lemmy giants are going to change over time: does that mean that 50 years from now on, the posts I’m posting here today might get lost in time because the instances that annex it will have shut down by then?
I probably misunderstand how the fediverse works, but my worry is that the small instances won’t be able to hold an ever-growing amount of data forever.
I spoke in absolutes for the sake of readability, but I’m as in-the-dark as can be.
You must log in or register to comment.
No, after a sufficient amount of time has passed, we would run out of useable matter and energy in the universe. This theorized end-state of heat death puts a finite cap on the size of the Fediverse.
Constrained to Earth, it’d probably be fine. Though I do see it splintering eventually, with sub-communities existing independently from the main organism.
But would it work with spherical servers in vacuum?
Time to invent the Dyson Server!
We already got one: the dyson.com server. /jk
Is this amount of time clearly stated or defined. Indefinite does not mean infinite.
Mostly serious answer: the current implementation is not going to scale effectively with growth. The software implementation is still rough around the edges, and the ActivityPub protocol probably needs more knobs to handle bulk data synchronization. Within the service, moderaton is a serious challenge with many unanswered questions.
Likewise, the back end software implementation is monolithic, meaning it’s one software stack that does everything from sign in to subscriptions to synchronization and scheduling. Housekeeping and garbage collection probably isn’t that tight, either. This is mostly speculation as I’ve watched things over the last couple of weeks’ growth.
I believe the data store is based on Postgres RDBMS, which while being robust and scalable is fussy and needs tuning when turning over large amounts of highly unique data.
None of this is an indictment on the devs! Rather the opposite, because the software IS chugging along while experiencing tremendous growth.
I expect over time the back end will devolve into micro services that communicate over a highly scalable, or stream-based messaging bus. Larger instances could probably also benefit from static caching and CDN techniques to keep pages loading quickly even while the back end thrashes.
The structure.if the ecosystem needs to strike a balance between fewer large instances and many-many small instances. In the first scenario, the scaling limit is in the monolithic stack, which introduces I/O bottlenecks and serialization delays (even if massively threaded). In the latter scenario, message state and synchronous distribution become challenging because a full mesh of federations could scale faster than network state tables have room to support. Some middle tier might be needed, and I have no idea what that might even look like.
So to answer your question, can it scale indefinitely? Probably not because we hit scaling limits pretty quickly on a number of dimensions. Nevertheless, smart people.are starting to hang out here, and I expect will take an interest in how it all works. Improvement is inevitable, and I think the early roadblocks will be overcome easily enough
There’s nothing wrong with a monolith. Microservices are not inherently more scalable. Their advantage is around scaling teams. If anything, a monolith can be more performant as in-process calls are much faster thent network calls.
There can be better efficiencies by disaggregating the full stack into microservices and making IPC calls among scalable workers versus strictly service-per-server models which, yes, incur scaling issues from network iowait. Modern network operating systems do this, which allows heavier loaded processes more access to resources while lesser loaded processes are deferred.
The world produces 15Mt of beans every year. The average shit post with beans has 700g of beans in it. This means Lemy can scale to around 22 billions shitposts/year. We have some margin.
#shitpost
This math checks out. I ran it through the bean calculator using OpenBeanAI. 32.33% of the simulations show these numbers.
Repeating of course.
Of course
Each instance only needs to hold the data from communities its users are subscribed to. And images live on their host instances anyway. No instance needs to hold the entirety of Lemmy. :)
Smaller instances don’t grab everything from every other server, it only grabs data from other servers when their users are subscribed to specific communities, also I suspect it doesn’t grab all historical data automatically (though I don’t know how much it does grab by default)
Right now there’s no migration tool for when instances shut down, but it should be technically possible someone just needs to implement it.
It doesn’t get everything. At least it’s not what kbin is doing, and I expect lemmy is the same. How it works for kbin is, once you subscribe you will start getting info about ANY change. A like, comment or anything like that.
So, someone likes a comment, anywhere on any instance. We’ll get sent that like. But we don’t have the comment. So, we fetch the comment. But, wait this one was a comment on a previous comment. So we’ll fetch that too. All the way to the post. Also any users involved (the liker, and commenters back to original poster) will also be fetched. The hierarchy to the comment that was liked is then built, and then the like itself is applied.
That’s why you will start to see a lot of old posts, but not all of them. It’s just going to slowly build up over time as people interact and of course you’ll get anything new.
Even with this I’m getting a LOT of content right now delivered.
EDIT: Not sure why I posted without finishing typing before.
I see, that clears up a lot, thank you! I just hope that Lemmy is, as you suspect, doing the same as kbin.
I guessy answer is, who cares? Don’t treat a social media account as some immortal time capsule of your life. Keep a photo album, write some diary entries, but don’t rely on any form of social media to be the historical record of your existance. If it’s inportant keep it somewhere you can ensure the preservation.
I’m pretty sure the world will continue long after we’ve forgotten beans and not pooping for X days.
I needed to be reminded of this, thanks.
Still, Reddit is probably the biggest and most accessible source of information in the world, written out of passion by people, experts, professors, neckbeards… trolls… uni students, researchers,
and I wish Lemmy could also become the archive that Reddit is, but if information has a high likelihood to get lost with time, why bother? It should then really only be treated as a very temporary social media which is… okay, I guess.
Everything is temporary. Nothing is permanent. Embrace it and live in the now.
I think people need to be reminded of two big things when it comes to Lemmy:
-
It is impermanent. Not intentionally, I’m sure most instances will try to keep all the posts for as long as possible. But we’re just hosting this stuff on independent servers (also known as “somebody else’s computer”) and we can’t rely on them to stay online forever.
-
Lemmy is NOT PRIVATE. You cannot delete your posts, and this is by design. You can edit them, but there’s an edit history, and even if there wasn’t, it would be impossible to ensure that the old versions of your posts aren’t stored on some random, rarely used instance. There is no big man in charge like Mark Zuckerberg that you can sue to delete your data. If you want to use Lemmy privately, DON’T POST YOUR PERSONAL INFO. Don’t post things that can be used to identify you. This is a public forum. Treat it like one. If you don’t like that, go somewhere else.
Sorry, #2 is kinda off topic, but I see a lot of confusion about what Lemmy is and isn’t.
Thanks, I’m also definitely confused about what Lemmy is and isn’t. This clears up a lot.
-
I’ve been wondering about that too. Specifically how long can old posts persist, and how long are instances compelled to host older content especially from other servers.
The Fediverse needs to encourage different instances. It’s the only way it can work. It has the technical framework to do it and for it to be transparent to the enduser but I feel like it’s not there yet.
For example I think users should be strongly encourages to chose regional instances instead of lemmy.world (I know know, ironic coming from me). It should be default and require the user to go out of their way to select a different instance. It should also be concisely explained that your instance doesn’t matter and that you can see any other federated instance. Yes, this is not always true but it doesn’t matter to someone just joining. Let them get here first and then they’ll naturally learn about the intricacies. Don’t scare them away at the gates.
I imagine the devs aren’t worried about this yet.
Long-term, I imagine that archiving and culling old data would maybe make sense.
Maybe there will be the equivalent of archive.org for lemmy only one day.
you can trigger https://web.archive.org/ to “Save Page Now”
It says below the input URL box “Capture a web page as it appears now for use as a trusted citation in the future.”
archive.org also has an extension that automatically scrapes webpages that haven’t been downloaded in 90/60/30/7days/24hrs
Mastodon already has something to purge posts older than X days And that’s with reletively small snippits of text generally. It’ll be essential for any kind of ling term running on small nodes to automate removal of posts to avoid blowing up the host’s disk space. There’s also the possibility of ‘purge after X period of time with no activity’ which would make sense for something like this where things can turn into long discussions more often than it does with a toot/tweet.
It doesn’t even work right, lemmy.ca and beehaw.org are not synchronized even if they federate with each others, certainly the same with multiple instances.
Yes, I’ve noticed that on Kbin.social, there are fewer visible comments than on Lemmy.world. It looks like quite a few comments aren’t being federated properly.
Why would an instance annex another instance? I can’t imagine the pain in trying to merge two databases like that, then handling the changing of the address.
If anything, I expect a single or a collection of larger instances could choose to defederate from the larger federation.
My bad. By annexing, I meant downloading posts from other instances.
It is basically like when a person goes to a webpage. If anything, it is more likely that a group of smaller instances to take down a big instance hosting tons of communities.
