It’s not the only one. Aussie.zone, being geographically far away from Lemmy.World, has suffered greatly for like six months already. And even my prior instance Discuss.Online that was so on the ball seems to have fallen prey eventually - unless that was just a gap during its upgrade to 0.19.7.
e.g. here’s a post with ~185 upvotes, though when viewed from Lemmy.world it shows 192 up and 2 down, so 190 total, but the real trick is that when viewed from discuss.online it has only 97 upvotes - that’s almost half of them they got lost!?!?!? And that’s nothing: check out the post when viewed from StarTrek.website, with zero additional upvotes (beyond the default first one) and also no comments visible on it as well.
As we see there it is not just votes - it is also comments, and as we are discussing here about cats, while entire posts too (although sometimes those catch up, whenever someone votes or comments on it later on and that triggers the entire post to have another chance to federate to an instance that would have skipped it otherwise). This level of lack of proper federation is enough to have killed off Lemmy entirely imho, even if that process took several months, if it weren’t for the fact that a fix has already been deployed and we are all just waiting for Lemmy.World to upgrade to it on 0.19.6 or its follow-up bugfix 0.19.7.
we’re currently aware of delayed federation from lemmy.ml towards lemmy.world and still working identifying the root cause - see https://lemmy.world/post/22196027 (still needs updating that it’s happening again).
aussie.zone has been about 6 weeks behind lemmy.world for a few weeks i think at this point, which at least means they’re no longer losing activities, but it’s still taking ages to reduce the lag.
i don’t know what issue there might be with discuss.online right now, but for startrek.website the explanation is rather simple. as you can see in the sidebar, there are 0 local subscribers for the community. when there aren’t any subscribers to a community on an instance, the instance will not receive any updates for posts in that community. this includes posts, comments, as well as votes.
startrek.website also had federation issues over the last weeks due to accidentally blocking lemmy instances in some situations.
lemdro.id has recently had some db performance issues that caused it to get around 3d behind lemmy.world, they’ve been slowly catching up again over the last days.
Thanks for the info! If discuss.online was out for a few hours, would that explain the missing content, if it happened during the outage and so it just gets lost forever, or given enough time will it catch up?
Do you know when Lemmy.World plans to update to 0.19.6 or 0.19.7? I really hope that helps bring stability! Although I can understand not wanting to do it at the same time as the sync issue with lemmy.ml still happening.
downtime should not result in missing content when the sending instance is lemmy 0.19.0 or newer. 0.19.0 introduced a persistent federation queue in lemmy, which means it will retry sending the same stuff until the instance is available. depending on the type of down, it can also be possible that there is a misconfiguration (e.g. “wrong” http status code on a maintenance page) that could make the sending instance think it was successfully sent. if the sending instance was unreachable (timeout) or throwing http 5xx errors, everything should be preserved.
we are planning to post an announcement about the current situation with lemmy updates and our future plans in the coming days, stay tuned for that. you can find some info in my comment history already if you are curious.
Ah, then something is indeed very wrong if discuss.online is missing so much content from a week ago (I thought after something like 7 days it will simply give up and stop trying), and startrek.website is doing far worse than that even.
Though sh.itjust.works caught up even as we were talking about it so… there’s some hope I suppose. And either way, thanks for any efforts you are doing to help with it - well, on the LW side at least:-).
there is indeed a cutoff. there is exponential delay for retrying and at some point lemmy will stop trying until it sees the instance as active again.
there is also a scheduled task running once a week that will delete local activities older than a week. downtimes of a day or two can generally be easily recovered from, depending on latency it can take a lot more time though. if an instance is down for an extended time it shouldn’t expect to still get activities from the entire time it was offline.
Although the example I mentioned earlier is only 6 days old, so if the cutoff is 7 days then that point should not have been reached yet.
That post also says “Comments are disabled.”, so now I wonder if something special is going on with it, which either caused the comments to be disabled or perhaps the disabling of comments (for whatever reason? I didn’t see anything happening inside that looked remotely contentious, but all the posts submitted around that time seem to have had that happen to them? perhaps a new mod learning the ropes, maybe even fat-fingering buttons unintentionally?) caused federation issues specifically with that content. On the other hand, many other examples abound, even if this were not a great example on its own.
But it is a quite striking one: the original post still has 185 upvotes, when viewed from Lemmy.World it has a handful more, but when viewed from discuss.online it is still far behind at merely 97 upvotes, and when viewed from StarTrek.website still has just the default 1 upvote… and zero comments visible underneath it. At this point if looks like if the delay lasts for one more day the situation will become permanent for that post.
And I’ve already decided to abandon the Startrek.website instance due to constant issues like this - especially if a post takes a day or two to federate, or almost worse it does federate but then the comments cannot make it back from LW to the STW instance for me to be able to respond to them (as has happened before), then on that instance I am essentially out of the loop wrt my own postings on Lemmy, unable to respond to what I can’t even see, at least there on STW where I have a login account, although I can see it on LW where I do not.
I really hope that the upcoming 0.19.6-7 helps make usage of smaller instances more viable as we continue this experiment in federated Lemmy.:-)
as far as i know, locking a post does not affect voting, only prevents new comments from being federated.
the other example you mentioned, i assume that you’re referring to the inconsistency on lemmyverse.net? i haven’t looked at how that application works, but it’s unlikely to be working with activitypub/federation, instead it’s most likely just connecting to various different instances and using their APIs. i’ve also left a comment over there about that.
i already explained before why these posts don’t see votes on startrek.website - there is no local subscriber on that instance. once at least one person from that instance subscribes to the community it’ll start seeing updates, which includes votes. there has also been a comment by one of the startrek.website admins about the federation issues caused by them accidentally blocking certain traffic from other instances here.
for discuss.online, there does not seem to have been a longer federation delay according to this dashboard, only about 1.5h delay at some point that was recovered from fairly quickly. it is also very possible that the first subscriber to the community on discuss.online only subscribed after the post was created, as the more recent posts seem to be doing just fine with their vote counts when comparing discuss.online and lemmy.world numbers. looking at our database, i can see the first subscriber to that community from discuss.online joined about 5 hours after the post was posted, which would easily explain the partial votes.
I might have a misunderstanding about how federation of posts works: when there are no subscribers to a remote community on a particular instance, I thought in the past that I have seen not only missing votes and comments, but the entire post itself would be absent as well? e.g. if the last subscriber from an instance leaves a community, then it will retain the archived posts but not bring in any new ones, until someone subscribes again (I had this situation with the “just post” community, from discuss.online, iirc). So then I do not understand why I am seeing posts at all for that community on StarTrek.Website, if nobody from there was subscribed to it? On the other hand, I do begin to see what you are saying: every single post there has zero upvotes and zero comments.
Thank you for explaining all of this btw, and for weighing in on that Lemmyverse post as well:-).
Yeah it looks fixed now. I went to [email protected] and counted 10 posts from the last 3 hours, then viewed the community from SJW and it too has 10 posts from the last 3 hours.
So… are you enjoying the cats flooding your feed now?:-D Or if you aren’t seeing them, it must be for some other reason now.
Ah that seems to explain it. Active prioritizes many comments iirc, but on most of these posts I see few if any comments. Using Hot though, they can dominate a feed due to being highly upvoted.
Ah, yeah that seems to be it. I switched to all+hot and I’m now seeing a lot of cats. I wonder what other content I’m missing out on by sorting by “active”? When I sort by “all”, I’m trying to get the posts that are trending the most on Lemmy. Maybe active isn’t the best way to do this? Interestingly, the posts that I see when I sort by “hot” don’t have as many upvotes as those when I sort by “active” so idk.
Hot also shows newer content, as these cat posts tend to be, while Active will allow older content that is still being discussed. You should notice a lot more turnover of your feed with Hot than with Active. You can change your default sort in your Settings so you don’t have to keep switching it every time.
Yeah, I have my default sort set to “Subscribed” + “Scaled” (I want to primarily see the stuff that I follow, and I don’t want the small communities to be drowned out by the large ones). When I want to see what’s trending (big topics that are being talked about a lot) right now, I’ll switch to “All” + “Active”. It seems like “Hot” favors boosting trending groups rather than trending single posts.
It’s not the only one. Aussie.zone, being geographically far away from Lemmy.World, has suffered greatly for like six months already. And even my prior instance Discuss.Online that was so on the ball seems to have fallen prey eventually - unless that was just a gap during its upgrade to 0.19.7.
e.g. here’s a post with ~185 upvotes, though when viewed from Lemmy.world it shows 192 up and 2 down, so 190 total, but the real trick is that when viewed from discuss.online it has only 97 upvotes - that’s almost half of them they got lost!?!?!? And that’s nothing: check out the post when viewed from StarTrek.website, with zero additional upvotes (beyond the default first one) and also no comments visible on it as well.
As we see there it is not just votes - it is also comments, and as we are discussing here about cats, while entire posts too (although sometimes those catch up, whenever someone votes or comments on it later on and that triggers the entire post to have another chance to federate to an instance that would have skipped it otherwise). This level of lack of proper federation is enough to have killed off Lemmy entirely imho, even if that process took several months, if it weren’t for the fact that a fix has already been deployed and we are all just waiting for Lemmy.World to upgrade to it on 0.19.6 or its follow-up bugfix 0.19.7.
we’re currently aware of delayed federation from lemmy.ml towards lemmy.world and still working identifying the root cause - see https://lemmy.world/post/22196027 (still needs updating that it’s happening again).
aussie.zone has been about 6 weeks behind lemmy.world for a few weeks i think at this point, which at least means they’re no longer losing activities, but it’s still taking ages to reduce the lag.
i don’t know what issue there might be with discuss.online right now, but for startrek.website the explanation is rather simple. as you can see in the sidebar, there are 0 local subscribers for the community. when there aren’t any subscribers to a community on an instance, the instance will not receive any updates for posts in that community. this includes posts, comments, as well as votes.
startrek.website also had federation issues over the last weeks due to accidentally blocking lemmy instances in some situations.
lemdro.id has recently had some db performance issues that caused it to get around 3d behind lemmy.world, they’ve been slowly catching up again over the last days.
Thanks for the info! If discuss.online was out for a few hours, would that explain the missing content, if it happened during the outage and so it just gets lost forever, or given enough time will it catch up?
Do you know when Lemmy.World plans to update to 0.19.6 or 0.19.7? I really hope that helps bring stability! Although I can understand not wanting to do it at the same time as the sync issue with lemmy.ml still happening.
downtime should not result in missing content when the sending instance is lemmy 0.19.0 or newer. 0.19.0 introduced a persistent federation queue in lemmy, which means it will retry sending the same stuff until the instance is available. depending on the type of down, it can also be possible that there is a misconfiguration (e.g. “wrong” http status code on a maintenance page) that could make the sending instance think it was successfully sent. if the sending instance was unreachable (timeout) or throwing http 5xx errors, everything should be preserved.
we are planning to post an announcement about the current situation with lemmy updates and our future plans in the coming days, stay tuned for that. you can find some info in my comment history already if you are curious.
Ah, then something is indeed very wrong if discuss.online is missing so much content from a week ago (I thought after something like 7 days it will simply give up and stop trying), and startrek.website is doing far worse than that even.
Though sh.itjust.works caught up even as we were talking about it so… there’s some hope I suppose. And either way, thanks for any efforts you are doing to help with it - well, on the LW side at least:-).
there is indeed a cutoff. there is exponential delay for retrying and at some point lemmy will stop trying until it sees the instance as active again.
there is also a scheduled task running once a week that will delete local activities older than a week. downtimes of a day or two can generally be easily recovered from, depending on latency it can take a lot more time though. if an instance is down for an extended time it shouldn’t expect to still get activities from the entire time it was offline.
Although the example I mentioned earlier is only 6 days old, so if the cutoff is 7 days then that point should not have been reached yet.
That post also says “Comments are disabled.”, so now I wonder if something special is going on with it, which either caused the comments to be disabled or perhaps the disabling of comments (for whatever reason? I didn’t see anything happening inside that looked remotely contentious, but all the posts submitted around that time seem to have had that happen to them? perhaps a new mod learning the ropes, maybe even fat-fingering buttons unintentionally?) caused federation issues specifically with that content. On the other hand, many other examples abound, even if this were not a great example on its own.
But it is a quite striking one: the original post still has 185 upvotes, when viewed from Lemmy.World it has a handful more, but when viewed from discuss.online it is still far behind at merely 97 upvotes, and when viewed from StarTrek.website still has just the default 1 upvote… and zero comments visible underneath it. At this point if looks like if the delay lasts for one more day the situation will become permanent for that post.
And I’ve already decided to abandon the Startrek.website instance due to constant issues like this - especially if a post takes a day or two to federate, or almost worse it does federate but then the comments cannot make it back from LW to the STW instance for me to be able to respond to them (as has happened before), then on that instance I am essentially out of the loop wrt my own postings on Lemmy, unable to respond to what I can’t even see, at least there on STW where I have a login account, although I can see it on LW where I do not.
I really hope that the upcoming 0.19.6-7 helps make usage of smaller instances more viable as we continue this experiment in federated Lemmy.:-)
I couldn’t tell you the reason for this, but several posts in [email protected] have been locked by a moderator: https://lemmy.world/modlog/959443
as far as i know, locking a post does not affect voting, only prevents new comments from being federated.
the other example you mentioned, i assume that you’re referring to the inconsistency on lemmyverse.net? i haven’t looked at how that application works, but it’s unlikely to be working with activitypub/federation, instead it’s most likely just connecting to various different instances and using their APIs. i’ve also left a comment over there about that.
i already explained before why these posts don’t see votes on startrek.website - there is no local subscriber on that instance. once at least one person from that instance subscribes to the community it’ll start seeing updates, which includes votes. there has also been a comment by one of the startrek.website admins about the federation issues caused by them accidentally blocking certain traffic from other instances here.
for discuss.online, there does not seem to have been a longer federation delay according to this dashboard, only about 1.5h delay at some point that was recovered from fairly quickly. it is also very possible that the first subscriber to the community on discuss.online only subscribed after the post was created, as the more recent posts seem to be doing just fine with their vote counts when comparing discuss.online and lemmy.world numbers. looking at our database, i can see the first subscriber to that community from discuss.online joined about 5 hours after the post was posted, which would easily explain the partial votes.
I might have a misunderstanding about how federation of posts works: when there are no subscribers to a remote community on a particular instance, I thought in the past that I have seen not only missing votes and comments, but the entire post itself would be absent as well? e.g. if the last subscriber from an instance leaves a community, then it will retain the archived posts but not bring in any new ones, until someone subscribes again (I had this situation with the “just post” community, from discuss.online, iirc). So then I do not understand why I am seeing posts at all for that community on StarTrek.Website, if nobody from there was subscribed to it? On the other hand, I do begin to see what you are saying: every single post there has zero upvotes and zero comments.
Thank you for explaining all of this btw, and for weighing in on that Lemmyverse post as well:-).
See my comment.
Yeah it looks fixed now. I went to [email protected] and counted 10 posts from the last 3 hours, then viewed the community from SJW and it too has 10 posts from the last 3 hours.
So… are you enjoying the cats flooding your feed now?:-D Or if you aren’t seeing them, it must be for some other reason now.
I’m not sure. Maybe the volume of cats has just been exaggerated? Sorting all+active only shows 1 cat in the first 80 posts that I looked at.
Ah that seems to explain it. Active prioritizes many comments iirc, but on most of these posts I see few if any comments. Using Hot though, they can dominate a feed due to being highly upvoted.
Ah, yeah that seems to be it. I switched to all+hot and I’m now seeing a lot of cats. I wonder what other content I’m missing out on by sorting by “active”? When I sort by “all”, I’m trying to get the posts that are trending the most on Lemmy. Maybe active isn’t the best way to do this? Interestingly, the posts that I see when I sort by “hot” don’t have as many upvotes as those when I sort by “active” so idk.
Hot also shows newer content, as these cat posts tend to be, while Active will allow older content that is still being discussed. You should notice a lot more turnover of your feed with Hot than with Active. You can change your default sort in your Settings so you don’t have to keep switching it every time.
Yeah, I have my default sort set to “Subscribed” + “Scaled” (I want to primarily see the stuff that I follow, and I don’t want the small communities to be drowned out by the large ones). When I want to see what’s trending (big topics that are being talked about a lot) right now, I’ll switch to “All” + “Active”. It seems like “Hot” favors boosting trending groups rather than trending single posts.
Yeah I believe that is the purpose of Hot to begin with, though it’s so great that we have so many choices to pick from:-).