I find it interesting that so many subs going private caused issues with the entire site. I’m not a programmer/tech person, but I don’t really understand why that would happen. Did it just create a bandwidth bottle neck or something as users flooded non-blacked out subs?
I’m a backend server programmer, and although I have never worked on resdit I can imagine the conversation a few years back:
Dev A: How do we implement private? Should each post have a private flag or do we join on the subreddit to get it?
Dev B: that join might be heavy for the front page since we might have several different subreddits, a flag in each post is faster.
Dev A: But that means that whenever a subreddit goes private we need to flag all of the posts there
Dev B: yeah, but that doesn’t happen often, the front page gets accessed millions of times, most subreddits never change status, especially larger ones.
Dev A: I guess you’re right, even a large subreddit will be done in minutes, what are the odds of several large subreddits going private at the same time?
The funniest thing about this is that means that the subreddits in question could keep flipping the switch back and forth and with each change cause outages.
We don’t know what switching a sub to private, will trigger on their backend. Maybe there is some internal shifting of data somewhere, or something similar to this.
Doing this, on a lot of the big subs, could lead to large-scale performance issues.
Interesting/fascinating. I have no idea how development works, but it’s interesting to me to see the impact. Someone else mentioned that it could be resource allocation locking up too, which makes sense when I think about it more. A subreddit with subscribers in the millions would need more resources allocated to it due to the increased demand, so if they’re suddenly unavailable I can kind of understand why/how that would creat issues. All very neat.
I’m a backend developer and have to deal with this nonsense at my job. There are lots of options, each with tradeoffs, such as:
rely on checking at the subreddit level - this means that every time you access a page, you need to check whether the subreddit is private, which means slower page loads
include the flag on every post - flipping between private and public can be very expensive on the server
embed some info in the URL - breaks links when switching back and forth and is also expensive, but you can cut down on some checks (e.g. no database hit for most checks; i.e. if it’s private and you aren’t logged in, you cannot access it)
And if you go for one of the two last options, you can reduce the hit when switching by being eventually consistent, meaning you don’t change everything all at once but instead set the flag or change the URL in batches over a period of time. That takes effort to get right, and there’s going to be inconsistency until everything is done.
So there are multiple ways to do it, and each way has a big chance at devs messing something up when they don’t consider something like a bunch of subreddits going private all at once.
Or it could just be a run of the mill DDOS because some redditors wanted to troll Reddit.
Maybe they optimised the site to have certain servers allocated to the bigger subs, and with them all going dark it screwed with that balance or something?
Maybe an issue on the back end skipping over all of the new private subs? I assume that many of the big subs had popular posts just before they went private, so the server has to spend more time skipping over private posts when rendering popular?
The theory is that the way Reddit caches its subreddit content for the front page didn’t jive with a bajillion subs being gone all at once when America woke up this morning.
I find it interesting that so many subs going private caused issues with the entire site. I’m not a programmer/tech person, but I don’t really understand why that would happen. Did it just create a bandwidth bottle neck or something as users flooded non-blacked out subs?
I’m a backend server programmer, and although I have never worked on resdit I can imagine the conversation a few years back:
Dev A: How do we implement private? Should each post have a private flag or do we join on the subreddit to get it?
Dev B: that join might be heavy for the front page since we might have several different subreddits, a flag in each post is faster.
Dev A: But that means that whenever a subreddit goes private we need to flag all of the posts there
Dev B: yeah, but that doesn’t happen often, the front page gets accessed millions of times, most subreddits never change status, especially larger ones.
Dev A: I guess you’re right, even a large subreddit will be done in minutes, what are the odds of several large subreddits going private at the same time?
The funniest thing about this is that means that the subreddits in question could keep flipping the switch back and forth and with each change cause outages.
back and forth, forever…
That’s r/talesfromtechsupport material lol.
We don’t know what switching a sub to private, will trigger on their backend. Maybe there is some internal shifting of data somewhere, or something similar to this. Doing this, on a lot of the big subs, could lead to large-scale performance issues.
Interesting/fascinating. I have no idea how development works, but it’s interesting to me to see the impact. Someone else mentioned that it could be resource allocation locking up too, which makes sense when I think about it more. A subreddit with subscribers in the millions would need more resources allocated to it due to the increased demand, so if they’re suddenly unavailable I can kind of understand why/how that would creat issues. All very neat.
I’m a backend developer and have to deal with this nonsense at my job. There are lots of options, each with tradeoffs, such as:
And if you go for one of the two last options, you can reduce the hit when switching by being eventually consistent, meaning you don’t change everything all at once but instead set the flag or change the URL in batches over a period of time. That takes effort to get right, and there’s going to be inconsistency until everything is done.
So there are multiple ways to do it, and each way has a big chance at devs messing something up when they don’t consider something like a bunch of subreddits going private all at once.
Or it could just be a run of the mill DDOS because some redditors wanted to troll Reddit.
Nah I think they just knew there’d be a drop in traffic and scheduled some maintenance, lol
If you read the article there’s a quote that distinctly mentions unexpected issues not planned maintenance.
Yes. Let us trust what Reddit staff say about it. That feels logical.
I mean in this case, you can decide between some theory based on no information at all but unaffiliated, or based on all information but biased.
There’s still a good chance that the downtime is unexpected, downtime in general is never good for a site that earns money by people being on it.
So I think the “unexpected” case is still more likely than it being intentional.
Maybe they optimised the site to have certain servers allocated to the bigger subs, and with them all going dark it screwed with that balance or something?
I would guess it’s the influx of traffic from all the news coverage, combined with people repeatedly refreshing thinking the site is “broken.”
I’d imagine that would be offset by all the people who are redditing much less because most of the subs they frequent are private?
I would imagine this also, honestly.
Maybe an issue on the back end skipping over all of the new private subs? I assume that many of the big subs had popular posts just before they went private, so the server has to spend more time skipping over private posts when rendering popular?
deleted by creator
The theory is that the way Reddit caches its subreddit content for the front page didn’t jive with a bajillion subs being gone all at once when America woke up this morning.