Hey everyone. If you want to post links or discuss the Reddit blackout, please localize it to this thread in order to keep things tidy!
Hey everyone. If you want to post links or discuss the Reddit blackout, please localize it to this thread in order to keep things tidy!
I’d love to know what it is about subreddits going private that caused issues.
Maybe some overload caused by a process having to dig deeper to find best/top posts?
apparently that’s exactly the case.
That is an interesting aspect no engineer could have foreseen!
You’d be surprised how much critical infrastructure was implemented through trial and error and has just been left like that for years…
Anything less than 99% of infrastructure working that way would be surprising. Everything is held together with scotch tape and scotch whisky.
I’ll be sure to repeat that last line to my fellow team members :D
I like this idea. I imagine that with the top subs being dark the automated top posts that get scrounged up may be too terrifying for the front page and they hit the panic button while they scramble to curate through the absolute worst filth they’ve ever seen.
“It’s merely coincidence. But starting Wednesday, our servers will be more robust and you can browse the site using our official app.” - Spez, while sniffing a decanter of human shit
God we need indefinite blackouts.
It’s entirely possible that they’ve made some assumptions about what a “normal” level of traffic looks like when writing code for their backend, which has caused some things to break when that has changed.
Not our fault if their code is shit.
How is that an example of bad code?
Honestly, it’s probably not - if I’m actually right this is likely an issue that Reddit’s engineers never predicted would happen so never planned for it. I was being hyperbolic.
It’s not reactive. A proper reactive system can handle fluctuations in usage patterns more robustly.
I’m having a hard time believing the claim that Reddit’s code isn’t reactive.
Wouldn’t be surprised if it’s just a gigantic mess of nested if-else statements.
Gotos all the way down
Maybe, but this was a huge increase in usage. Reddit never expected to deal with anywhere near thousands of subs going private simultaneously.
Tildes’ dev Deimos used to work for Reddit and had this guess https://tildes.net/~tech/163e/reddit_appears_to_be_down_during_blackout_day_1#comment-87v1.
The servers run on the tears of bitter whiny CEOs.
They’re lying. Fish swim, birds fly, sun shines, Reddit lies.
Probably a drop in usage flagged some internal test
deleted by creator
Probably because they had to change attributes (private or read-only) of millions of posts.
Another comment suggested that whatever cluster management automation they have in place may be configured to suspend services during low load to reduce costs. But if usage drops unexpectedly low then you enter a state that has never been tested. It may be that some services are (incorrectly) suspended across the board and causing failures.