Pirate video releases generate heaps of traffic, and they won't be on the news. Earthquakes (mild ones, not the "cities collapse" kind) cause people to immediately go online and check on their friends. So do missile strikes, but those tend to generate extremely popular videos as well. During the holy month of Ramadan, you can see a rapid and deep traffic drop in Muslim countries right at their local sunset, when people have iftar, breaking their daily fast. A country in North Africa shuts down their internet access completely during their school exams. Popular web infrastructure sometimes reroutes your requests to distant data centers, leading to request latency exploding together with your queue lengths. In summer, the morning user activity peaks later than in autumn because schools are on their summer breaks.
Here's a fun one I remember seeing a little video on: the UK power authority has to anticipate commercial breaks in major broadcasts like the World Cup because everyone turning on their electric kettle spikes the grid.
The employee shaming on here has scared away all the people with interesting war stories from big companies. Anyone active on HN nowadays probably hasn't worked on anything significant.
at the end 1999, my employer hosted what I guess was half NYE Party and half Incident Response. We were optimistic that we wouldn't have issues but figured it'd be wise to have a sufficient crew of designated drivers, so to speak.
Our monitoring system included a mercator projection on some big screens, with colored that showed where our gear was and whose color indicated whether it was down, being hammered, etc.
We had no equipment in UTC+14, so there was an hour of waiting between that zone reaching y2k and when we'd maybe see some action. The control room was mobbed since we also had some televisions running cable news there and everyone wanted to see it.
Despite the anxiety, the gear was looking fine everywhere, until about 5 minutes before midnight in UTC+13, when our locations in New Zealand started to turn red. A hush fell over the room as the operators attempted to open connections to those boxes. The connection attempts appeared to hang, and the party got somber. But after the eternity of a few seconds, we were in. The machines were alive but being hammered; the alerts just indicated the excessive traffic caused by mass numbers of users refreshing NZ based websites as the date change approached, to see whether they'd fallen off the net or not. Breathing resumed.
After 15 minutes or so had passed, the traffic fell off, and by the time y2k reached us in UTC+5 we'd been in full festive mode for hours. I still can't look at New Zealand on a map without seeing it covered in angry red circles, though.
Big street party here in UK on Dec 31 1999. Myself and a few neighbours who thought about these things relaxed a bit after 9pm (Midnight Moscow time) and started partying seriously.
Having been at a finance firm trying to implement SRE without SRO, and without staffing either, I do find the story a bit entertaining.
That is - we were mandated to implement all the SRE tooling on top of our apps, with like 1 guy owning the SRE infra, but with no one looking at the charts proactively.
The idea was sold to dev teams that it was something like a black box recorder for the on-call rotation to check first when you got called in the middle of the night/weekend.
In reality it turned into CTO reporting metrics of whether SLOs were being met, lol.
Constantly feel like other industries read like 1/4 of a book about how FAANG does stuff and then adopts the laziest worst implementation of the part they skimmed.
> Constantly feel like other industries read like 1/4 of a book about how FAANG does stuff and then adopts the laziest worst implementation of the part they skimmed.
I think this is less wrong than you think it is. In the P&C insurance industry, I saw a handful of initiatives that seemed like they were started because a senior manager read about something that sounded cool and high-tech in an industry publication, and/or heard it in a sales pitch from a Microsoft rep, without actually checking to see if it was feasible or even useful.
One of my clients back at a large financial software and data provider, was a southern market making firm.
They kept complaining about "the whole desk being slow" and we spent weeks trying to figure out what was wrong with our software.
Eventually we figured out the whole firm was nuts about golf, and they'd all stream golf tournament to their workstations at the same time and saturate their ISDN or whatever they had.
10+ years ago the networking team at a big European bank I worked for, observed that the biggest usage of network bandwidth on the trading floor desktop network was not the Bloomberg Terminal, market data, or trading application traffic.. but Youtube :-)
I was working in a Telco that owns a big backbone network in south America.
Our director was very sensitive to traffic changes. Every week or two there was a big meeting at her office to explain what happened when some traffic went down.
That day she was really mad at us. There afternoon before there was 20% less traffic in the backbone and nobody knew where it went.
I was on call that week for the management systems and I was questioned about why the operators didn't have received any alarm.
Turns out Spain was playing soccer against some Latin American team. Spain was our biggest customer. I don't remember if the other team was Argentina, Chile, or Brazil... but it was our 2nd biggest market.
People just decided to watch TV instead of web browsing (that was before mobile phones had an affordable internet connection).
Funny enough eMule traffic spiked during the match. That was the reason I could justify there was no alert in the system for down interfaces.
Funny enough eMule traffic spiked during the match.
If your bandwidth is limited enough such that background downloading would slow down your browsing significantly, then it would make perfect sense to let the downloads saturate the connection while you watched TV.
Lol. During "high velocity events" at Amazon, we would have a war room with all the engineers and a couple old hands directing responses.
Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days.
These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.
But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minutes before a sale kicked off.)
To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."
We often got problem reports via Twitter before they showed up on our dashboards.
This footnote is almost more interesting than the post itself:
> There's a deep beauty to the thought that untold millions
of people using the app randomly, but oh so slightly
habitually, aggregated together, makes such a predictable pattern
That 'beauty' can be extremely scary from another angle. It's evidence of what makes Facebook, and other social media, such powerful mass manipulation tools.
One service I used to work on processed business-to-customer and business-to-business emails almost exclusively, so you could see a recurring weekly pattern, bumps throughout the day when the US east coast, US west coast, Asia, and Europe woke up, and spikes at the top of the hour from automation. So one day we got a new boss, and he called me into his office in a panic so I could explain why his chart of the traffic kept going up and down. Took about four tries, but I think he eventually got it.
I also recall seeing someone push a global update to that system that was packaged wrong, and watching the graph gradually drop and flatline as 20,000 VM hosts across the planet stopped taking traffic. That had its own subtle beauty, in no way diminished by the fact I was just a bystander and couldn't get in trouble for it.
I don't see the fright. So many things do this - electricity for example - you turn the lights on and off and run the laundry machine whenever it pleases you, but on the scale of millions of people the power companies predict the daily usage patterns very well.
Imagine if the power company could see everything you do at home, and control your lights, laundry machine and other electronics to influence your behavior. They don't have that kind of power, but social networks do.
Another part of my career, I worked on a team that ran an automated content detection service (like YouTube ContentId.)
The database holding signatures for known music samples was sharded by artist. Not as crazy as you might think. You get a trial sample and you send it to all the shards which chunk on it in parallel then you just wait for the servers hosting each shard to respond.
But then Prince died.
The queue for the server that owned the shard Prince was in backed up.
Then our proxy that distributed trials to each shard backed up.
Then our regular reverse proxy backed up.
Then our anemic load balancer fell over.
What we learned for about the bazillioneth time is keep a backup of each shard handy so you can add it into the rotation and pay an intern to watch Facebook looking for news about recently deceased musicians.
Maybe it’s just the trained operator in me but the whole time he’s describing the depressed metrics and blip I’m screaming in my head: it’s exogenous! Check the news! He finally gets there.
For us (at normal company scale) I would actually go look at recent activity, new customers, new workloads we are running, but I guess at a Facebook scale it’s a lot harder to do that.
One interesting aspect of the story is that the author was a junior at the time (almost brand new to the job, if I read it correctly), and it was one of the more experienced operators who realized it was exogenous.
in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!
Author here, and that's exactly right! I think it was in my second or third month on the job.
And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.
Early in my career, in the late 90s, I was at Cox Interactive Media on the team responsible for the web farm which hosted all of Cox Enterprises news sites: newspapers, radio stations, TV stations. This was before the days of SREs and SROs and dev-ops. We were just system admins and programmers and some of us could do both.
The web farm was about two-dozen Sun Ultra 2s connected on a FDDI loop (we eventually upgraded to GigE), with content on NetApps (4GB drives). A couple Sun E450s. Apache[1] on the Ultra 2s. Apache + mod_perl on the E450s. Hosted at a Global Center DC in Sunnyvale (same DC that early Yahoo was hosted in).
Monitoring with MRTG[2].
It's 1998. The Ken Starr report drops. Now, we knew it was coming, and we did our best to be prepared, but this is the late 90s. There was only so much load testing we could do and didn't really know how much traffic it would drive. We kept the site up, but it meant, as I recall, a lot of fine tuning of the mod_perl box and disabling interactive parts[3] of the site.
I really wish I still had some of the traffic graphs.
[3] The forums. OMG forum software was so terrible. I think we eventually wrote our own after nothing based on Netscape Application Server or that we found open source worked well at all.
That doesn't ring any bells. It's been years. I think it was on Borregas Ave and it may have been owned by company called ISI at the time. Equinix has a DC in the approximate area now, not sure if that's the same building and the address changed or my memory is off.
A lot of time it would end up being DNS or routing somewhere outside of our control.
Sometimes a disk would get close to filling up and a cron would clear it just in time so you would see decreasing performance then it would clear.
Other times we would have a customer that did something that we just did not expect with the system and it would cause SQL queries to slow down in ways we could not imagine. And it was transient so those would be hard to find or explain.
Or we would hit a limit in the load balancer (haproxy) in really odd ways. Too much traffic on the frontend or not enough capacity in the backend. And many other various ways of things not working. Haproxy was and still is amazing software. Really almost magical.
This reminds me of this YouTube video: https://youtu.be/slDAvewWfrA. There's a monitoring room to make sure people are ready to switch backups and reroute in case of some kind of grid failure, but also in very specific scenarios.
Britain being filled with Brits, what would happen is that once the show was over, half the nation got up from the couch to turn on the kettle for a cup of tea. Those electric kettles are quite demanding, especially if millions of them turn on at the same time.
So every time there was a major event, dedicated people are monitoring the grid frequency, power plants on standby and foreign contracts at the ready when needed, just to increase capacity at the right time. You can't just schedule this stuff, because if a football match runs over its allotted time, you may suddenly add power to the grid without any load, causing all kinds of problems like the grid frequency increasing and making digital clocks run ahead. There is a temporary demand of gigawatts of power that rises within five minutes and lasts until the kettles are done.
The YouTube video provides an example of 600MW of power being requested from France... for the end of an episode of Eastenders.
I always knew the British like their tea and that there's some kind of planning going on for events like street lights turning on, but the combination of the two is a great example of complex behaviour that's easy to overlook.
> Britain being filled with Brits, what would happen is that once the show was over, half the nation got up from the couch to turn on the kettle for a cup of tea. Those electric kettles are quite demanding, especially if millions of them turn on at the same time.
Particularly British kettles, which pull 13 amps at 240v = over 3000 watts. A lot more than (standard) American outlets can deliver.
For a while there was a theory going around that this is why Americans typically do not have electric kettles, because they would boil much slower in the USA, but the real answer is that Americans just don't drink much tea.
> For a while there was a theory going around that this is why Americans typically do not have electric kettles, because they would boil much slower in the USA, but the real answer is that Americans just don't drink much tea.
I would drink tea and use an electric kettle if it would boil faster...
I use an electric kettle to boil water. I got it a while back from Costco. I find it to be faster than our earlier method - using the microwave to boil water.
If fast hot water is all that's holding you back, and hot water on standby is an acceptable alternative to freshly boiled, I can highly recommend Zojirushi's line of products. Mine [1] keeps water at my desired temperature in a vacuum insulated canister that uses barely any continuous power. Any time I want coffee/tea/etc., hot water is instantly available. Recovery time after a refill is under 10 minutes on a USA-standard 15A wall socket.
Zojirushi has been in business since 1918 and (in my understanding) is a household name in Japan. Their more expensive (to be blunt) lines are made entirely Japan and replacement parts (like rubber seals, which can wear out over time) are available for purchase through their site. I fully expect mine to last 15-20 years of continuous duty.
Edit for additional info: the Zojirushi replaced my Sunbeam Hot Shot single cup boiler that had an unfortunate dry-run and became unusable. They were unavailable in late '21, but appear to be back on the market now. They boil a cup of water in under a minute. If the Zojirushi didn't exist (or I hadn't been willing to bite the bullet on the price) I would buy another in a heartbeat.
Value is what is left when price has been forgotten. Both of these utensils have been high-value for me, but the Sunbeam has a lower initial cost of ownership.
I can verify the parts are easy to get. We got the boiler, sans power cord, for 5 usd at a yard sale. we ordered a replacement cord and there wre no snags anD as I recall the price was good. That was 5 years ago and it still works well.
Many American homes receive 240V, with the right wiring and plugs you can use a 240V electric kettle in the USA just fine. You may need to import a European kettle and rewire it to a standard American 240V socket (or use one of those travel adaptors but that may be dangerous). These sockets are already used for high power devices like car chargers, AC, and other heavy electrical equipment. Your kitchen may even have one intended for induction cooking that you can use.
Don't be foolish like some and have 240V wired to a standard plug, though! NEMA 6-15 and 6-20 exist just so you don't mistake your beefed up socket for a normal American socket and fry your devices/zap yourself or others.
It amazes me how most of the world is on 240V I'm in Canada and we are like the USA 120V 20A circuits.
Today I made tea and I used a standalone electric kettle which is odd to most of us here in Canada and the US. Most times I would have used the microwave to boil the water. But I used the kettle 120V x 12.5A or 1500W and popped the breaker since the dishwasher was operating and on the same circuit.
In Canada we drink a lot of tea; Red Rose, King Cole and especially south-eastern Canada The Maritimes and Atlantic Canada.
Most US homes actually do receive 240v, it's just done in an unusual biphase configuration with two "hot" phases 180 degrees apart. If you connect a device across the phases it will receive the full 240 volts, which is often done for cookers, clothes dryers, and EV chargers.
In theory you could sell a 240 volt electric kettle but your kitchen would have to have a 240 volt outlet, probably a NEMA 6-15. They're not common.
From watching a bit of YouTube, I think Canada’s kitchens have more 220 outlets than American kitchen. They even seemed to have dual outlets with each outlet on a different phase, so all the ingredients are right next to each other to get an outlet un between phases.
I'm in Canada in The Maritimes in PEI my electric stove uses a big honking plug for 240V/220V that plugs into the wall. Not wired directly but an actual plug.
I installed a 240V NEMA 6-15 socket in my kitchen in California. It's similar to a regular US/Canada plug but the blades are sideways [0]. It's allowed by the code, but not very common. Then I took a European electric kettle and put a matching plug on it, and I can boil a teapot worth of water in 60 seconds!
> Most US homes actually do receive 240v, it's just done in an unusual biphase configuration with two "hot" phases 180 degrees apart. If you connect a device across the phases it will receive the full 240 volts, which is often done for cookers, clothes dryers, and EV chargers.
It's technically a single phase with two hot legs and a center tap for neutral, also called "split phase". From the neutral to either leg is 120V, from leg to leg is 240V.
It would technically be possible to have a reverse-Y extension cord that plugged into two outlets and gave you a 240V circuit. The two outlets would have to be on different legs, of course. This is basically what a 240V breaker is doing. I've contemplating wiring up one of these for home and buying a British tea kettle, but the aesthetics got my plan vetoed.
> problems like the grid frequency increasing and making digital clocks run ahead
Ahem: the ones driven by the grid frequency are analogue clocks.
This function was so important that if load changes caused the frequency to fluctuate the power companies will compensate as soon as they can to make sure the clocks can catch up. I said "was" because I have no idea (and does anyone?) how many such clocks are still in service. But it's a legacy function and, like POTS, such features are still maintained. Because you never know who'll need it.
Digital clocks too. Plenty of clocks are still being manufactured that rely on the grid frequency, although as quartz crystals have become cheaper, some companies have started using those instead of mains-referenced clocks. Yet the old 70s PMOS mains-referenced clock ICs --- one of the first mass-produced LSI ICs for consumer electronics --- are apparently still in production and widespread use.
It's been my experience that, in the long term, mains-referenced clocks are extremely accurate, while local quartz ones seem to drift significantly. Clock also tend to last long enough that these are relatively common around here: https://en.wikipedia.org/wiki/Telechron
yes. the US grid operators also seem do go through a lot of effort to make sure there are exactly 5,184,000 cycles per 24-hour period, specifically because it helps those clocks keep time.
Not just analogue clocks! In Europe we had our digital clocks run late for quite a while after a power crisis resulting from some political energy dispute in Serbia/Kosovo dropped the frequency for a while. This caused all manner of clocks to run late by five minutes or more. They even intentionally cranked up the frequency for a while to bring those clocks back in sync.
Alarm clocks, microwave clocks, all manner of time indicators without sync mechanism built in were affected because relying on Europe's near perfect 50 Hz grid frequency is pretty damn cheap.
Can confirm to some extent - I started out in data analytics for call centres, and important episodes of Eastenders et al would completely bork the demand predictions by causing exactly the pattern of depressed traffic followed by a massive surge described in the blog post. Some of this was less about tea and more about just having four television channels.
At a place I worked that sold a retail product online we had a weird rate problem that we never were able to figure out.
If you graphed the visit rate to our shopping cart page it would follow a curve that was sort of like a sine wave (well, a sine wave plus a constant because you can't have negative cart traffic) with a 24 hour period. I'm just going to call it a sine wave.
The amplitude varied seasonally, and was different on weekends and weekdays, but generally any given day was a sine wave.
If you graphed order rate it also followed a sine curve too, in phase with the visit curve and generally with a fixed ratio between them. So if the cart was getting N visits per minute, we'd be getting fN order per minute for some number 0 < f < 1, and f was pretty much constant.
This continued as the business grew, so the overall amplitude of the visit curve increased. But at some point the order rate seemed to hit a wall.
What we would then see is that as the visit rate was heading toward its peak for the day when it got to a particular fraction of that peak, say 80% of the peak, the order rate would stop tracking the visit rate and would become flat. It would stay flat as the visits peaked and declined until the visit rate was back down to 80% of its peak and then orders would resume tracking visits.
So far nothing too weird about this. What was weird is that when we'd have a day of especially high cart traffic such as when something in the news caused a big increase in demand for a few days for the kinds of things we sold, and we'd get 3 or 4 (or sometimes even 20 or 30) times the normal cart visit rate throughout the day--the plateau on the order rate would still start when the visit rate hit 80% of the visit rate peak for that day and go away when the visit rate fell back below 80% of peak.
The order processing was handled by a different server than the shopping cart. They did use the same database for product and customer information, but we could not find any signs that the database was a bottleneck.
This just made no sense. Suppose we'd been seeing on normal days that visits would peak at 1000 per minute, and orders were 10% of visits. Then we'd see visits grow from 0 to 800 and orders grow from 0 to 80. Then orders plateaued at 80 while visits continued on to 1000 and back to 800. Then as orders fell from 800 to 0, orders fell in step from 80 to 0.
Now suppose the next day we had 10 times the activity due to some news item. So the next day visits rise from 0 to 10000. The order system would now not plateau until the visit rate got to 8000 and the order rate got to 800.
And then when the increased demand was over in a few days and we were back to 1000 visits peak the order system would again plateau at 80.
So whatever was causing the plateau was somehow tied to the visit rate, making the plateau depend on what the peak visits would be for that day.
But how on a given day when the visits would first rise to 800 and the orders to 80...how did it "know" if the day was going to be a 1000 visit peak day or a 10000 visit day so it could "know" if it was time to plateau?
We never figured this out. Reasoning about our systems and their dependencies I could not figure out any way in theory that this could be happening nor could I find anything no matter how much logging and tracking I added all over the place.
Everything I could think of that could even remotely cause a plateau would have caused a fixed plateau, not some weird plateau that's level gets set daily based on what the future peak visit rate is going to be for that day.
If it was really close to the shape on an actual sine wave, then the derivative at any earlier time encodes information about the future peak. Any way some system could be bottlenecked by the rate of change of the visit rate (i.e. the visit acceleration)?
might be too long ago, but did you check the response codes out of your system? I had a similarish issue that turned out that we were sending out 429s from a different system
It's funny - the very first thing I thought while reading this was "check the news". That's probably because I've been involved in markets and when you see this sort of behavior unfold in market price and volume, the first thing you do is check the news.
Yeah! By the end of my time on the team, checking the news when I saw a rapid change in traffic was second nature. And later on in my career becoming interested in markets, even more so. Forex in particular - markets with trillions in daily liquidity don't take huge leaps unless something huge happened.
Google didn't have SRO. The closest thing was Traffic Team's oncall; Which was stressful, for sure, and hired some dedicated alcoholics, but it wasn't anything like Facebooks (I knew people on both teams during my time at each). Google had a constant stream of alerts that someone was aware of and handling. Facebook had a constant stream of fire that someone was beating back the flames of.
SRO is madness. SRO is admitting that you're enough underwater and there's enough unexpected critical events to dedicate a team to burn out on it. There was a lot of spend in overcapacity and failover automation testing that kept it that way, and explicit mandates by a C-Suite and VP to keep keeping it that way. More importantly, though, it was baked into the architecture.
Everyone's talking this week about how Twitter will go up in flames in the next few days; I don't think so. I think they're closer to Google in that respect. There's a lot of automation that will keep things alive and a lot of redundancy that will handle the immediate problems of having a lot of the team leave. I actually think the company can continue to run indefinitely if they just follow the operations manuals and replace failed drives and machines. But I think there will be a major outage when they try to launch a new feature and fail.
Author here - a few months after the time of this story, we reoriented around a single purpose for the team, "Automate ourselves out of existence". It took a few months to remove all of the dependencies, tag all the alarms, train up all the team-specific oncalls, but we managed to do it, and I had the great honor of taking the very last SRO oncall on March 31, 2014 :)
Once when I was working in a big bank in London during World Cup 2010, they told us all to stream the England games on BBC Online since it would be a good test of their network capacity.
They knew not much work was going to get done either way.
I've got a good one. I was working a "tip of the spear" job in threat intelligence (I suppose I was always on call). One of the things we monitored was DNS.
It was always noisy. If it's not the miscreants, its the normal stupidity (DNS is so amazingly survivable of misconfigurations, some of them become folk wisdom; like disabling TCP/53, but I digress).
Othodox Easter Sunday morning. Really really early in the morning, like I can't remember if it was light out yet (US Pacific coast time). The SOC calls me. Nothing is happening.
Nothing is happening. There is no noise. So they've been looking at this, and everything is working. So they called me. At dark-thirty on Sunday morning.
I confirm everything appears to be working correctly. I repeat diagnostics, do ad hoc stress tests, everything. About two hours later the normal crap starts coming back.
We have to conclude that the miscreants, as well as the sysadmins, had taken the morning off.
Just to note, it's the finals games that attract so much attention. The early games don't drive quite so much attention. The first hand was yesterday but the one to watch out for us the final one, which is Dec 18th at 1500 UTC
67 comments
[ 2.9 ms ] story [ 143 ms ] threadI bet every seasoned SRE has a few to add.
Our monitoring system included a mercator projection on some big screens, with colored that showed where our gear was and whose color indicated whether it was down, being hammered, etc.
We had no equipment in UTC+14, so there was an hour of waiting between that zone reaching y2k and when we'd maybe see some action. The control room was mobbed since we also had some televisions running cable news there and everyone wanted to see it.
Despite the anxiety, the gear was looking fine everywhere, until about 5 minutes before midnight in UTC+13, when our locations in New Zealand started to turn red. A hush fell over the room as the operators attempted to open connections to those boxes. The connection attempts appeared to hang, and the party got somber. But after the eternity of a few seconds, we were in. The machines were alive but being hammered; the alerts just indicated the excessive traffic caused by mass numbers of users refreshing NZ based websites as the date change approached, to see whether they'd fallen off the net or not. Breathing resumed.
After 15 minutes or so had passed, the traffic fell off, and by the time y2k reached us in UTC+5 we'd been in full festive mode for hours. I still can't look at New Zealand on a map without seeing it covered in angry red circles, though.
That is - we were mandated to implement all the SRE tooling on top of our apps, with like 1 guy owning the SRE infra, but with no one looking at the charts proactively.
The idea was sold to dev teams that it was something like a black box recorder for the on-call rotation to check first when you got called in the middle of the night/weekend.
In reality it turned into CTO reporting metrics of whether SLOs were being met, lol.
Constantly feel like other industries read like 1/4 of a book about how FAANG does stuff and then adopts the laziest worst implementation of the part they skimmed.
I think this is less wrong than you think it is. In the P&C insurance industry, I saw a handful of initiatives that seemed like they were started because a senior manager read about something that sounded cool and high-tech in an industry publication, and/or heard it in a sales pitch from a Microsoft rep, without actually checking to see if it was feasible or even useful.
They kept complaining about "the whole desk being slow" and we spent weeks trying to figure out what was wrong with our software.
Eventually we figured out the whole firm was nuts about golf, and they'd all stream golf tournament to their workstations at the same time and saturate their ISDN or whatever they had.
Maybe it would be useful to have a local company radio streaming service, haha.
Our director was very sensitive to traffic changes. Every week or two there was a big meeting at her office to explain what happened when some traffic went down.
That day she was really mad at us. There afternoon before there was 20% less traffic in the backbone and nobody knew where it went.
I was on call that week for the management systems and I was questioned about why the operators didn't have received any alarm.
Turns out Spain was playing soccer against some Latin American team. Spain was our biggest customer. I don't remember if the other team was Argentina, Chile, or Brazil... but it was our 2nd biggest market.
People just decided to watch TV instead of web browsing (that was before mobile phones had an affordable internet connection).
Funny enough eMule traffic spiked during the match. That was the reason I could justify there was no alert in the system for down interfaces.
If your bandwidth is limited enough such that background downloading would slow down your browsing significantly, then it would make perfect sense to let the downloads saturate the connection while you watched TV.
Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days. These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.
But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minutes before a sale kicked off.)
To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."
We often got problem reports via Twitter before they showed up on our dashboards.
I also recall seeing someone push a global update to that system that was packaged wrong, and watching the graph gradually drop and flatline as 20,000 VM hosts across the planet stopped taking traffic. That had its own subtle beauty, in no way diminished by the fact I was just a bystander and couldn't get in trouble for it.
The database holding signatures for known music samples was sharded by artist. Not as crazy as you might think. You get a trial sample and you send it to all the shards which chunk on it in parallel then you just wait for the servers hosting each shard to respond.
But then Prince died.
The queue for the server that owned the shard Prince was in backed up.
Then our proxy that distributed trials to each shard backed up.
Then our regular reverse proxy backed up.
Then our anemic load balancer fell over.
What we learned for about the bazillioneth time is keep a backup of each shard handy so you can add it into the rotation and pay an intern to watch Facebook looking for news about recently deceased musicians.
For us (at normal company scale) I would actually go look at recent activity, new customers, new workloads we are running, but I guess at a Facebook scale it’s a lot harder to do that.
in addition to being a good story about the site reliability, it's also a great lesson in the value of collaboration, mentorship, and having senior people around whom you can ask for help!
And I really agree with your second point too! Without that in-person transfer of tribal knowledge, it would have taken me ages to learn the ropes well enough to take such a high-pressure high-stakes oncall.
The web farm was about two-dozen Sun Ultra 2s connected on a FDDI loop (we eventually upgraded to GigE), with content on NetApps (4GB drives). A couple Sun E450s. Apache[1] on the Ultra 2s. Apache + mod_perl on the E450s. Hosted at a Global Center DC in Sunnyvale (same DC that early Yahoo was hosted in).
Monitoring with MRTG[2].
It's 1998. The Ken Starr report drops. Now, we knew it was coming, and we did our best to be prepared, but this is the late 90s. There was only so much load testing we could do and didn't really know how much traffic it would drive. We kept the site up, but it meant, as I recall, a lot of fine tuning of the mod_perl box and disabling interactive parts[3] of the site.
I really wish I still had some of the traffic graphs.
Fun times.
[1] Receipt: https://github.com/apache/httpd/blob/1.3.x/src/CHANGES#L5754
[2] Receipt: https://github.com/oetiker/mrtg/blob/master/src/CHANGES#L310...
[3] The forums. OMG forum software was so terrible. I think we eventually wrote our own after nothing based on Netscape Application Server or that we found open source worked well at all.
Space Park 1?
Famously, Yahoo wasn't quite up to the task on 9/11 which led to considerable thought about planning for the unexpected 100x event.
A lot of time it would end up being DNS or routing somewhere outside of our control.
Sometimes a disk would get close to filling up and a cron would clear it just in time so you would see decreasing performance then it would clear.
Other times we would have a customer that did something that we just did not expect with the system and it would cause SQL queries to slow down in ways we could not imagine. And it was transient so those would be hard to find or explain.
Or we would hit a limit in the load balancer (haproxy) in really odd ways. Too much traffic on the frontend or not enough capacity in the backend. And many other various ways of things not working. Haproxy was and still is amazing software. Really almost magical.
Ah, good old "it's always DNS"!
Britain being filled with Brits, what would happen is that once the show was over, half the nation got up from the couch to turn on the kettle for a cup of tea. Those electric kettles are quite demanding, especially if millions of them turn on at the same time.
So every time there was a major event, dedicated people are monitoring the grid frequency, power plants on standby and foreign contracts at the ready when needed, just to increase capacity at the right time. You can't just schedule this stuff, because if a football match runs over its allotted time, you may suddenly add power to the grid without any load, causing all kinds of problems like the grid frequency increasing and making digital clocks run ahead. There is a temporary demand of gigawatts of power that rises within five minutes and lasts until the kettles are done.
The YouTube video provides an example of 600MW of power being requested from France... for the end of an episode of Eastenders.
I always knew the British like their tea and that there's some kind of planning going on for events like street lights turning on, but the combination of the two is a great example of complex behaviour that's easy to overlook.
Particularly British kettles, which pull 13 amps at 240v = over 3000 watts. A lot more than (standard) American outlets can deliver.
For a while there was a theory going around that this is why Americans typically do not have electric kettles, because they would boil much slower in the USA, but the real answer is that Americans just don't drink much tea.
I would drink tea and use an electric kettle if it would boil faster...
Zojirushi has been in business since 1918 and (in my understanding) is a household name in Japan. Their more expensive (to be blunt) lines are made entirely Japan and replacement parts (like rubber seals, which can wear out over time) are available for purchase through their site. I fully expect mine to last 15-20 years of continuous duty.
Not affiliated, just a thrilled customer.
[1] https://www.zojirushi.com/app/product/cvjac
Edit for additional info: the Zojirushi replaced my Sunbeam Hot Shot single cup boiler that had an unfortunate dry-run and became unusable. They were unavailable in late '21, but appear to be back on the market now. They boil a cup of water in under a minute. If the Zojirushi didn't exist (or I hadn't been willing to bite the bullet on the price) I would buy another in a heartbeat.
Value is what is left when price has been forgotten. Both of these utensils have been high-value for me, but the Sunbeam has a lower initial cost of ownership.
Don't be foolish like some and have 240V wired to a standard plug, though! NEMA 6-15 and 6-20 exist just so you don't mistake your beefed up socket for a normal American socket and fry your devices/zap yourself or others.
Today I made tea and I used a standalone electric kettle which is odd to most of us here in Canada and the US. Most times I would have used the microwave to boil the water. But I used the kettle 120V x 12.5A or 1500W and popped the breaker since the dishwasher was operating and on the same circuit.
In Canada we drink a lot of tea; Red Rose, King Cole and especially south-eastern Canada The Maritimes and Atlantic Canada.
In theory you could sell a 240 volt electric kettle but your kitchen would have to have a 240 volt outlet, probably a NEMA 6-15. They're not common.
Not sure why you'd think so, but no.
(ovens and dryers are not wall plugs)
[0] https://en.wikipedia.org/wiki/NEMA_connector#NEMA_6
It's technically a single phase with two hot legs and a center tap for neutral, also called "split phase". From the neutral to either leg is 120V, from leg to leg is 240V.
https://en.wikipedia.org/wiki/Split-phase_electric_power
It would technically be possible to have a reverse-Y extension cord that plugged into two outlets and gave you a 240V circuit. The two outlets would have to be on different legs, of course. This is basically what a 240V breaker is doing. I've contemplating wiring up one of these for home and buying a British tea kettle, but the aesthetics got my plan vetoed.
Something like: https://www.nooutage.com/images/adaptor-wye.jpg
Ahem: the ones driven by the grid frequency are analogue clocks.
This function was so important that if load changes caused the frequency to fluctuate the power companies will compensate as soon as they can to make sure the clocks can catch up. I said "was" because I have no idea (and does anyone?) how many such clocks are still in service. But it's a legacy function and, like POTS, such features are still maintained. Because you never know who'll need it.
https://hackaday.com/2018/03/29/ask-hackaday-is-your-clock-t...
It's been my experience that, in the long term, mains-referenced clocks are extremely accurate, while local quartz ones seem to drift significantly. Clock also tend to last long enough that these are relatively common around here: https://en.wikipedia.org/wiki/Telechron
I don't have any clocks that do this, so I bought one of these, which keeps accurate time without intervention from me or the power company: https://mitxela.com/projects/precision_clock_mk_iii
Alarm clocks, microwave clocks, all manner of time indicators without sync mechanism built in were affected because relying on Europe's near perfect 50 Hz grid frequency is pretty damn cheap.
If you graphed the visit rate to our shopping cart page it would follow a curve that was sort of like a sine wave (well, a sine wave plus a constant because you can't have negative cart traffic) with a 24 hour period. I'm just going to call it a sine wave.
The amplitude varied seasonally, and was different on weekends and weekdays, but generally any given day was a sine wave.
If you graphed order rate it also followed a sine curve too, in phase with the visit curve and generally with a fixed ratio between them. So if the cart was getting N visits per minute, we'd be getting fN order per minute for some number 0 < f < 1, and f was pretty much constant.
This continued as the business grew, so the overall amplitude of the visit curve increased. But at some point the order rate seemed to hit a wall.
What we would then see is that as the visit rate was heading toward its peak for the day when it got to a particular fraction of that peak, say 80% of the peak, the order rate would stop tracking the visit rate and would become flat. It would stay flat as the visits peaked and declined until the visit rate was back down to 80% of its peak and then orders would resume tracking visits.
So far nothing too weird about this. What was weird is that when we'd have a day of especially high cart traffic such as when something in the news caused a big increase in demand for a few days for the kinds of things we sold, and we'd get 3 or 4 (or sometimes even 20 or 30) times the normal cart visit rate throughout the day--the plateau on the order rate would still start when the visit rate hit 80% of the visit rate peak for that day and go away when the visit rate fell back below 80% of peak.
The order processing was handled by a different server than the shopping cart. They did use the same database for product and customer information, but we could not find any signs that the database was a bottleneck.
This just made no sense. Suppose we'd been seeing on normal days that visits would peak at 1000 per minute, and orders were 10% of visits. Then we'd see visits grow from 0 to 800 and orders grow from 0 to 80. Then orders plateaued at 80 while visits continued on to 1000 and back to 800. Then as orders fell from 800 to 0, orders fell in step from 80 to 0.
Now suppose the next day we had 10 times the activity due to some news item. So the next day visits rise from 0 to 10000. The order system would now not plateau until the visit rate got to 8000 and the order rate got to 800.
And then when the increased demand was over in a few days and we were back to 1000 visits peak the order system would again plateau at 80.
So whatever was causing the plateau was somehow tied to the visit rate, making the plateau depend on what the peak visits would be for that day.
But how on a given day when the visits would first rise to 800 and the orders to 80...how did it "know" if the day was going to be a 1000 visit peak day or a 10000 visit day so it could "know" if it was time to plateau?
We never figured this out. Reasoning about our systems and their dependencies I could not figure out any way in theory that this could be happening nor could I find anything no matter how much logging and tracking I added all over the place.
Everything I could think of that could even remotely cause a plateau would have caused a fixed plateau, not some weird plateau that's level gets set daily based on what the future peak visit rate is going to be for that day.
SRO is madness. SRO is admitting that you're enough underwater and there's enough unexpected critical events to dedicate a team to burn out on it. There was a lot of spend in overcapacity and failover automation testing that kept it that way, and explicit mandates by a C-Suite and VP to keep keeping it that way. More importantly, though, it was baked into the architecture.
Everyone's talking this week about how Twitter will go up in flames in the next few days; I don't think so. I think they're closer to Google in that respect. There's a lot of automation that will keep things alive and a lot of redundancy that will handle the immediate problems of having a lot of the team leave. I actually think the company can continue to run indefinitely if they just follow the operations manuals and replace failed drives and machines. But I think there will be a major outage when they try to launch a new feature and fail.
They knew not much work was going to get done either way.
It was always noisy. If it's not the miscreants, its the normal stupidity (DNS is so amazingly survivable of misconfigurations, some of them become folk wisdom; like disabling TCP/53, but I digress).
Othodox Easter Sunday morning. Really really early in the morning, like I can't remember if it was light out yet (US Pacific coast time). The SOC calls me. Nothing is happening.
Nothing is happening. There is no noise. So they've been looking at this, and everything is working. So they called me. At dark-thirty on Sunday morning.
I confirm everything appears to be working correctly. I repeat diagnostics, do ad hoc stress tests, everything. About two hours later the normal crap starts coming back.
We have to conclude that the miscreants, as well as the sysadmins, had taken the morning off.