So If I understand it correctly, the electrical connector was genderless? If so, that's relatively rare (the only ones I can think of are Anderson Powerpole, which I don't think are rated for interplanetary vehicles) and extremely stupid.
Edit: I suppose he could've been using alligator leads.
You can configure an Anderson connector pair in a way that it won't reverse. I use them quite frequently, and have a local (though admittedly undocumented...) standard for what orientations mean what voltage. It's not bulletproof, but it does make me think when things don't line up.
A breakout box usually has both male and female sides, along with the banana or whatever breakout in the middle, so it can be plugged into either side of the circuit, or both at once and watch signals while the system operates. It's not genderless, it's a pass-through.
Making an only-faces-the-motor breakout, and a separate only-faces-the-driver breakout, might've been prudent, presuming that they used unique and consistent connectors, for instance a single gender always on the motors. But that's quite an assumption and I can imagine a ton of reasons it might not apply.
Why would you ever allow people to work 12-hour days on something so important? Grad student labour is cheap, surely trying to have one person do the work of two is a false economy.
Another relevant bit of info from hospital accidents: hand-offs between shifts are known to increase the risk of a mistake in care and are part of the reason nurses and doctors work such long hours.
I'm not questioning your logic here, but how do you keep intimate knowledge of the seasonal vagaries of shift changes at every department of your local hospital?
You can probably make a good guess, but at this point I wouldn't be surprised to find websites or Facebook groups dedicated to tracking this information for hospitals in any given area.
If you’ve been to that hospital you can easily take note. And most places have a shift change between 5-7am. This one is almost universal, as far as I have observed, even in different countries.
> work is too important to work 12 hour shifts on it
Yes. Because it is know that exhausted people make mistakes. The work is too important to let exhausted people screw it up so you should make sure everyone working on it is well rested.
> your solution is throwing “cheap” grad students at it
Yes? It is testing an electric motor. They can do it. The solution is that you employ enough people so nobody needs to work 12 hour heroic shifts.
That “solution” is nothing more than typical HN backseat driving.
In the real world there are budget, personnel and hiring constraints. You don’t get to hire all the people you want. You make do with what you have, and try to push the mission forward, even in suboptimal conditions.
On your next hospital visit for life-saving care, I'm sure you will be comforted to know that nurses (in the US) typically work 12 hour shifts and they're on their feet the whole time.
This is true, but IMO doesn't refute the point, it just makes me concerned about care quality. Were there any studies that showed that the long, grueling shifts are actually better or is it simply this way because it's always been that way and change would be hard and expensive, and because "my grandma walked up hill to school both ways, so young people can too"?
Who said it's gruelling? In the case of the rover, it's not year round, it's when they are preparing for a launch. Also, you're on HN, so surely you have heard of flow.
Nurses work 3x12 shifts to reduce the number of times patient care is handed off. If everyone worked 8 hour shifts you’d have 3 handoffs per day and a minimum of 3 different people caring for the patient. With 12 hour shifts you have only 2 handoffs per day and can have 2 people trade off on patients if their schedules line up.
I’m sure it varies by location, but my nurse friends only work 3x12, giving them 4 days off per week. Working 12 hour shifts is much more acceptable when you have more days off than days spent working. They’re virtually unavailable on days they work, but then they’re off traveling or having fun for 4 days, some times more if they combine their days off back to back. My close nurse friend routinely takes week long vacations without actually taking any time off at all.
Health care professionals is a weird one, because while long shifts are dangerous, patient handover is also dangerous and there may be an argument that longer shifts means fewer handovers which could result in better patient outcomes.
Would we accept this if they were dealing with nukes, rather than people? Yeah we let people who haven't had sleep in 36 hours handle the nukes because having the people involved talk to each other between shifts is hard.
They do know roughly how long it takes to take care of a patient & should be set up with overlapping shifts and to be winding down towards a normal shift (i.e. no new patients) so that there's no handoff of a single patient but no one is working long hours. Some patients might take longer than a single shift, but handoff is inevitable at some point. You can improve your handoff processes but you can't improve the decision making of someone working a 12 hour shift.
Maybe but I won't claim to know how to quantify things to evaluate proposals. I do know that even in tech with low stakes, hand off is a problem. I recall hearing teams trying to do 3 on-call teams in 3 different timezones and the team requested to scale back to 2 with longer hours because of the handoff problem (& these hand-offs were occurring daily).
I would prefer fewer patients per doctor then. It seems that the problem is due to the limited supply of doctors. In both countries where I lived, supply of doctors was artificially limited by regulation.
One of the hardest problems in a workplace is coordinating the workers. There's also a substantial overhead cost for every employee. I'm sure workers are less efficient at the end of a 12 hour shift, but shift changes also cost a lot and introduce lots of opportunities for errors.
I am guessing from my personal experience, but for the most part people tend to ask more repetitive technical jobs (like testing motors) to people lower in the ladder, so (relatively) less experienced people are going to touch the parts more.
Then such people are doing the "actual work" and overloaded with tasks, working overtime is a rule rather than the exception. The justification for this is that he was "getting experience" for trying to move up in his career. So all good.
Most people are going to remember that he messed up, rather than that he was working overtime to meet expectations, maybe except for the guy that pat him in his shoulder, he saw enough to understand it.
What a great write up. The tension preceding launch must have been tremendous, and lack of sleep adding to the possibility of such a small yet critical error. Good lesson for all of us.
Are the electronics in these rovers really so bespoke that they don't have multiple copies of each electronic component warehoused on-site?
I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?).
But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis should be doable in 2 weeks, but what do I know?
Typically your best bet is going with a provider that already produces whatever item, isn't bespoke to that particular program, and has an assembly line of sorts already set up. However the other problem is they are often built in low numbers to begin with so getting hold of a flight qualified unit will most likely be an issue as well. Also everything is tightly packed together which usually makes replacing something involve messing with a bunch of other items.
To be fair; there may be COTS Mars rovers for sale (possibly even cheap), they're just not certified or engineered specifically for Mars. Even an RC car might last long enough for a mission if it was free to get there.
Sometimes I wonder whether something like the SpaceX approach would work for these sorts of mission: develop a way to cheaply and reproducibly build the mission hardware, then iterate on it until it works.
They almost certainly had flight spares but with two weeks until your launch window, there is zero chance you are deintegrating multiple systems, swapping in the spare, reintegrating, and re running your acceptance test campaigns. And that is assuming that they damaged a subsystem. Back powering the entire spacecraft could have wrecked your power system and anything connected to it. You'd have to disposition every part of the system that was touched. It's much more involved than just swapping in the spare and sending it.
The implied context here is that you'd forgo the usual tests, because the alternative is to send nothing to Mars.
According to Wikipedia they could have stretched those 2 weeks to around 3 weeks, but after that they'd have missed the launch window.
The usual processes are there to have a near-certainty of a working rover, but under these circumstances I'd think they'd just YOLO it and hope for the best.
But that assumes they've got spare electrical components, or alternatively a better use for the booster sitting on the pad than such an improvised mission.
Spirit/Opportunity had the SSTB1 test rover, which supposedly had a complete set of scientific instruments. If it was fully qualified and tested, swapping it out could have been as easy as dropping it in the lander and writing a different serial number in the paperwork.
(I really doubt it was fully tested. But why else have a flight spare vehicle?)
I've worked on satellites, and yeah - everything is super bespoke, very low quantity, very expensive. There is probably a qualification unit, or a flight spare that may be available for many subsystems, but maybe not.
Integration is a long and complicated process. Pulling apart this bot, with however many fasteners, joints, etc, and then reassembling it correctly would be a decidedly non-trivial project that could easily take a month or two, given that testing has to be performed at each step along the way to ensure that every sensor and actuator is fully functional throughout the integration process.
This style of traditional aerospace assembly / integration is not particularly efficient. The only reason it is done this way is that for these kinds of missions you only get one shot, and total cost is ridiculously high so everything must be done correctly.
Any idea why they would use brushed motors? When every gram counts I would think ditching the mechanical commutator would be a no-brainer, but maybe adding another leg to the H-bridge is a bigger penalty?
In 2003, small brushless DC motors were way less mature and available as they are now, particularly for low-speed/high-torque applications*. Brushless controllers are much more complicated than brushed controllers, particularly on the control software front, so sticking with simpler and more reliable brushed controllers for a space application makes sense (remember, it probably needed to be radiation hardened - doing that for an H-bridge is much easier than a BLDC controller).
*A notable example of this is in the world of RC cars, where rock-crawlers only very recently have started switching to brushless motors using field-oriented control to deliver acceptable very-low-speed behavior. Until FOC controllers became available, brushed motors offered much better low-speed handling.
Without disagreeing with your point, would availability be an issue in this case? They need one or two, have an enormous budget, and if the technology exists can make their own.
Availability is often strongly correlated with technical maturity. Small brushless motors with FOC didn't become widely available and mature until really the late 2010s. Arguably the foundation of nearly all of DJI's product lines is due to their early mastery of small brushless motor control (drones, gimbals, lens controls, robots, etc), and that's a company founded in 2006, well after the events of the article.
You can get good control of brushed motors with just a couple of transistors. Good brushless control means FOC, which really requires a fairly capable microcontroller in addition to all the power electronics for variable-frequency drive. While brushed motors certainly have limitations, those were quite well understood by the early 2000s (to the point here that assessing whether or not damage had occurred was "just" a question of "have these few transistors suffered from voltage applied in an unintended manner"). Brushless motors involve way more components with way more integration required to make then small. Far more complexity and potential failure modes need to be understood.
I'd guess it's mostly because it was 2003 and decent BLDCs were not super common-place yet. There were some older forms (steppers, PMSMs, etc) but they generally didn't have very good torque/weight performance. Brushed motors would probably have been the answer at the time.
Closed loop control of brushless motors is just more complex, in addition to needing 3 phase AC output, you also need either hall sensors or an encoder of some kind to be able to start the motor smoothly, and you need a dedicated IC or MCU for each motor to manage commutation and read the sensors.
I don't think FOC type controllers were anywhere near common back then either, which is needed to run a brushless motor smoothly.
There is just so much more that can go wrong with a brushless setup, vs brushed where you just apply power and that's it.
My understanding is that these electronics need to be radiation hardened (eg; RADHARD) to prevent them from misbehaving in ways far worse than the Belgian bit flipping incident (https://radiolab.org/podcast/bit-flip). If you want to use commercial off the shelf (COTS) parts, you need to install them in triplicate and have them "vote". (https://apps.dtic.mil/sti/citations/ADA391766)
Your average installed Mars probe widget is not a fungible component; it's been integration tested six ways to Sunday and is certified in its current configuration, as is the assembly of the whole thing, center of gravity, cleanroom status, the torque on every fastener and so on. Even if it's an off the shelf component, it may not be possible to replace it without a chain of sign-offs that would require months of work.
I like this - “Let your scars serve you; they are an invaluable learning experience and investment in your capability and resilience.”
I have had couple of scars of mine. I feel like sometimes you become risk-averse and When you are launching new things you will face the fear of failure.
"My mistake that seemed to potentially cost up to $500M, but then it turned out fine" would be more accurate. While a more appropriate title would be something akin to "learning from mistakes" or "a Mars rover testing scare".
I'm reminded of the phrase - if your intern deleted the production database you don't have a bad intern; you have a bad process.
Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE.
I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of the ray of hope. That weight, I've never had a project that so many people depended on.
You should never blame the individual for organizational failures like this. I see two process issues:
1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation
2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it
Blaming individuals only gets them fired and weakens the entire organization. You just fired the one person who learned an expensive lesson.
The only time when an individual should be blamed is when they intended harm, at which case the law could kick in
You can't apply process thinking here, where the scenario is custom testing a unique probe, and you don't know what other constraints are in play (for example, the reason for the plug design). If NASA were sending these things to Mars by the dozen, then you can start to formalize things like test procedures and look for places mistakes can happen. But in this scenario, you're just disempowering your staff by not letting them choose the most effective and low-risk way to do one-off, highly specialized testing work.
Aerospace-grade connectors are specifically designed to support multiple keyings that prevent this kind of thing. It's definitely a problem preventable by careful design if the interface supports making this kind of mistake.
Can confirm. Source: I used to work for NASA, and I'm a private pilot. There are literally millions of electrical connections that get made on aircraft and spacecraft on a regular basis and I can't think of ever hearing of an incident caused by one of them being made backwards. (Now, mechanical connections getting made backwards is not unusual. That's why you check to make sure that the flight control surfaces move in the right direction as one of the last checklist items before you take off. Every. Single. Time.)
I can think of very few kinds of connectors for which this type of error is even possible. You would need two cable terminations which can connect to each other, for which either side can plug into the same jack.
So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors.
All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
A break out box could very sensibly have both sides of the connector on it and then have the various pins broken out into individual connections for flexibility.
In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
One guy in our workshop had to provide DC to a display with a round 4-pin connector. He soldered two neighboring pins to Gnd and the other two to Vcc. There were two chances to short the powersupply, one to brick the display and one to get it right. Guess what we had to replace until we found out.
Looks like the author tried to double-power the motor with both the spacecraft motor driver and through the breakout box that MITMs the driver and the motor. In such event, the free-wheeling diode in the driver will allow reverse current to be fed back to the driver's power supply up to certain amounts. This will absorb back-EMF, or energy from "regen" from the motor.
I'm suspecting the breakout wasn't literally sitting between the driver and the motor, but rather all internal connections are broken out to the box for testing; and likely the author's mistake was to not mess with the spacecraft to temporarily disconnect the driver.
But I'm not sure if I'd "just" made the right call and done so nonchalantly on a Mars rover to launch in few weeks.
It depends on what you mean by "that". Getting control surfaces actually reversed is not very common, but it does happen, typically after maintenance when a mechanic inadvertently re-connects a control cable backwards.
Control cables also can and do break, but that too is fairly rare.
What is not rare is control mechanisms jamming. Here is an example:
There have been several cases of the landing gear up/down lever getting wired backwards during maintenance. Not to worry, the gear has a 'squat switch' sensor that prevents the gear from being raised when the plane is on the ground. Unless you taxi over a bump and the switch decides it's now airborne. Crunch.
The one process part that can be controlled and jumped out from the first paragraphs is not letting people touch billion dollar equipment at the tail end of a 12-hour shift.
If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued.
I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.
I can't say about NASA, but I can say about my experience at ESA (European Space Agency), where I worked on Mars lander hardware. You have very very formal procedures and detailed checks as soon as you approach any parts which is going to fly.
The simplest task you can imagine takes incredible proportions (for good reasons).
Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ...
Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.
The Hacker News mic drop strikes again. I have nothing super substantive to add except to agree with your point and add that yes, it feels like work to put in the formal policies and procedures, but when the stakes are high enough (rocket to mars? its high enough), even the work that doesn't intuitively feel 'worth it' to someone is DEFINITELY worth it.
"It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone.
I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
Exactly :). The funny part is, the thing actually crashed! [1]
Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie !
Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
For one incidental mistake of course not. For repeated inattention (like plugging mars rover's cables wrongly several times) at an attention-demanding job -- yes.
Not in this case. It's a one-off very custom-built rover, the first of its name. There's already all kinds of processes established, but no one can foresee everything. Yes, they probably fixed the process after that, but remember that it was their first time.
PS: Also, more rules and better processes are not necessarily a good thing. Sometimes there are just too much red tape and bureaucracy that makes already super-slow NASA even slower. In those first-of-its-kind missions sometimes you need to risk and depend on people, not processes.
At my first real job as a web dev after school, I crashed the production website on my very first day. Tens of thousands of visitors were affected, and all our sales leads stopped.
Thankfully, we were able to bring it back up within a few minutes, but it was still a harrowing ordeal. The entire team (and the CEO in the next room) was watching. It ended up fine and we laughed about it after some minor hazing :)
But by the time I left that job a couple years later, we had turned that fragile, unstable website into something with automatic testing, multiple levels of backups and failover systems across multiple data centers, along with detailed training and on-boarding for new devs. (This was in the early days of AWS, and production websites weren't just a one click deploy yet.)
That one experience led to me learning proper version control, dev environments, redis, sharding and clustering, VMs, Postgres and MySQL replication, wiki, monit, DNS, load balancers, reverse proxies, etc. All because I was so scared of ever crashing the website again.
That small company took a chance on me, a high school dropout with some WordPress experience, and paid me $15/hour to run their production website, lol. But they didn't fire me after I screwed up, and gave me the freedom and trust to learn on the job and improve their systems. I'm forever grateful to them!
At a major brokerage firm I accidentally hit prod with a testing script that did several millions of dollars of fake FX test trades.
The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.
Back in school, my roommate's mom worked for a hedge fund and he did part-time work for them. He factored out a common trading engine from individual strategies, and one day the head of the fund asked him to run a strategy that had made a bunch of money in the past, but had been retired after failing to make money for a while. So, he put the strategy back in production without any testing, forgetting that he had recently done some minor refactoring of the trading engine. He typo'd one variable name for a similar variable name, so in the loop where it broke down large orders into small orders, it actually had an infinite loop. Luckily the engine had an internal throttle, so it wasn't trading as fast as it could send messages over the network.
I was chatting with him when he noticed the stock the strategy was trading (KLAC) was gradually declining linearly. He looked at the L2 quotes and saw that someone using his brokerage was repeatedly putting out small orders, and then he realized they were his orders.
The fund got a margin call and had to shift some funds between accounts to make margin, and they had to contact regulators and inform them of the bug, and they had to manually trade their way out of the massive short position they traded. However, they ended up making $60,000 that day off of his mistake.
No, it was caught by our trading ops guys. A few minutes after I hit enter I got. Rather chilling phone call from them. So that part of the system worked
I can't help but think about what would've happened if the Rover had indeed been destroyed though. It seems the only thing that stopped that from happening was sheer luck as they could as easily ( I guess ) have connected to another wrong lead that wouldn't have the protection required to survive the charge? That is, it was outside the author's actual abilities to have stopped that and he could just as easily have been the destroyer of the Rover and forever remembered for that fact, as he had feared he would.
The fact that they were still being made to work after completing a 12 hour shift (which is already too long to be safe) means this was a process error
Ohhhh I hate things that you “just have to not screw up.” A fiddly manual process with so many possible ways to screw something up is almost guaranteed to see a catastrophic failure.
If this really manual fiddly process was really the only way they could test the motors, I’d say that’s a big failure on the design engineer’s part.
Test requirements often arise after system design is complete. Engineers need time to examine the system, theorize failure modes, and design the tests. Also time to test the system, find some unexpected failure mode, and then design yet more tests.
Would it be so hard to add a diagnostic interface? That’s all we’re talking about here. It seems like you’re making the problem more difficult than it is. And they’d had to have known ahead of time if they wanted to test voltage curves of motors.
Your answer is good in the general case, but for the anecdote, the design was clearly bad.
Or it just needs a budget that can dramatically increase and/or have a deadline that can continue to be pushed back to allow for corrections. As exhibit A, I'd like to present the james webb space telescope.
Any test that could cause a fatal destructive error should be risk assesed and a suitable protocol approved with four eyes approval on a final checklist before going hot on the electrics. The issue here is poor project governance not human error.
Yeah, the borrowed multimeter really hammers how silly this all was. You don't touch other peoples lab equipment unless the other ends of the wires are hanging free. A finger pointing to a meter needs to be followed up with a clear confirmation that the wires can be disconnected, and special care isn't needed in the process. If I need something that's connected, I always ask the person to disconnect it for me. Definitely a process/culture problem.
I guess the idea is that it potentially could have cost $500 million therefore it was a $500 million mistake. It's not exactly accurate but it does help contextualize the gravity of their mistake.
Another failure is from the person who used the multimeter to complete a circuit - and then didn't even leave a note on the multimeter. That person could reasonably have anticipated this error mode, and taken steps to prevent it.
Reminds me of a quote attributed to Thomas J Watson:
Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?
I guess the idea is that if the problem was incompetency they will find some other way to mess up as opposed to a genuine mistake made by a competent person.
> I guess the idea is that if the problem was incompetency they will find some other way to mess up as opposed to a genuine mistake made by a competent person.
Are you suggesting that a competent person never messes up/makes a mistake?
The most fundamental part of life is learning from mistakes, and today even AI is starting to do this. Mistakes and evolution are what _make us_ human and living.
Right, but we can all agree that they still make them, and sometimes big ones. Unless they are so careful that they severely limit both their potential and contribution to group. Or they are the rare person who happens to be both "perfect" and lucky.
One thing about long aerospace missions like this with huge lead times that always gets me - you can spend years of your life working on a mission, only for it all to fail with potentially years until you can try again.
This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission failed after all of this work? How could you cope?" to the folks at (now bankrupt) Masten Aerospace during a job interview, and maybe it was a bad time to ask such a question, but I didn't get the sense they knew either. "The best thing we can do is learn from failure," one of them told me. An excellent thing to do, but not exactly what I asked. This to me stands out as the defining personal risk of caring about your job and working in aerospace. Get too invested, and you may literally see your life's work go up in flames.
Did they have a narrow launch window they couldn't afford to miss? I'm not talking about missions where you eat a big monetary loss on the launchpad and try again, I mean missions which rely on planetary alignments that may not happen again for years, or even the rest of your life, such as Voyager. Or even just missions where you launch successfully, but then after months (or years) of flight time the spacecraft is lost.
As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day.
Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had in place was awful and nothing like what we have today—just about every aspect of the process was manual. Anyway, everything was going well. I wrote a few scripts and SQL queries to automate the parts I could. They gave me the production credentials for when I’m ready to deploy. I decided to run what you could call my migration script one last time just to be sure I’m ready. The very moment after I hit the Enter key, I realized I had made a mistake: I had just updated the script with the production credentials just before I made the decision to do another test run. The errors started piling and their service was unresponsive. I was 100% sure I had just wiped their database and I was losing it internally. What saved me was that one of their guys had just a couple hours earlier completed a backup of their database in anticipation of the launch; in the end, they lost a tiny bit of data but most of it was recovered via the backup. Ever since then, “careful” is an extreme understatement when it comes to how I interact with database systems—and production systems in general. Never again.
Your excellent story compelled me to share another:
We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client.
Our test suites drop and recreate our tables, or truncate them, as part of the test run.
One day, a lead developer ran “make test” after he’d been doing some exploratory work in the prod database as part of a bug fix. The test code respected the environment variables and connected to prod instead of docker. Immediately, our tests dropped and recreated the production tables for that database a few dozen times.
Ours are not named with a common identifier and this also needs constant effort to maintain while refactoring and there's still scope for a mistake.
*ideally* devs should not have prod access or their credentials should only have limited access without permissions for destructive actions like drop/truncate etc.
But in reality, there's always that one helpful dba/dev who shares admin credentials for a quick prod fix with someone and then those credentials end up in a wiki somewhere as part of an SOP.
And when your last line of defense fires... you don't just breath a sigh of relief that the system is robust. You also must dig in to how to catch it sooner in your previous lines.
For instance, test code shouldn't have access to production DB passwords. Maybe that means a slightly less convenient login for the dev to get to production, but it's worth it.
Just yesterday, I did C# Regex.Match with a supersimple regex: ^\d+ And it seemed not to work. I asked ChatGTP and he noted that I had subtle mistake: the parameters were other way around... :facepalm:
I've added a similar safety to every project. It's not perfect, but this last line of defense has saved team members from themselves more than once.
For Django projects, add the below to manage.py:
env_name = os.environ.get("ENVIRONMENT", "ENVIRONMENT_NOT_SET")
if env_name in TEST_PROTECTED_ENVIRONMENTS and "test" in sys.argv:
raise Exception(f"You cannot run tests with ENVIRONMENT={env_name}")
I think runtime checks like this using environment variables is great however what has burned me in the past is that when debugging problems, not knowing what happened at runtime when the logs were produced was problematic. So when test protected environments environment variable needed to be updated I might have a hard time back tracking to it
Our test harness takes an optional template as input and immediately copies it.
It’s useful to distribute the test anyway, especially for non-transactional tests.
If the database initialisation is costly that’s useful even if tests run on empty, as copying a database from a template is much faster than creating one DDL by DDL, for postgres at least.
We had this - 10 years ago. In our case there was a QA environment which was supposed to be used by pushing code up with production configs, then an automated process copied the code to where it actually ran _doing substitutions on the configs to prevent it connecting to the production databases_. However this process was annoyingly slow, and developers had ssh access. So someone (not me) ssh'd in, and sped up their test by connecting the deploy location for their app to git and doing a git pull.
Of course this bypassed the rewrite process, and there was inadequate separation between QA and prod, so now they were connected to the live DB; and then they ran `rake test`...(cue millions of voices suddenly crying out in terror and then being suddenly silenced). The DB was big enough that this process actually took 30 minutes or so and some data was saved by pulling the plug about half-way through.
And _of course_ for maximum blast radius this was one of the apps that was still talking to the old 'monolith' db instead of a split-out microservice, and _of course_ this happened when we'd been complaining to ops that their backups hadn't run for over a week and _of course_ the binlogs we could use to replay the db on top of a backup only went back a week.
I think it was 4 days before the company came back online; we were big enough that this made the news. It was a _herculean_ effort to recover this; some data was restored by going through audit logs, some by restoring wiped blocks on HDs, and so on.
One place I worked (some 20 years ago) had a policy that any time you run a sudo command, another person has to check the command before you hit enter. Could apply the same kind of policy/convention for anything in production.
That's not a good advice IMO, as most sudo commands will mess-up just one host, and it's something you should generally be prepared for. You're more likely to develop a culture where engineers think about hosts as critical resources whereas they should be generally considered as instances that can be thrown away. It's better to identify hosts that are SPOF and be cautious on those only.
I can think of a larger blast radius when deleting files on a shared mount point for example but it's not representative to the regular use of sudo.
I have a rule when working on production databases: Always `start transaction` before doing any kind of update - and pay close attention to the # of rows affected.
In your .psqlrc and then you can never forget the begin transaction; every statement is already in a transaction, its just the default behaviour to automatically commit the statements for some ungodly reason.
Years ago I hired an experienced Oracle developer and put him to work right away on a SQL Server project. Oracle doesn't autocommit by default, and SQL Server does. You don't want to learn this when you type "rollback;". I took responsibility and we had all the data in an audit table and recovered quickly. I wonder if there are still people who call him "Rollback" though.
That's good from the DBA perspective, but relying on that default as a user is risky in itself, when you deal with multiple hosts and not all are set up this way.
As a young consultant, I was once one Enter away from causing a disaster, but something stopped me. I still shudder even though it didn't actually happen. Nothing of the sort in many years since, so a great lesson in retrospect I guess.
Anecdote: I ran a migration on a production database from inside Visual Studio. In retrospect, it was recoverable, but I nearly had a heart attack when all the tables started disappearing from the tree view in VS…
…only to reappear a second later. It was just the view refreshing! Talk about awful UI!
Around 15 years ago, I was packing up getting ready to leave for a long weekend. One of our marketing people I was friends with comes over with a quick change to a customers site.
I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut.
I write a quick little UPDATE query to update some marketing text on a product and when the query takes more than an instant I knew I had screwed up. Reading my query, I quickly realize I had ran the UPDATE entirely unbounded and changed the description of thousands and thousands of products.
Our database admin with access to the database backups had gone home hours earlier as he worked in a different timezone. It took me many phone calls and well over an hour to get ahold of him and get the descriptions restored.
The quick change on my way out the door ended up taking me multiple hours to resolve. My friend in marketing apologized profusely but it was my mistake, not theirs.
As far as I remember we never heard anything from the client about it, I put that entirely down to it being 5pm on Friday of a holiday weekend.
That's why I always write a BEGIN statement before executing updates and deletes. If they are not instant or don't return the expected number of modified rows I can just rollback the transaction.
Or is the lesson to _always_ attempt such critical changes on a Friday? After all, in this instance the client didn't notice any problems, apparently because they were already off to their weekend.
For me personally the much bigger issue would be harming the client, their business or our relationship. Doing a few hours of overtime to fix my mistakes would probably only feel as well deserved punishment...
At a place I was consulting about 10 years ago one of the internal guys on another product dropped the prod database because he was logged into his dev db and the prod db at the same time in different windows and he dropped the wrong one. Then when they went to restore the backups hadn't succeeded in months (they had hired consultants to help them with the new product for good reason).
Luckily the customer sites each had a local db that synced to the central db (so the product could run with patchy connectivity), but the guy spent 3 or 4 days working looooong days rebuilding the master db from a combination of old backups and the client-site data.
> logged into his dev db and the prod db at the same time in different windows
I am very worried about doing the wrong thing in the wrong terminal, so for some machines I colour-code my ssh windows, red for prod, yellow for staging and green for dev. e.g. in my ~/.bashrc I have: echo -ne '\e]11;#907800\a' #yellow background
About 10 years ago I literally saw the blood drain from a colleagues face as he realised he had dropped a production database because he thought he was in a dev environment.
A DBA colleague sitting nearby laughed and had things restored back within a few minutes....
A friend once had to remotely do an OS update of a banking system. Being cautious, he thought he'd back up some central files, just in case and went "mv libc.so old_libc.so". Had to call some guy in that town to throw in the Solaris CD on prem at 2:30 in the morning...
What strikes me as remarkable in all such stories is how almost always, the person committing the mistake is a junior who never deserves the blame. And how cavalier the handoff/onboarding by the 'seniors' working on the projects are.
Having worked in enough of these though, I am aware that even they (the "seniors") are seldom entirely responsible for all the issues. It's mostly business constraints that forces cutting of corners and that ends up jeopardizing the business in the long run.
As I said on Slack the other day in response to a similar story, "If, on your first day, you can destroy the prod database, it's not your fault."
(One of my standard end-of-interview questions is "how easy is it for me to trash the production database?" Having done this previously[1] and had a few near misses, it's not something I want to do again.)
[1] In my defence, I was young and didn't know that /tmp on Solaris was special. Not until someone rebooted the box, anyway.
it gets wiped on reboot. I remember around 2007 on Gentoo Linux, this behavior changed. I was using /tmp as pretty much a "my documents" type folder, I updated, and one day all my stuff was gone! I was flabbergasted. But yeah, it was reckless to store things on a folder that pretty has "temp" in the name!
The blog post lays it on a bit thick with the $500 million number and the "launch only two weeks away" given that the article itself is illustrated with a photo of the Sojourner flight spare. Spirit had the SSTB1 test rover. If he had actually blown out the entire electrical system, they could have launched it instead. Swapping out the entire vehicle right before launch would have been an awful job, but it's not flat out impossible.
Not rude at all! I appreciate the reply. Only reason I deleted my message was because right after posting, I scrolled down and saw someone asking the exact same question at the top level, so I felt like it was best to conserve effort and not repeat them.
I liked that other people pointed out that risk could have been eliminated by using polarized connectors (I hope they started doing this after the incident), but also made me wonder about "back-EMF" caused by solar flares. In other words, maybe all thick wires and ground/power planes should be hardened against current surges simply due to a solar event hitting mars (which may incidentally cover the case of back-powering the driver circuits).
One way I mistake-proof things in SQL Management Studio is to have different colors for production vs test databases.
To do that, on the "connect to server" dialog, click "options". On the tab "connection properties" in the "connection" option group, check "use custom color". And I pick the reddest red there is. The bottom of the results window will have that color.
edit: my horrible foul-up was restoring a database to production. The "there is trouble" pagers were all Iridium pagers since they loved climbing mountains (where there was no cell service back then). But then that place didn't use source control, so it was disasters all the way down.
Reminds me of the NOAA-N Prime satellite that fell over, because there weren't enough bolts holding it to the test stand.
The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the floor. Repairs cost $135M.
Bravo to the author. I think if I had made the same mistake, and if the disaster is permanent, I would probably be scarred for life and never recover from it.
Interesting to see that the worry could have been avoided if they had lined up their timelines better in the first place. If they'd compared the timestamp on the test readout to the last timestamp from the telemetry system, they'd have seen that the telemetry failed BEFORE the test was executed. Partially caused by using imprecise language "we seem to have lost all spacecraft telemetry just a bit ago" rather than an accurate timestamp.
A cautionary lesson in properly checking how exactly events are connected during an incident. Easy to look at two separate signals and assume they must be causal in a particular direction, when in reality it is the other way around.
I work in TV. During my first job at a small market station 30 years ago, I was training to be a tape operator for a newscast. All the tapes for the show were in a giant stack. There were four playback VTRs. My job was to load each tape and cue it to a 1-second preroll. When a tape played and it was time to eject that tape, it was _very_ easy to lose your place and hit the eject button on the VTR that was currently being played on the air instead of the one that they just finished with. The fella who was training me did something very annoying, but it was effective: every time I went to hit the eject button, he would make a loud cautionary sound by sucking air through his closed teeth as if to tell me I was about to make a terrible mistake. I would hesitate, double check and triple check to make sure it was the right VTR, and then I would eject the tape. He made that sound every single time my finger went for the eject button. It really got on my nerves, but it was a very good way to condition me to be cautious. Our station had a policy: the first time you eject a tape on the air got you a day off without pay; the second time put you on probation; the third time was dismissal. I had several co-workers lose their jobs and wreck the newscast due to their chronic carelessness. Thanks to my annoying trainer, I learned to check, check again, and check again. I never ejected a tape on the air. It certainly would not have been a half-billion dollar mistake if I had, but at that point in my career it would have felt like it to me.
That explains the old blooper reels that were popular on TV in the early 80's, where the reporter would be talking about something, and get video of something completely bonkers in the background instead.
360 comments
[ 3.4 ms ] story [ 308 ms ] threadEdit: I suppose he could've been using alligator leads.
Making an only-faces-the-motor breakout, and a separate only-faces-the-driver breakout, might've been prudent, presuming that they used unique and consistent connectors, for instance a single gender always on the motors. But that's quite an assumption and I can imagine a ton of reasons it might not apply.
Yes. Because it is know that exhausted people make mistakes. The work is too important to let exhausted people screw it up so you should make sure everyone working on it is well rested.
> your solution is throwing “cheap” grad students at it
Yes? It is testing an electric motor. They can do it. The solution is that you employ enough people so nobody needs to work 12 hour heroic shifts.
In the real world there are budget, personnel and hiring constraints. You don’t get to hire all the people you want. You make do with what you have, and try to push the mission forward, even in suboptimal conditions.
Sir, this is NASA not Kerbal space program ;)
I’m sure it varies by location, but my nurse friends only work 3x12, giving them 4 days off per week. Working 12 hour shifts is much more acceptable when you have more days off than days spent working. They’re virtually unavailable on days they work, but then they’re off traveling or having fun for 4 days, some times more if they combine their days off back to back. My close nurse friend routinely takes week long vacations without actually taking any time off at all.
Then such people are doing the "actual work" and overloaded with tasks, working overtime is a rule rather than the exception. The justification for this is that he was "getting experience" for trying to move up in his career. So all good.
Most people are going to remember that he messed up, rather than that he was working overtime to meet expectations, maybe except for the guy that pat him in his shoulder, he saw enough to understand it.
I figured this was going to be a story about trying to measure voltage with the meter set up on the 10A current range.
I'd expect that the rover body itself would be bespoke this late in the process, although a parallel test vehicle would be useful, do they have that?).
But in case someone fried the rover's electronics I'd think tearing it apart and replacing them while maintaining the chassis should be doable in 2 weeks, but what do I know?
According to Wikipedia they could have stretched those 2 weeks to around 3 weeks, but after that they'd have missed the launch window.
The usual processes are there to have a near-certainty of a working rover, but under these circumstances I'd think they'd just YOLO it and hope for the best.
But that assumes they've got spare electrical components, or alternatively a better use for the booster sitting on the pad than such an improvised mission.
(I really doubt it was fully tested. But why else have a flight spare vehicle?)
*A notable example of this is in the world of RC cars, where rock-crawlers only very recently have started switching to brushless motors using field-oriented control to deliver acceptable very-low-speed behavior. Until FOC controllers became available, brushed motors offered much better low-speed handling.
You can get good control of brushed motors with just a couple of transistors. Good brushless control means FOC, which really requires a fairly capable microcontroller in addition to all the power electronics for variable-frequency drive. While brushed motors certainly have limitations, those were quite well understood by the early 2000s (to the point here that assessing whether or not damage had occurred was "just" a question of "have these few transistors suffered from voltage applied in an unintended manner"). Brushless motors involve way more components with way more integration required to make then small. Far more complexity and potential failure modes need to be understood.
I see what you mean. Yes, agreed.
I don't think FOC type controllers were anywhere near common back then either, which is needed to run a brushless motor smoothly.
There is just so much more that can go wrong with a brushless setup, vs brushed where you just apply power and that's it.
I have had couple of scars of mine. I feel like sometimes you become risk-averse and When you are launching new things you will face the fear of failure.
Whether this was a process problem or a human one we don't really get to judge since we do expect more from a FTE.
I'll just say putting myself into his shoes made me tear up as I read the dread and pangs of pain upon realizing what happened - then to have life again after the failure of the ray of hope. That weight, I've never had a project that so many people depended on.
All heroes in my book.
1. The plug was allowed to be connected backwards. Either this should be impossible or this hazard should be identified and more than one human should verify orientation
2. In use tools like multimeters should never be disconnected. At worst you get problems like this at best you annoy whoever was using it
Blaming individuals only gets them fired and weakens the entire organization. You just fired the one person who learned an expensive lesson.
The only time when an individual should be blamed is when they intended harm, at which case the law could kick in
Like say they have one that is setup to test the motor driver circuitry and another one that is setup to test the motor?
Or say the breakout box intentionally has both sides of the connection on it, so that you can get in-between the driver and motor?
So either the ends are literally the same (e.g. Anderson Powerpole), or there is some kind of weird symmetry or inadequate keying. Or maybe the two cables don’t connect directly and instead go through some of kind of interface? The latter is fairly common in networking, e.g. “feed-through” patch panels and keystone jacks and quite a few kinds of fiber optic connectors.
All of these seem like utterly terrible ideas in an application where you would take the thing apart after final assembly and where the person doing the disassembly or reassembly could possibly access the wrong side of the panel.
In that case keying or whatever isn't going to prevent you from connecting to the wrong side, because both sides are present.
I'm suspecting the breakout wasn't literally sitting between the driver and the motor, but rather all internal connections are broken out to the box for testing; and likely the author's mistake was to not mess with the spacecraft to temporarily disconnect the driver.
But I'm not sure if I'd "just" made the right call and done so nonchalantly on a Mars rover to launch in few weeks.
"The incident was featured in season 23, episode 5 of the Canadian documentary series Mayday . . ." [1]
Season 23 - I'm glad I don't fly!
1. https://en.wikipedia.org/wiki/Air_Astana_Flight_1388
How … how often does that go wrong?!
Control cables also can and do break, but that too is fairly rare.
What is not rare is control mechanisms jamming. Here is an example:
https://www.ntsb.gov/news/press-releases/Pages/NR20230928A.a...
If you are putting people in a situation with absolutely no safeguards, you can’t have them go into it fatigued.
I’m guessing the people working on that team also weren’t getting great sleep by the discussion of high stress and long hours. Recipe for disaster.
The simplest task you can imagine takes incredible proportions (for good reasons).
Disconnect and reconnect that plug? Please inform persons X and Y, person Z must be present, only person W can touch that plug, and do perform a functional test according to the procedure in this document before and after and file these reports etc ...
Cleaning a part? Oh glob. Get ready for 3 months of adventure talking to planetary protection experts and book the cleanest room in the continent.
"It's a waste of time" is very often a fallacy, especially when the risk cannot be easily undone.
I (mostly mentally) complete the phrase "It's a waste of time" with "what's the worst that could happen?", and when I'm actually saying the phrase out loud, stare at whoever said that for 5 full seconds.
Why? Bad error handling in the software (primarily). What is the worst that could happen? An instrument saturate, a variable gets stuck at a value, but keeps being integrated, the spacecraft computes a negative altitude and thinks it'a below ground level (negative altitude) but is in fact in full descent and at 3+ km from the surface. Oopsie !
[1] https://exploration.esa.int/web/mars/-/59176-exomars-2016-sc...
Agree on the blame point, but not on firing point. As a manager, sometimes you need to fire people, that's a necessary part your job. And no, changing the hiring process cannot prevent that.
PS: Also, more rules and better processes are not necessarily a good thing. Sometimes there are just too much red tape and bureaucracy that makes already super-slow NASA even slower. In those first-of-its-kind missions sometimes you need to risk and depend on people, not processes.
At my first real job as a web dev after school, I crashed the production website on my very first day. Tens of thousands of visitors were affected, and all our sales leads stopped.
Thankfully, we were able to bring it back up within a few minutes, but it was still a harrowing ordeal. The entire team (and the CEO in the next room) was watching. It ended up fine and we laughed about it after some minor hazing :)
But by the time I left that job a couple years later, we had turned that fragile, unstable website into something with automatic testing, multiple levels of backups and failover systems across multiple data centers, along with detailed training and on-boarding for new devs. (This was in the early days of AWS, and production websites weren't just a one click deploy yet.)
That one experience led to me learning proper version control, dev environments, redis, sharding and clustering, VMs, Postgres and MySQL replication, wiki, monit, DNS, load balancers, reverse proxies, etc. All because I was so scared of ever crashing the website again.
That small company took a chance on me, a high school dropout with some WordPress experience, and paid me $15/hour to run their production website, lol. But they didn't fire me after I screwed up, and gave me the freedom and trust to learn on the job and improve their systems. I'm forever grateful to them!
The first thing mentioned in the post mortem call was “No one is going to blame the guy who did those trades. It was an honest mistake. What we are going by to do is discuss why a developer can hit the production trading API without any authentication at all”.
I was chatting with him when he noticed the stock the strategy was trading (KLAC) was gradually declining linearly. He looked at the L2 quotes and saw that someone using his brokerage was repeatedly putting out small orders, and then he realized they were his orders.
The fund got a margin call and had to shift some funds between accounts to make margin, and they had to contact regulators and inform them of the bug, and they had to manually trade their way out of the massive short position they traded. However, they ended up making $60,000 that day off of his mistake.
If this really manual fiddly process was really the only way they could test the motors, I’d say that’s a big failure on the design engineer’s part.
Your answer is good in the general case, but for the anecdote, the design was clearly bad.
Here's one they just made up: "near miss". When two planes almost collide, they call it a near miss. It's a near hit. A collision is a near miss.
[1]: https://www.quotes.net/mquote/35854
The first mistake is the $500 million rover is fried.
The second mistake is believing the first mistake.
(or put another way, the first mistake cost $500 million, the second mistake which they didn't realise at the time, saved $500 million)
But you can't explain the second mistake without first explaining the first mistake, hence the title.
Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?
Are you suggesting that a competent person never messes up/makes a mistake?
The most fundamental part of life is learning from mistakes, and today even AI is starting to do this. Mistakes and evolution are what _make us_ human and living.
This is a refreshingly humanizing article, but is also one written from the perspective of a survivor. Imagine if the rover were actually lost. I asked the question "what would you do if the mission failed after all of this work? How could you cope?" to the folks at (now bankrupt) Masten Aerospace during a job interview, and maybe it was a bad time to ask such a question, but I didn't get the sense they knew either. "The best thing we can do is learn from failure," one of them told me. An excellent thing to do, but not exactly what I asked. This to me stands out as the defining personal risk of caring about your job and working in aerospace. Get too invested, and you may literally see your life's work go up in flames.
I would argue that if we don't charge the process to prevent this kind of catastrophic failure mode then we really haven't learned from the failure.
Incidentally, this happened to Lewicki a few years later when Planetary Resources' first satellite blew up on an Antares rocket: https://www.geekwire.com/2014/rocket-carrying-planetary-reso...
As a software engineer, I have a couple stories like this from earlier in my career that still haunt me to this very day.
Here’s a short version of one of them: Like 10 years ago, I was doing consulting work for a client. We worked together for months to build a new version of their web service. On launch day, I was asked to do the deployment. The development and deployment process they had in place was awful and nothing like what we have today—just about every aspect of the process was manual. Anyway, everything was going well. I wrote a few scripts and SQL queries to automate the parts I could. They gave me the production credentials for when I’m ready to deploy. I decided to run what you could call my migration script one last time just to be sure I’m ready. The very moment after I hit the Enter key, I realized I had made a mistake: I had just updated the script with the production credentials just before I made the decision to do another test run. The errors started piling and their service was unresponsive. I was 100% sure I had just wiped their database and I was losing it internally. What saved me was that one of their guys had just a couple hours earlier completed a backup of their database in anticipation of the launch; in the end, they lost a tiny bit of data but most of it was recovered via the backup. Ever since then, “careful” is an extreme understatement when it comes to how I interact with database systems—and production systems in general. Never again.
Great story, thanks for sharing.
We rarely interact directly with production databases as we have an event sourced architecture. When we do, we run a shell script which tunnels through a bastion host to give us direct access to the database in our production environment, and exposes the standard environment variables to configure a Postgres client.
Our test suites drop and recreate our tables, or truncate them, as part of the test run.
One day, a lead developer ran “make test” after he’d been doing some exploratory work in the prod database as part of a bug fix. The test code respected the environment variables and connected to prod instead of docker. Immediately, our tests dropped and recreated the production tables for that database a few dozen times.
One of the reasons I put interactions between databases behind a cli.
*ideally* devs should not have prod access or their credentials should only have limited access without permissions for destructive actions like drop/truncate etc.
But in reality, there's always that one helpful dba/dev who shares admin credentials for a quick prod fix with someone and then those credentials end up in a wiki somewhere as part of an SOP.
If you need access for a quick prod fix, your key gets added to the machine with that explanation and a week (or lees) lifetime.
There is no code that will protect your db/data. Only replication to a read-only storage will help in such situations.
For instance, test code shouldn't have access to production DB passwords. Maybe that means a slightly less convenient login for the dev to get to production, but it's worth it.
For Django projects, add the below to manage.py:
It’s useful to distribute the test anyway, especially for non-transactional tests.
If the database initialisation is costly that’s useful even if tests run on empty, as copying a database from a template is much faster than creating one DDL by DDL, for postgres at least.
Of course this bypassed the rewrite process, and there was inadequate separation between QA and prod, so now they were connected to the live DB; and then they ran `rake test`...(cue millions of voices suddenly crying out in terror and then being suddenly silenced). The DB was big enough that this process actually took 30 minutes or so and some data was saved by pulling the plug about half-way through.
And _of course_ for maximum blast radius this was one of the apps that was still talking to the old 'monolith' db instead of a split-out microservice, and _of course_ this happened when we'd been complaining to ops that their backups hadn't run for over a week and _of course_ the binlogs we could use to replay the db on top of a backup only went back a week.
I think it was 4 days before the company came back online; we were big enough that this made the news. It was a _herculean_ effort to recover this; some data was restored by going through audit logs, some by restoring wiped blocks on HDs, and so on.
We never hear about first time launch deploys that wipe ALL data because whoever is so unlucky probably never got to browse hacker news
I can think of a larger blast radius when deleting files on a shared mount point for example but it's not representative to the regular use of sudo.
Years ago I hired an experienced Oracle developer and put him to work right away on a SQL Server project. Oracle doesn't autocommit by default, and SQL Server does. You don't want to learn this when you type "rollback;". I took responsibility and we had all the data in an audit table and recovered quickly. I wonder if there are still people who call him "Rollback" though.
That's good from the DBA perspective, but relying on that default as a user is risky in itself, when you deal with multiple hosts and not all are set up this way.
…only to reappear a second later. It was just the view refreshing! Talk about awful UI!
I had access to the production database, something I absolutely should not have had but we were a tiny ~15 person company with way more clients than we reasonably should have. Corners were cut.
I write a quick little UPDATE query to update some marketing text on a product and when the query takes more than an instant I knew I had screwed up. Reading my query, I quickly realize I had ran the UPDATE entirely unbounded and changed the description of thousands and thousands of products.
Our database admin with access to the database backups had gone home hours earlier as he worked in a different timezone. It took me many phone calls and well over an hour to get ahold of him and get the descriptions restored.
The quick change on my way out the door ended up taking me multiple hours to resolve. My friend in marketing apologized profusely but it was my mistake, not theirs.
As far as I remember we never heard anything from the client about it, I put that entirely down to it being 5pm on Friday of a holiday weekend.
Oh, and I absolutely refuse to do anything but the most critical stuff against prod on Fridays.
For me personally the much bigger issue would be harming the client, their business or our relationship. Doing a few hours of overtime to fix my mistakes would probably only feel as well deserved punishment...
Luckily the customer sites each had a local db that synced to the central db (so the product could run with patchy connectivity), but the guy spent 3 or 4 days working looooong days rebuilding the master db from a combination of old backups and the client-site data.
I am very worried about doing the wrong thing in the wrong terminal, so for some machines I colour-code my ssh windows, red for prod, yellow for staging and green for dev. e.g. in my ~/.bashrc I have: echo -ne '\e]11;#907800\a' #yellow background
A DBA colleague sitting nearby laughed and had things restored back within a few minutes....
https://news.ycombinator.com/item?id=14476421
Having worked in enough of these though, I am aware that even they (the "seniors") are seldom entirely responsible for all the issues. It's mostly business constraints that forces cutting of corners and that ends up jeopardizing the business in the long run.
(One of my standard end-of-interview questions is "how easy is it for me to trash the production database?" Having done this previously[1] and had a few near misses, it's not something I want to do again.)
[1] In my defence, I was young and didn't know that /tmp on Solaris was special. Not until someone rebooted the box, anyway.
I’ve had a search but can’t work out why it’s special.
"why didn't they have a hot-spare" They do! Flight spares are complete, flight-rated copies of spacecraft built for exactly this contingency: https://en.wikipedia.org/wiki/Flight_spare After launch the flight spares are used for terrain testing and troubleshooting. (The "mars yard" has flight spares for Curiosity and Perseverance https://www-robotics.jpl.nasa.gov/how-we-do-it/facilities/ma... which were used to test some wheels to destruction after Curiosity started showing some wear https://www.planetary.org/articles/08190630-curiosity-wheel-... )
The blog post lays it on a bit thick with the $500 million number and the "launch only two weeks away" given that the article itself is illustrated with a photo of the Sojourner flight spare. Spirit had the SSTB1 test rover. If he had actually blown out the entire electrical system, they could have launched it instead. Swapping out the entire vehicle right before launch would have been an awful job, but it's not flat out impossible.
I liked that other people pointed out that risk could have been eliminated by using polarized connectors (I hope they started doing this after the incident), but also made me wonder about "back-EMF" caused by solar flares. In other words, maybe all thick wires and ground/power planes should be hardened against current surges simply due to a solar event hitting mars (which may incidentally cover the case of back-powering the driver circuits).
I have been burned by this in some version of Ubuntu and have assumed it was normal behaviour ever since.
This brief moment in time has a name: an ohnosecond.
https://en.wiktionary.org/wiki/ohnosecond
But you should definitely have bought that man a beer :)
To do that, on the "connect to server" dialog, click "options". On the tab "connection properties" in the "connection" option group, check "use custom color". And I pick the reddest red there is. The bottom of the results window will have that color.
edit: my horrible foul-up was restoring a database to production. The "there is trouble" pagers were all Iridium pagers since they loved climbing mountains (where there was no cell service back then). But then that place didn't use source control, so it was disasters all the way down.
The roots cause, and someone correct me if this is not accurate, was that the x-ray tested bolts to hold it down were so expensive, that they had been "borrowed" to use on another project, and not returned, so that when the time came to flip the satellite into a horizontal position, it fell to the floor. Repairs cost $135M.
https://en.m.wikipedia.org/wiki/NOAA-19
THE LITTLE VAX THAT COULD https://userpages.umbc.edu/~rostamia/misc/vax.html
A cautionary lesson in properly checking how exactly events are connected during an incident. Easy to look at two separate signals and assume they must be causal in a particular direction, when in reality it is the other way around.