I had a product that had mostly stopped growing in usage. It was running on say fifty machines. I had put considerable effort into some memory optimizations, which was the scaling point for new hardware, so I talked Ops into bunching the active traffic load onto fewer machines. All of the active traffic load. Started hitting the memory limits (32 bits linux) and our server framework exited on malloc failure, so lots of exiting of long lived processes and loss of expensive state, delayed alerts etc.
I still think it was over-provisioned, but they told Ops to stop listening to me unless someone else agreed. Probably ran on the 50 machines till it was discontinued 10 or 15 years later, but I left so who knows.
Years ago, my employer was light on funds so we cobbled together plugs to use as a loopback when testing and identifying network jacks. It worked great. Insert the plugs in cubilces then test the open ports in the wiring closet. Worked great many times until one day we plugged it in and went to lunch. When we came back, we were told the network had slowed to a crawl and captures showed floods. This was during the days of primitive DoS via broadcast floods. Well, this flood was self induced. The loopback plug was inserted into a jack that had a connection back to the network hub. It dutifully retransmitted everything it saw back onto the network. Whoops.
Exact same scenario but it was connected to the on premise data center. I was imaging devices and figured why not use a switch and provision 4 at a time. Started seeing everything go down figured the network team was doing things. Checked my images and they had stopped, unplug a cord and everything is working. I don't think anything of it and plug it back in and went back to waiting for the images to deploy but the storm started. After about 5 hours customers started receiving their electricity after the network team found my device.
First time I got into the computer room back in the days when one unix mini-computer ran hundreds of terminals. I was asked to put a reel tape into the drive to load a tar archive into the Oracle database.
I couldn’t get the tape drive door open so I looked around and saw a key next to the door.
That didn’t open the door either. I was stood there scratching my head when the double doors burst open and half a dozen sysadmins came running in like a SWAT team.
I was a bit surprised until I glanced down and notice all the lights were off.
Yes the key that turned the power off had been left in the machine.
I worked in online advertising and pushed infinite loops that froze browsers to millions of unsuspecting victims.
On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period to an int literal.
On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it.
We deployed some machine learning demos via API and they got used in some exhibits for some powers-that-be. But we ran it in the cloud on shared infrastructure like fools, and the code was pretty bad, so it failed due to heavy load while Modi was looking at it. There was an outpouring of anger afterward. Here is Modi staring at one loading spinner. https://imgur.com/a/kLklVZV
I was 18 and had just taken over the website of a car and horse trailer dealership. I typed “;rm -fR ~” into an e-mail form to show my coworkers what would happen. On purpose. I quickly restored it (I was ready for this.) They were pretty amused. We had no concept of “on prod”. “Dev” was my local PC. Damn teenager.
Doom on a single PC was fun. But the same game on our network with all those broadcasts running through our net, which hosted a university's PCs. That was bad. And I started it several times until I noticed that "I" was the cause for network slowdown.
Best part was that a sysop entered the room and within a second knew that I was the problem and that I was playing doom. He just told me "if you wanna play games instead of studying than play doom 2. This does not broadcast"
Near miss: my first job I was working on a CRUD app for a huge bank. I was dumb and it was early in the enterprise era of software and I had built my own simple O/R tool based on codegen. Not a terrible tool all things considered and I was pretty pleased with myself.
One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database.
i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.
I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root.
we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.
in one web shop i worked with, developers who were fanaticslly using constructions "WHERE user LIKE '%$user%' "
we've had some really heated debates, but nobody would listen to me. and then one day DELETE happened where $user was empty. they spent half day restoring the database, but nobody made any conclusions and nothing changed...
And this is why we use things like database users that don't have delete permission, and row-level security so users can delete things that don't belong to them.
I have learned this from a very similar experience.
And why it’s often best to mark a record deleted and then have a reaper remove the records at a later point. But you must make sure all normal queries don’t see deleted items.
That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.
Very first job, as an intern, I was tasked with building a "free text search engine" for the product, using their api. Maybe my first week or so there, I left a script running over lunch. Turns out the internal IP addresses weren't subject to the rate limiting, and my script's queries were growing exponentially (I was sending the response back to the same endpoint which was querying with the response, and giving me back a larger response etc..) Within 20-30 minutes or so every production machine was stuck running one of my queries. And it happened on the day that the engineering team were taking the new intern out for a team lunch...
At the time I was mortified, but in hindsight the fact that I was able to do that in the first place was really the issue, not my script.
Long ago this one (close to 2000), but we were hosting some of our clients on machines in our office because it was (a lot) cheaper for our startup to do so. We had 3 rather large (for our company) clients running at the time in the server room in the office. The servers where hooked up to 2nd hand APCs so power failures went unnoticed if they were short. One Friday afternoon, we had some drinks (tgif) and a bunch of us (I was the cto....) were fooling around in one of the rooms throwing tennis balls; I threw one straight into the firealarm. This was just a glass with a button behind it: if you pressed the glass it would trigger for the entire (20 story) building. This cut the power and switched on the emergency lights. There was no fire obviously, but, for bureaucracy rules, the firebrigade had to pull up, inspect the building, we had to sign docs etc and then they switched off the alarm and on the power. Too late for our APCs: everything was down and (like said: long time ago for Linux etc) we had to run fsck and basically spent a large portion of the evening getting it all back. We moved to xs4all co location after that incident...
I was working on Nationbuilder, a horrible all-in-one master-of-none thing, back in 2016 or so, and ran so many concurrent tasks in our BE that it started affecting all of their other clients. Waking up an engineer in CA was fun
That thing is a pile of absolute dog shit. New up and coming UK political party are using it - I volunteered to help with setting up / maintaining their tech stack, realized within 5 minutes all their eggs were in that godforsaken basket and changed my mind.
Early on in my career, I worked for a secret unit of a secret government law enforcement group that handled surveillance.
Being young, full of verve, and not nearly as smart as I thought I was, I was always trying to improve things and tinker.
Knowing nothing about networking, I plugged a switch into itself. Due to the configuration it knocked the entire surveillance network offline and everyone was freaking out.
I was cool as a cucumber, because it couldn’t have been me. Must have been a coincidence right?
Right?
The sense of dread the dawned on me as the former Navy Seal turned Network Engineer (and later doctor) started sniffing around the switch I had just touched was palpable.
Luckily for me, He kept my mistake quiet and fixed it quickly.
Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`.
Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that address - or a filesystem path, as the case may be. Meaning that the command (under root) overwrote /bin/bash with 0 bytes.
Within minutes, jobs started failing, the machine couldn't be SSH'd into, but funnily enough, as /bin/bash was the login shell for all users, not even logging in via a tty through KVM worked.
Perhaps not the best story, but certainly a fun way to blow your foot off on a Monday morning :)
Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”
Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents.
In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you could pull a copy from another machine or backups.
In the wsgi case of replacing a regular file with an AF_UNIX socket, the destination would have needed to to be unlinked first - the bind() call will otherwise fail.
On Linux you can't overwrite a binary that is currently being executed (you'll -ETXTBSY when you try to open it for writing, and truncate will also fail).
I was unaware of the first part of this, and decided to find a reference.
This stack overflow answer[0] provides some detail on ETXTBUSY and the fact that writing directly writing to a running executable file will fail. (I had thought it would unlink the file like in the second case, simply because that's what I always see when recompiling something that's already running. This obviously uses the second method though).
Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.
Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reinstallation of the package that provides bash.
Where I have worked, its devs and ops who bork the systems then have to set about fixing it while management run off to face the customer. I wouldn't change place with management for love nor money.
You see a lot of mea culpa outage stories on HN, many written by management. If you have never read one you should.
The outcome of a mistake is a learnin.
Altho it is also fun to share fail stories just for the scale of the fsck up. ;)
Can It be a story I was involved in but I didn't do it?
I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves.
Our user directories were about 100GB at the time. I had sourced this special tape drive that could do that, but it was fidgety (which is not something you want in a backup drive admittedly). The backups worked, I'd say, 3/4ths of the time. I think the hardware was buggy, but the vendor could never figure it out. Also, before you lecture me, we were very constrained with finances, I couldn't just order something else.
So I graduated, and as such had to find a new admin. We interviewed two people, one was very sharp and wore black jeans and a black shirt-- it was obvious he couldn't afford a suit which would have been the correct thing to wear. The other candidate had suit, and he was punching below his weight. Over my objections, suit guy gets hired.
Friday night, my last day of employment I throw tapes into the machine and start a full L0 backup which would take all weekend to complete.
Monday morning I get a panicked phone calls from my former colleagues. "The new guy deleted the home directories!"
The suit guy literally, had in his first few hours destroyed the entire labs research. All of it. Anyways, I said something to the effect of, "Is the light on the AIT array green or amber?"
"Green."
"You're some lucky sons of bitches. I'll be down in an hour and we'll straighten it out."
As the others have correctly intuited-- its a tape backup system. We couldn't afford a proper tape library with a robot. I can't remember why, but there was a limitation of the software or SunOS that we needed to be able to get an L0 onto one tape. This thing was two Sony AIT tape machines that had a special SCSI board that made them look like one single drive to the host thus doubling the capacity. It was just enough to skate by. I didn't have much faith in differentials and didn't like to let them run more then a few days as well.
I always assumed the fault was in that SCSI board. The hand-off between tape-1 and tape-2 was what usually failed. The problem might occur 24 hours into a backup so it was difficult to get good backups. Also, I was not a full time employee (being a student), so I couldn't babysit this thing 5 days a week like a full time employee. Also, it kind of killed the performance of the system, so I had to do them at odd hours.
I am proud to say I never lost a single bit at that job.
In essence if this backup had failed months of research would have been lost (maybe two to 4 week old backup x 12 researchers).
We worked for Flanders radio and television (site was one of Flanders biggest radio stations).
The site was a angularjs Frontend with a CMS backend.
The 40x and 80x pages fetched content from the backend to show the relevant message (so editors can tweak it).
The morning they started selling tickets for Tomorrowland I deploy the frontend breaking the js fetching a non existing 5x page, looping to doing this constantly.
in a matter of seconds the servers were on fire and I was sitting sweating next to the operations people. Luckily they were very capable and were able to restore the peace quite quickly.
And also (other radio station) deleting the DB in production.
And also (on a Bank DB2) my coworker changing the AMOUNT in all rows of cash plans in stead of in 1 row (and OR and braquets and trust you know).
I wrote some software to handle charging customer's credit cards. It worked fine in the dev environment so a week later we deployed it since we hadn't been charging customers at all until that point. We would run bills once a day until all the unprocessed billing was caught up.
Well, in dev, the database was refreshed with prod data every night at midnight, so we never saw the bug in my code. I had a sign error in updating the customer's balance so instead of lowering their balance by the payment amount, my code increased their balance. Geometric growth is an amazing thing. A few days later we had calls from angry customers because we had maxed out their credit cards. Miraculously, I was not fired. In retrospect, I think that it might have been because the manager would have then had to explain why he had not made sure there was not adequate testing on something so central to the business.
The year is 2002, OS is Solaris, trying to compile some httpd add-on straight on the production server (because why not) kept giving some weird error about /etc/ld.so not being right. So junior me does:
Back in 2003 or so, I was in tech support for a company that used desktop computers running java applets to connect to a mainframe via Telnet (IBM Host-on-Demand IIRC). Most of the core business processes were handled by mainframe apps, which the company largely developed. I used to hang out in the data center with the mainframe guys who coded in COBOL all day.
On a Friday afternoon, I was working on testing deployment of an update to the java terminal client applet. Everything seemed to work fine in testing, and it was a minor update, so (idiot me) I went ahead and pushed it to the server.
Shortly after I pushed it out, the mainframe guys' phones started ringing with complaints that the mainframe was down. Then my phone started ringing. Then all of the phones started ringing.
Turns out, something I did in the update (I honestly can't remember the specifics now) reset every local users' mainframe connection information for the applet. Across the whole company. So as soon as they exited the applet, they couldn't get back in.
If you remember, how did this end up getting fixed? Did users have to re-input their connection info (meep) or were you able to re-fill everything in through some heuristic?
I was so excited to meet a legit/professional dev team the first day of my career.
I was paired with a Sr dev and sat in his cubicle so he could show me some backend update to a prod app with 20K internal users... "normally I'd run this on the dev server, but, its quick & easy so Ill just do it on prod"
...watched him crash the whole thing & struggle the rest of the day to try and bring it up. I just sat there in awe, especially as everyone came running over and the emails poured in, while the Sr Dev casually brushed it all aside. He was more interested in explaining how the mobile game Ingress worked.
The amount of "yep, I know" I have to do any time anything goes down... at least Sales usually only has one person report it to me, other teams 3 different people will tell me at the same time...
They wouldn't read my email until after they reported it... so then I'd get two emails from each person: one saying there's an issue, and another saying "oh sorry just read your email that you already know about it"!
I had to reread this to check if you're really not talking about Ingress servers, because Niantic really seems to be doing changes directly in production all the time
also I completely understand how invested one can get in explaining Ingress
3 years ago. Took on a large traffic media site running on Wordpress. The usual good stuff - outdated plugins, running on a cheap VPS box, hosting vendor washed hands off saying they can't handle the tech support and traffic.
I was working in a different company back then and I was contacted by this consultancy that specialized in Google Cloud (which was always my personal favorite anyway). I was offered a very handsome pay for just what seemed to be just 4 days worth of work. To me, it sounded really simple, like get in, migrate and get out.
After I signed the contract and everything I got to know the client was promised a $200/mo budget and a very wrong technical solution proposal by an Engineer in the consultancy from what they were paying then which is definitely in multiples of what was quoted. And to make matters more interesting, this guy just quit after realizing his mistake. And that's how I even got this project.
So, I went in, tried many cost effective combinations including various levels of caching and BAM!, the server kept going down. They had too much traffic for even something like Google's Paas to hold (it has autoscaling and all the good stuff, but it would die even before it could autoscale!). Their WP theme wasn't even the best and made tons of queries for a single page load. Their MySQL alone costed them in the 1000s. So, I put them on a custom regular compute box, slapped some partial caching on bits they didn't need and managed to bring the cost to slightly higher than what they were paying with their previous cheap hosting company. All this lead to a 4 hour downtime.
I apologized to them profusely and built them a CMS from scratch that held their traffic and more and dropped their cost to 1/4th of what their competitors are paying. Today, this client is one of my best friends. They went from "Fuck this guy" to "Can we offer you a CTO role?" :)
I make it sound like it's so easy, but it was a year long fight almost bundled with lots of humiliation for something I didn't do just to earn their trust and respect. Till date, they don't know the ex-consultant's screw up.
In retrospect, this downtime is the best thing that happened to me and helped me to understand how you handle such scenarios and what you should do and not to do. In such situations it is tempting to blame other people around you, but in the long term, it pays off if you don't and solve it yourself.
Years ago we were using NetApp Filers as storage for our database servers in a colo facility. During a planned maintenance window I installed a NetApp OS upgrade and brought everything back on line. At first it seemed fine but as soon as the database servers got some load they started dropping their connections to the NetApps and everything crashed.
Of course I blamed NetApp and called their tech support screaming for help with their OS "bug". After hours of troubleshooting we finally figured out that the NetApp OS upgrade had included a network performance optimization and it was now sending out packets fast enough to overflow the buffer on our gigabit Ethernet switch. The packet loss rate was huge. Fortunately we had a newer switch back in the office so after swapping that out and repairing some corrupt databases I was able to get production back on line. Didn't get any sleep that night though.
Many years ago (decades, in fact) as a fresh new excuse for a unix admin, I needed to hide the passwd binary so that users couldn't find it and change their local password on the terminal box (this was early ISP days). SunOS 4.1.4, as I recall.
Anyway, I hid that binary. In /etc, where they'd never think to look.
Gosh we do some dumb things, eh? LOL. That took a while to find a solution for, and no small amount of luck. The owner of the ISP walked back in the office a couple hours later and said "I heard you had some excitement?" I said, "Oh yes, it was pretty ugly for a bit." "Is it fixed now?" "Yup." "Carry on."
For sure thought my ass was fired and I'd only been on the job a month or so.
Ouch, indeed. We ended up getting lucky and found a workstation where someone had left themselves at a root prompt on another machine that had a shared NFS mount. This was before protection from this kind of attack, so we were able to create a setuid root script and run it on the main server to get root access to fix the broken passwd file.
Our next step was going to be rebooting the server. We were pretty sure that faced with a corrupt passwd file, SunOS would drop to single user mode. Never tested that theory. Glad we didn't have to, the server in question was a hack job as it was. Copied over (literally, as files) from a previous server, it wasn't even 100% in agreement with itself on its own hostname, so I always kinda wondered how it would react to any big changes.
Doesn't explain why exactly the asterisk was put in that particular position. Maybe someone felt like it was odd to lead the word with an asterisk. :shrug:
The asterisk is there to avoid writing the word out in full, like "G*d" or "f*ck". There was a time when Unix[1][2] was a trademarked name and if you used it you had to attribute it. If you wanted to refer to the general family of Unix/POSIX/SysV/BSD/etc systems, you might be tempted to write "Unix"[1][2], but to avoid the presumably-Sauron[3]-like eye of the trademark holders, you'd bowdlerize it a little.
[1] Unix is a registered trademark of AT&T Bell Labs.
[2] Unix is a registered trademark of The Open Group.
[3] Sauron is a registered trademark of Tolkien Enterprises
My bad. I somehow expected him to put it into /etc and rename it to something else. Indeed if he overwrote /etc/passwd then all hell would break loose.
This was 20 years ago now - it was my first day in a new job working for a startup.
Our startup was based in the garden office of a large house and the production server was situated in a cupboard in the same room.
The day I started was a cold January day and I’d had to cycle through flooded pathways to get to work that morning - so by the time I arrived my feet were soaked.
Once I’d settled down to a desk I asked if I could plug a heater in to dry my shoes. As we were in a garden office every socket was an extension cable so I plugged the heater in to the one under my desk.
A few minutes later I noticed that I couldn’t access the live site I’d been looking through - and others were noticing the same.
It turned out the heater I was using had popped the fuse on the socket. The extension I was using was plugged into the UPS used by the servers. So the battery had warmed my feet for a few minutes before shutting down and taking the servers down too.
And that’s how I brought production down within 3 hours of starting my first job in the web industry…
300 comments
[ 350 ms ] story [ 2878 ms ] threadI still think it was over-provisioned, but they told Ops to stop listening to me unless someone else agreed. Probably ran on the 50 machines till it was discontinued 10 or 15 years later, but I left so who knows.
I couldn’t get the tape drive door open so I looked around and saw a key next to the door.
That didn’t open the door either. I was stood there scratching my head when the double doors burst open and half a dozen sysadmins came running in like a SWAT team.
I was a bit surprised until I glanced down and notice all the lights were off.
Yes the key that turned the power off had been left in the machine.
So I crafted an .asp to do my maintenance.
Only I was calling CreateObject() in the for loop to get a new AdoDb.Connection for each of the array entries of the data.
That creaky IIS server crashed like the economy.
On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period to an int literal.
On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it.
Okay, I'll bite.
You can't just drop a war story headline like that and walk off. Please expand.
This really sounds like something that can only happen when unpredictable machine learning models are involved^^
One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database.
Not a fun drive in.
Yes, we drove in in those days.
how
I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.
we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.
Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.
we've had some really heated debates, but nobody would listen to me. and then one day DELETE happened where $user was empty. they spent half day restoring the database, but nobody made any conclusions and nothing changed...
aaand it’s gone
I have learned this from a very similar experience.
At the time I was mortified, but in hindsight the fact that I was able to do that in the first place was really the issue, not my script.
Right?
The sense of dread the dawned on me as the former Navy Seal turned Network Engineer (and later doctor) started sniffing around the switch I had just touched was palpable. Luckily for me, He kept my mistake quiet and fixed it quickly.
Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that address - or a filesystem path, as the case may be. Meaning that the command (under root) overwrote /bin/bash with 0 bytes.
Within minutes, jobs started failing, the machine couldn't be SSH'd into, but funnily enough, as /bin/bash was the login shell for all users, not even logging in via a tty through KVM worked.
Perhaps not the best story, but certainly a fun way to blow your foot off on a Monday morning :)
In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you could pull a copy from another machine or backups.
This stack overflow answer[0] provides some detail on ETXTBUSY and the fact that writing directly writing to a running executable file will fail. (I had thought it would unlink the file like in the second case, simply because that's what I always see when recompiling something that's already running. This obviously uses the second method though).
[0] https://unix.stackexchange.com/questions/187931/modifying-bi...
But thanks, I've just added another technique to my toolbox.
> When a user successfully logs in, sshd does the following:
> ...
> 9. Runs user's shell or command. All commands are run under the user's login shell as specified in the system password database.
However I’ve only ever heard stories where management lays the blame.
You see a lot of mea culpa outage stories on HN, many written by management. If you have never read one you should. The outcome of a mistake is a learnin.
Altho it is also fun to share fail stories just for the scale of the fsck up. ;)
One ISP router in production with 20k active connections... one "backup" router fresh from the box.
My job was to backup the production firmware and flash the config to the spare box.
The opposite happened and the customer support telephones lit up like a Christmas tree.
I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves.
Our user directories were about 100GB at the time. I had sourced this special tape drive that could do that, but it was fidgety (which is not something you want in a backup drive admittedly). The backups worked, I'd say, 3/4ths of the time. I think the hardware was buggy, but the vendor could never figure it out. Also, before you lecture me, we were very constrained with finances, I couldn't just order something else.
So I graduated, and as such had to find a new admin. We interviewed two people, one was very sharp and wore black jeans and a black shirt-- it was obvious he couldn't afford a suit which would have been the correct thing to wear. The other candidate had suit, and he was punching below his weight. Over my objections, suit guy gets hired.
Friday night, my last day of employment I throw tapes into the machine and start a full L0 backup which would take all weekend to complete.
Monday morning I get a panicked phone calls from my former colleagues. "The new guy deleted the home directories!"
The suit guy literally, had in his first few hours destroyed the entire labs research. All of it. Anyways, I said something to the effect of, "Is the light on the AIT array green or amber?"
"Green."
"You're some lucky sons of bitches. I'll be down in an hour and we'll straighten it out."
By all reports, he eventually became a well liked and good admin. I just don't think he knew that much Unix when he started.
> "Is the light on the AIT array green or amber?"
Can you explain this? What is an AIT array?
Green = backup worked.
I always assumed the fault was in that SCSI board. The hand-off between tape-1 and tape-2 was what usually failed. The problem might occur 24 hours into a backup so it was difficult to get good backups. Also, I was not a full time employee (being a student), so I couldn't babysit this thing 5 days a week like a full time employee. Also, it kind of killed the performance of the system, so I had to do them at odd hours.
I am proud to say I never lost a single bit at that job.
In essence if this backup had failed months of research would have been lost (maybe two to 4 week old backup x 12 researchers).
Anyways, thank you for reading my silly story!
Does self DDOS count?
We worked for Flanders radio and television (site was one of Flanders biggest radio stations). The site was a angularjs Frontend with a CMS backend.
The 40x and 80x pages fetched content from the backend to show the relevant message (so editors can tweak it). The morning they started selling tickets for Tomorrowland I deploy the frontend breaking the js fetching a non existing 5x page, looping to doing this constantly. in a matter of seconds the servers were on fire and I was sitting sweating next to the operations people. Luckily they were very capable and were able to restore the peace quite quickly.
And also (other radio station) deleting the DB in production. And also (on a Bank DB2) my coworker changing the AMOUNT in all rows of cash plans in stead of in 1 row (and OR and braquets and trust you know).
Well, in dev, the database was refreshed with prod data every night at midnight, so we never saw the bug in my code. I had a sign error in updating the customer's balance so instead of lowering their balance by the payment amount, my code increased their balance. Geometric growth is an amazing thing. A few days later we had calls from angry customers because we had maxed out their credit cards. Miraculously, I was not fired. In retrospect, I think that it might have been because the manager would have then had to explain why he had not made sure there was not adequate testing on something so central to the business.
$ rm /etc/ld.so*
Back in 2003 or so, I was in tech support for a company that used desktop computers running java applets to connect to a mainframe via Telnet (IBM Host-on-Demand IIRC). Most of the core business processes were handled by mainframe apps, which the company largely developed. I used to hang out in the data center with the mainframe guys who coded in COBOL all day.
On a Friday afternoon, I was working on testing deployment of an update to the java terminal client applet. Everything seemed to work fine in testing, and it was a minor update, so (idiot me) I went ahead and pushed it to the server.
Shortly after I pushed it out, the mainframe guys' phones started ringing with complaints that the mainframe was down. Then my phone started ringing. Then all of the phones started ringing.
Turns out, something I did in the update (I honestly can't remember the specifics now) reset every local users' mainframe connection information for the applet. Across the whole company. So as soon as they exited the applet, they couldn't get back in.
That was a fun weekend.
If you remember, how did this end up getting fixed? Did users have to re-input their connection info (meep) or were you able to re-fill everything in through some heuristic?
I was so excited to meet a legit/professional dev team the first day of my career.
I was paired with a Sr dev and sat in his cubicle so he could show me some backend update to a prod app with 20K internal users... "normally I'd run this on the dev server, but, its quick & easy so Ill just do it on prod"
...watched him crash the whole thing & struggle the rest of the day to try and bring it up. I just sat there in awe, especially as everyone came running over and the emails poured in, while the Sr Dev casually brushed it all aside. He was more interested in explaining how the mobile game Ingress worked.
also I completely understand how invested one can get in explaining Ingress
I was working in a different company back then and I was contacted by this consultancy that specialized in Google Cloud (which was always my personal favorite anyway). I was offered a very handsome pay for just what seemed to be just 4 days worth of work. To me, it sounded really simple, like get in, migrate and get out.
After I signed the contract and everything I got to know the client was promised a $200/mo budget and a very wrong technical solution proposal by an Engineer in the consultancy from what they were paying then which is definitely in multiples of what was quoted. And to make matters more interesting, this guy just quit after realizing his mistake. And that's how I even got this project.
So, I went in, tried many cost effective combinations including various levels of caching and BAM!, the server kept going down. They had too much traffic for even something like Google's Paas to hold (it has autoscaling and all the good stuff, but it would die even before it could autoscale!). Their WP theme wasn't even the best and made tons of queries for a single page load. Their MySQL alone costed them in the 1000s. So, I put them on a custom regular compute box, slapped some partial caching on bits they didn't need and managed to bring the cost to slightly higher than what they were paying with their previous cheap hosting company. All this lead to a 4 hour downtime.
I apologized to them profusely and built them a CMS from scratch that held their traffic and more and dropped their cost to 1/4th of what their competitors are paying. Today, this client is one of my best friends. They went from "Fuck this guy" to "Can we offer you a CTO role?" :)
I make it sound like it's so easy, but it was a year long fight almost bundled with lots of humiliation for something I didn't do just to earn their trust and respect. Till date, they don't know the ex-consultant's screw up.
In retrospect, this downtime is the best thing that happened to me and helped me to understand how you handle such scenarios and what you should do and not to do. In such situations it is tempting to blame other people around you, but in the long term, it pays off if you don't and solve it yourself.
Of course I blamed NetApp and called their tech support screaming for help with their OS "bug". After hours of troubleshooting we finally figured out that the NetApp OS upgrade had included a network performance optimization and it was now sending out packets fast enough to overflow the buffer on our gigabit Ethernet switch. The packet loss rate was huge. Fortunately we had a newer switch back in the office so after swapping that out and repairing some corrupt databases I was able to get production back on line. Didn't get any sleep that night though.
Anyway, I hid that binary. In /etc, where they'd never think to look.
Gosh we do some dumb things, eh? LOL. That took a while to find a solution for, and no small amount of luck. The owner of the ISP walked back in the office a couple hours later and said "I heard you had some excitement?" I said, "Oh yes, it was pretty ugly for a bit." "Is it fixed now?" "Yup." "Carry on."
For sure thought my ass was fired and I'd only been on the job a month or so.
Ouch, indeed. We ended up getting lucky and found a workstation where someone had left themselves at a root prompt on another machine that had a shared NFS mount. This was before protection from this kind of attack, so we were able to create a setuid root script and run it on the main server to get root access to fix the broken passwd file.
Our next step was going to be rebooting the server. We were pretty sure that faced with a corrupt passwd file, SunOS would drop to single user mode. Never tested that theory. Glad we didn't have to, the server in question was a hack job as it was. Copied over (literally, as files) from a previous server, it wasn't even 100% in agreement with itself on its own hostname, so I always kinda wondered how it would react to any big changes.
I've seen it written *nix to grab Linux and Unix.
https://unix.stackexchange.com/questions/2342/why-is-there-a...
Doesn't explain why exactly the asterisk was put in that particular position. Maybe someone felt like it was odd to lead the word with an asterisk. :shrug:
[1] Unix is a registered trademark of AT&T Bell Labs.
[2] Unix is a registered trademark of The Open Group.
[3] Sauron is a registered trademark of Tolkien Enterprises
Our startup was based in the garden office of a large house and the production server was situated in a cupboard in the same room.
The day I started was a cold January day and I’d had to cycle through flooded pathways to get to work that morning - so by the time I arrived my feet were soaked.
Once I’d settled down to a desk I asked if I could plug a heater in to dry my shoes. As we were in a garden office every socket was an extension cable so I plugged the heater in to the one under my desk.
A few minutes later I noticed that I couldn’t access the live site I’d been looking through - and others were noticing the same.
It turned out the heater I was using had popped the fuse on the socket. The extension I was using was plugged into the UPS used by the servers. So the battery had warmed my feet for a few minutes before shutting down and taking the servers down too.
And that’s how I brought production down within 3 hours of starting my first job in the web industry…