Didn't they only say that their backups are fucked, but not that they didn't try to make them? I've definitely run into situations where the business is only willing to pay for the most rudimentary conception of a backup strategy, and then it turned out to have been subject to corruption.
Can't find it now, but I've often seen a phrase to the effect of "If you don't regularly test that you can restore from backups, then you don't actually have backups", and it seems to ring quite true here.
It's up there with "unless you test power supply failover, don't rely on that generator to save you".
I know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
I contracted at one place for a bit where they shut everything down in each of their data centres once a year and power everything back up.
When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
Yup. Have the problem when all the right people are awake and on-site to handle it.
I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment.
If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
They explicitly mention that they have backups, but that the "backups are fucked", whatever that means.
Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.
This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains which should at least bring their internal services back online and allow them to proceed further.
I always wrap the actual cp/mv/rm etc in echo first before actually running it e.g. running echo "mv $src i" first would have immediately highlighted the problem visually.
It is also helpful to have at least two tape drives, and check the backup tapes against both.
Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.
Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.
A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.
> > All domain information was wiped out and records became null.
> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.
To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath. A high enough rate that even if they are doing hours elsewhere some will be tempted to sign in after their work hours to help out.
A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.
To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath
Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.
We don’t know that the records are still in the database. This could be a matter of ON DELETE CASCADE followed by a delete of what looked like innocent data and then boom it’s all gone. Then the integration updates bind which helpfully erases all records and then once there are no domain names to talk to everything is down.
I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.
Yeah, the poster didn't make totally clear whether all of their business records are totally gone, but it does sound like that's the case. The ERP was very likely the source of truth for everything- including employee information that the poster says is gone (can't send SMS to remote employees).
Nothing in his post suggests the ERP's DB is gone, merely that the mechanism which populates DNS records from the ERP has broken down, which probably has ripple effects on everything else as their internal DNS is broken and services can't talk to each other.
But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
> Skimmed through the MySQL and Postgres databases, no signs of anything domain-related. I'm betting it's all stored in the custom integrated [and encrypted] KV database..
It doesn't, but in my experience, the ciphers and key sizes commonly used were often not future-proof :). This rests on the assumption that nobody updated the keys/encryption components in 20 years - which I think is quite possible with such internal enterprise deployment.
"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."
Elsewhere, they say that two years ago, they found they were unable to restore from backups...
That isn't the failure of the business; backup failure is a screwup of the IT, which I think is not that uncommon.
But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.
I'd say if it was such a house of cards, my best bet is that this security chain is full of holes. Maybe the ERP used a fixed password, which can be guessed/bruteforced.
it doesn't sound like modern software. At my current employer (to remain unnamed) when I started there were a bunch of MS-SQL servers with an SA password of <blank> (i.e. no password). It took a long time to fix that because we had hundreds of local installs of the client app that had SA/<blank> in the app config file (yes, they were setup to access as system administrator), so fixing it meant reconfiguring hundreds of local installs. (at the time we didn't have a mechanism for automating it.) We still have "enterprise" software that does no password rules, and would happily allow a blank password.
I agree. My first reaction was, "I think I could fix this". Worst case this may be be proceeded by a forensic recovery of the data. This doesn't seem impossible at all unless there is some other factor at play like ransomware/physical damage.
I did something similar for a customer of ours who was a major UK cinema chain - they had had a falling out with some small company that had built an online booking system.
Highlights:
1 Bookings where made at individual cinemas
2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels)
3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats)
4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am
% And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for
I'd love to read a detailed write-up about this. Why packet radio - what was the real-time requirement and why couldn't it be fulfilled by an internet connection (I understand internet was primitive back then, but ISDN should've been able to give you enough bandwidth?).
We never understood why the where using packet radio as you say in 1999 ip via ISDN would have been fine.
Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system.
It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.
What exactly happens if you break a three month resignation period? I don’t advocate doing this in generally but as a hypothetical exercise if this person just stopped showing up for work, what exactly could the company do?
while I agree that this is common in Europe, if the company ceases to exist, what good is the 3 months notice for the employee? If he/she has any options, I'd leave asap and not worry about amicable terms with a soon to be non-existent entity.
Fair point - in that case I suspect that would be highly dependent on whether there are any assets remaining in the company and how the local legal system treats employees as creditors.
Mind you - if the company is going down the tubes they are highly unlikely to enforce any job contracts.
What would be the practical consequences to the employee if they break that notice period in UK, i.e. stop coming to work (or, especially now with Covid, simply stop doing the remote work) sooner than these 3 months have passed?
I suspect you could be sued by your employer. Mind you I've never heard of anyone being sued in that way - can't imagine it would be worth it for the employer unless they could demonstrate a significant financial loss.
I investigated this once (while I was sitting around doing not much during a 3 month notice period, funnily enough). It's as you say, the breach of a contract is something where the employer would have to sue you for damages, and if you're sitting around not doing much, it's going to be hard to argue that losing you has caused them harm, much less enough to make it worth suing over.
I didn't break the contract anyway out of some combination of feeling like it was breaking a promise and the worry that Cambridge is a small town.
I guess it just seemed so implausible to me. As in, what happens if 2 weeks or 4 months in you just decide that the job is sufficiently ill-suited? Is there typically a trial period of equal time that precedes such an agreement?
Yes, if both parties agree, you can leave at once, otherwise employer can sue you for damages or lost profits (if damages are small, they don't bother) and you don't have unemployment benefits.
Perfectly common in large parts of Europe. I also have a 3 months notice period, which is pretty much the standard here in Sweden for tech workers or other highly-qualified occupations. For very senior positions, 6 months isn't unheard of.
It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.
How does getting a new job work? You tell the new company that you have to give 3 months notice? That's some long term planning on the part of the hiring company.
Yes, but that's nothing unusual when the entire industry expects that. The last time I changed jobs, I signed a contract at the end of March, to start in August. That was the three month notice period plus most people are on vacation in July, so my new company would rather have me in August.
There are contractor agencies filling the niche of providing employees on short notice (and usually for a short period), but generally having people begin 3 months later is just something everyone expects and factors into their processes.
The average length of time people spend at a single job is also longer than in the US.
Can't speak for Sweden, but we do pretty much exactly that with our India team. They have to give notice well in advance, and so when we hire a new team member we have to do so with the understanding that it will probably be a few months before they join the team.
Yes exactly. Sometimes they ask you if you can negotiate with your employer to shorten this period, which is often the case.
Of course, they would rather it was shorter, but hiring is already a multi-months long process in most cases, so a few more months is inconvenient, but not the end of the world. If you need someone now, you hire contractors.
That was my thought. It's typical in Canada that an employer will pay you out for the rest if your two weeks or whatever, and in the U.S I don't know, but have it really favours the business. I'm still at abloss as to how this works in what is apparently most of Europe. Notice is more like 2 or 3 weeks.
You would have to prove cause to fire without notice.
Also, depending on the country, firing/layoffs can be difficult without cause (and often even with cause) (e.g. you can't layoff someone at will, you might need to prove that you have an economic downturn before downsizing). But this varies way more between countries as far as I know.
I guess it could be a problem in the US for COBRA (health insurance continuation)? apart from that, if the company is dead I guess there's little they can take away that you aren't going to lose anyway.
I just checked for curiosity. The employer can sue for damages they'll have to show.
But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination.
Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of missed payments (the state will then try to collect from the employer). So it's relatively safe for an employee to stick with a sinking company. Just get that CV polished and start warming up the network.
It seems to me that they can sue for 3 months salary. Or the amount of days employee does not work. However in this situation I would imagine they have more immediate concerns so might not get around even to that.
Well, for one, your work references probably won't be the best.
Even if you're not responsible for the problem itself, leaving your employer in the ditch, when there's a major outage - well, it just doesn't look good (barring some scenario where your employer is completely unreasonable - like not paying you).
It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
But if they get in a situation where management is yelling at them and demanding they work 20 hours a day until this is fixed somehow and they have no idea how to. Management won't bring in consultants, etc. That ship has already sailed.
> It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
The firefighter who recognizes when the building is about to collapse and gets out of there is a good firefighter.
In America absolutely nothing happens. He might have some type of agreement where he gets a severance payment upon resignation, that would be forfeited. If I was him, assuming upper management starts yelling or whatever, I would just leave. Stress can kill you, and stressing over a job is absolutely never worth it because if you die today , they'll post a req for your position by the end of business.
It looks like they could probably sort something out, wouldn't be a bad idea to call up any of the old vets who retired to see if they could come down to the office and contract for a week or so.
Not sure what country the person works in. In addition it could be a situation of a small town / tech backwater and word getting around, regardless of the legal question, so quitting out of contract could get you blacklisted.
Ultimately though a person could not show up for any reason at all (e.g. hit by the proverbial bus) and you need to plan for that. Sounds like a company that just flies by the seat of its pants so they probably have that 3-month clause for that reason.
In terms of personal reputation, once you're in charge of backups at a company that went bankrupt due to the failure of their backups, I'm not sure it makes much difference to your reputation whether you diligently work your notice period or not...
Depends on the contract and jurisdiction. I had one contract that stipulated a fixed penalty, which made any attempt of ignoring notice periods a calculated risk (the fine wasn't that steep). In any other case, in Germany, I would expect a law suite, incurring costs, stress and time, followed by some kind of agreement between the two parties. As an employee, you don't necessarily have the time nor the money (there's insurance for legal disputes like that so).
In that particular case, my experience tells me the guy won't get any credit for finding a solution, I would consider sick leave. Burnout, whatever. Doesn't have any monetary consequences in Europe (at least no harsh ones), and Germany covers up to 6 weeks of salary.
Any way, you can forget about getting goods references. Not that would get them anyway, with an employer like that.
At least in Germany, while it's in principle possible to sue the employee for damages, it is very difficult to succeed with that if it isn't downright sabotage. You can't force the employee to work, it quickly goes into negligence on your part if you can't handle an employee going missing, ..., so you'll probably not reach much more than the salary - and for that just terminating the contract early would be easier, either in agreement with the employee or one-sided after they stopped showing up.
I’ve seen a few data disasters in my time but I think this takes the cake. It’s a good reminder of why you just need to prioritise the sort of work that has no immediate payoff.
It’s easy to imagine there are a lot of these type of time bombs out there, particularly in really old legacy systems (> 20 years for example). I was at one place where even building the application for deployment led to a one week outage when some core people were laid off.
Interestingly, some of these legacy systems run on things like IBM i (AS/400) which have an easy backup story and vendor support for recovery. I can have a new i machine receiving a backup and deployed fairly quickly (day or two) and I'm out in a rural area.
Up until 5 years ago, my employer's primary system (government, property tax assessment, billing, and collection) was from a company that had gone belly-up in the early 90's. We fortunately had the source, but it ran on a VERY legacy database called Unidata because it had originally be developed on the Pick operating system. Unidata was/is a Multivalue Database with a built in BASIC programming language. By the time 2014 rolled around there was one programmer in the organization that understood the system, and he was responsible for all code changes. I half-joked with management that we needed an insurance policy on his life/ability to work, because if we lost him, it would put us in a very difficult position, and that software accounted for 90% of our revenue (as well as a lot of the revenue that went to all the local towns and school districts for their operations.) That employee was also nearing retirement (he is retired now.) Fortunately we were able to successfully migrate off of that application, though we keep a minimal license pool to the old system so people can validate pre-migration data against post migration data.
Thanks. Reddit is such a good example of how "progress" hasn't really been progress in web development. The new front-end framework site is so much slower and more buggy.
They want people using their app, for some reason I don't understand. Maybe to make ad-blocking harder by getting people off computers where it's easy (most mobile browsers don't support plugins, so it can't be about them).
This surprised me. If ever there was a use case for an organizational structure, this is surely it.. just have every manager reach out to their direct reports? Following the tree communication should happen at something like nlogn speed
Division of responsibility can make this sort of thing very difficult to know.
If you aren’t running drills then transparency may not be in everybody’s self interest and someone somewhere is covering up for the fact that they only half know what they’re doing.
I keep wondering if there’s an underwriter angle here where you offer insurance and audit the company to set the premiums based on how broken their IT situation is.
How people handle this level of disaster is always very revealing. In every case I've ever experienced someone kept their head and a solution was found. Usually there was reputational damage but surprisingly most times the actual business bounced back. But it was always the actions of someone in particular in those first hours that made the difference.
Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).
Thinking you're fucked is unhelpful, but it's pretty normal to go full Alien Hudson in a dire situation and need someone else to assure you that things will be okay.
If you have been that person before, it can be fun and exciting to think up solutions and implement them under pressure. Definitely a real test in your ability to concentrate. But don't expect the credit you deserve for it, and don't even expect others to even bother to put in any effort towards fixing things.
Also, make sure everyone knows you are going on vacation once you resolve things.
These types of situations always wind up being the most fun I have at work and leave me feeling most fulfilled. I wish I could do that all day instead of going to standups.
I've thought about that before. You get the call. "It's 30 minutes away. I'll be there in 10"
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
I've worked such a job at a large enterprise. It really does feel like firefighting (minus all the smoke inhalation and physical strain and death risk).
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
I spoke with a CISO of a large F500, his biggest gripe was that he has a team of 30 that can barely keep their heads above water, let alone respond to incidents
I don't know of a company that does this sort of work, but I know of some technologies that experts in receive these types of calls. The one that comes to mind is an ERP-focused database system. It's called "Progress" by "OpenEdge". IMHO, it's awful, but this has no hindered adoption in the slightest. I wrote Progress/4gl (their query language) often enough in a prior position to have it on my resume. Every 2 or 3 months, I'll get an email/call, asking if I could be available for short-term contracting upwards of $200/hr for Progress emergencies. I have declined all of these, because I found it soul crushing to work with in the past. However, if you could enjoy that sort of thing, that's one example of a very lucrative field to dabble in.
I've gotten a few pings along similar lines for my HighJump experience (warehouse management system, for those not acquainted). And much like OpenEdge, it's pretty... rough around the edges (at its core it's basically a runtime for a VM that's programmed with a "language" (if you can call it that) driven entirely by conditional GOTOs developed entirely in a half-baked SQL-backed IDE called "Architect"; this is paired with a DB schema from hell for all data storage, and it's filled to the brim with sprocs because even fucking T-SQL is more ergonomic than anything doable in Architect). And yet, it was actually kind of fun (in the twisted, Dwarf-Fortress-esque sense of the word) to hack on that system and abuse the hell out of it.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
This is my ideal job, I think. Charge a big rate to be the consultant you call when everything dies from ransomware or OPs problem or whatever, I calmly fix it to normal standards (as possible), take three weeks off after.
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
CEO: We are doing project X. We have engaged so and so to do it. You are to support this.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
Damn that's so true. We hired a network expert to figure latency somewhere between our app and the end user. Spent a month asking autorisations for each gdmn net boxes which were owned by several organizations which were working in the same building but not able to cooperate because they were not paid by the same institution.
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
I would say the analysis is 90-99% of the work, based on my experience. Especially the more jacked up systems where cheapest devs found work on the codebase and it's a mess. 90% only comes in when it's so jacked up it requires a lot of refactoring.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
I remember stuff like that, once the COO called the entire company to the common room to present me a bottle of champagne for my efforts (another time I was given a fancy bottle of whiskey that some marketing guys drank 3/4 of before I ever got around to it). I rather not work any overtime instead.
One thing the situation does show is a lack of proper care for their most critical assets. When the dust settles, there should be hard questions for several people. One being-- when was the last time backups were tested..
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
> Management said we don't have enough money to fix any of this.
It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost.
I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner.
An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years.
I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
> I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
Deletion requests affect monthly backups as well, although it's sort of up in the air how many months the EU will accept as a reasonable business practice for keeping should-be-deleted user data.
The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
Afford in the strict sense of have enough money, yes of course. But backup is not the same as archive even when the backup exists. It is usually not easily searchable, it is hard to be sure which version is authoritative and so on.
I have no reason to doubt that this is factual. However, I have serious doubt that management will acknowledge they made that decision.
Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.
Looking through the comments, it appears concerns were raised for years and management refused to allow the necessary changes to save money.
While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
Reading comments https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... it sounds like he ran into a kernel panic, maybe because cosmic radiation messed with their full disk encryption because they went cheap on the ecc ram, and then he got discouraged, he probably asked management for the resources to rewrite the whole system from scratch, they said no, and then he sat on his hands collecting paychecks for two years until the thing he knew would happen finally happened. How else can you explain an MSP not having a copy of their zone file? That's their core business. It's also no secret in the MSP world how frequently these DNS servers get hacked. I read about it all the time. Especially on Reddit where you can just sense the pervasive fear. So you'd think that'd give a sysadmin a moment's pause to think, OK, have I scp'd my zone file in case that happens to me?
The first manager I ever respected was this goofball who worked at a fast food restaurant. I don't think the other managers took him seriously. He didn't really take himself seriously either (told the worst jokes), but when there was a rush he was the one you wanted running the kitchen. The other 2-3 people were kind of assholes under stress (they were assholes all the time if I'm honest). He was a machine, and it didn't hurt that he good at predicting orders before they were even made. We always made too much food in these situations but more of his would sell and the wait times were lower.
In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
I've heard this is a kind of symptom of people who have ADHD, some of them have grace under pressure.
I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD.
Well, it looks like someone better track down the people who knew how it worked and offer some serious cash.
The only actual backup is one that has been tested. With the cost of hardware, buy 2 or have an agreement with your vendor to have a machine ready to receive a backup.
This reminds me of when the Danish mega-corp Maersk was attacked by ransomware and had all of their computers locked and decrypted with no access.
All computers except one, which due to a power failure had been offline the whole time.
This computer had the critical DNS information needed to restore the network.
Only problem: it was in Africa, in a country that required Visa which would take weeks to apply for and receive. So someone from the African office had to bring the hard drive to an airport that someone from the London office could travel to and pick up the critical hard drive, then bring it on a plane back to London.
Sounds very interesting, but the story seems to be behind a paywall. I found some other articles[0][1] related to the incident, however none of them seem to mention the flying of a hard drive across continents.
Not just DNS, it was their entire Active Directory Global Catalog. If it weren't for that server, they would have needed to re-create their entire user database, all of their endpoint & server management policies, and it would have been almost impossible to safely restore their Exchange email database. They got very very lucky.
These kind of posts are awesome tabletop scenarios. (and fuel for https://twitter.com/badthingsdaily ) Coming up with potential fixes is a good exercise.
Things I would do that I haven't seen mentioned yet:
- Grab images from the relevant servers to have current backups before messing with anything.
- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.
- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.
- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)
- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.
- See if the customer domains can be collected from billing data / some email store of notifications.
(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )
Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?
It would work if they knew what the DNS entries were but seems they lost all DNS records and don't even know what records they have/had, so spinning up a new DNS server won't help as they don't know what it should serve.
> All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in the ERP which is now busted
That's part of what bothers me about this whole story--it says they were running BIND, which configs on text files for goodness sake. These critical records were tiny and could fit anywhere & transfer in the blink of an eye. That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling.
> That such a simple thing is buried under & dependent upon an entire complex and untested/maintained DR plan is mind-boggling
I find this to pretty common in many setups where the engineers don't focus on simplicity and removing layers of abstraction. If the workforce is young and inexperienced, over-engineering tends to happen everywhere and you end with situations like this.
I have seen worse in my years in the industry, that's for sure.
> We tried really hard to make management aware and actually succeeded in that, only for management to follow up and say we don't have enough money to fix it, and they understood the possible outcomes.
I actually don't see any issue that isn't simply a matter of money.
Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company.
Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
> customers will jump ship and the company will probably sink anyway.
Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is.
If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident.
The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.
There's the possibility that those employees are happily retired and don't care. Or they're just not available in many ways.
I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.
I bet if the hospital is to pay Bob/Charlie $1M for 6 months, post tax to fix it they would gladly do it. Note: this does not mean "We will pay you up to a million dollars, Net 90, after an invoice is approved and signed off by fifteen people". This means "We will wire a deposit of $250,000 today. The rest will be wired every month in 24 hours after an invoice is issued"
The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want. Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
That's called priorities. I'm sure as every public hospital it pays a lot money in salaries to those protecting and growing their personal fiefdoms within the hospital.
>For $1M they can get the new version of the same system installed and likely get a step by step walkthrough of the whole thing by any number of new engineers they want.
A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data.
All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems.
>Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
Hospital? He said they're an MSP on reddit:
>Yeah we're an awful MSP. I've been trying to find a new job for quite some time.
I have personally turned down an offer very much like this. I could have paid my (at the time) salary and benefits with about 2 days work a week with the option to bill as much time over that as I wanted as long as I agreed to work at least 2 days a week on thier top issues.
The ERP system was a nightmare, though, and the company was difficult to work with (key, long term employees). It was a system I inherited, built out of garbage, that I tried to improve. Before I left, I found a very experienced contractor to take over, on-boarded him, and stayed as a support resource for a couple months. Six months later, he's retired and I'm getting panicked phone calls. I valued (and still do) my current job far more, even though I would probably have done better financially in the short term with the alternative.
A few years ago, I heard they migrated to a new ERP at long last.
This modern-day feudalism with corporations is really the root of the problem. Managers "patch" broken IT systems with lots of manual labor to actively avoid pointing out how some other manager's system is borked. They doom the company to spend hundreds or even thousands of man hours a year to work around these processes, refusing to allow it to be fixed, keeping the "power" and influence of the broken system with some other manager. They do this hoping to curry favor for the sake of their career prospects.
After 25 years, I'm an expert at making small, efficient software tools to fix local IT workflows. God knows, in the Fortune 250's I've worked at, there are endless opportunities. And, while I've written many solutions, I've seen this political game of thrones thwart my efforts many times. This is how you get the kind of 20-year legacy mess described in the original post: People sacrificing long-term success of the company for short-term success in their careers.
I guess there are 2 kinds of people. Those that think this is just how the world works, and find this to be a perfectly-cromulent way of getting through it, and those, like me, who find it odious. Luckily, I now work for people like me. We do what we can.
This applies to similar problems in government as well. Some people care, some just kick the can to whoever is unfortunate enough to inherit the problem in 5-10 years. If you get a few of the latter people in political power, often the best thought out solutions are offered many times and denied because of internal political wrangling.
I knew someone who was a specialist in doing calibrations on some very specific industrial measurement equipment, much of which was out-of-date but massively expensive to replace.
He had retired, but took another 12 or 18 month contract flying around the country servicing these things. Apparently they asked, and not wanting to do it he quoted what he thought was an insane "go away and leave me alone" fee, but then they said yes. (I have no idea what that fee was or what multiplier he applied, though it definitely brings to mind patio11's standard advice [1]).
There's a pretty big gap between "happily retired with enough money to not be worried" and having zero interest in ever getting more money regardless of the rate. After a decade a happily retired person who left amicably and didn't hate their job might even enjoy a short-term contract if you can isolate them from all the petty annoyances that come with a normal job. Charlie leaving the country shouldn't be a dealbreaker either unless you literally have tried to track them down and couldn't.
If the long-gone ex-employees hated their employer and would be happy to see if go out of business then you're in quite a bit more trouble.
Not really related to the subject of the article, but your comment reminds me of employers in Romania complaining they can't find people to hire, while they offer small salaries and people just leave the country for better paying jobs abroad. The solution to their problem is really simple: make better offers.
This kinda sounds like saying "why are poor countries poor? Can't they just pay everyone more and then they won't be poor?" I get what you are saying but I don't think this will work if the problem is country-wide. You can't just magically increase everyone's wages in a country.
1. The problem isn't country wide. 2. Don't have to increase everyone's - just those with a "shortage". Companies want a market economy - except when it comes to salaries.
> The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
This. There is no amount of money anyone sane can be given to stay in Romania. Most recently i’ve read they (hospitals) were tying covid patients to their beds.
Couldn't help but comment on this as some of things i discovered are shocking for an eu member country (or any country for that matter), such as EU leading human trafficking, police corruption, collapsing health care, and the list goes on.
But sure, the simple solution is to give people money, not the constant abuse taking place at work and outside that makes people want to flee that place.
That's every poor country in existence. Not everyone has the same mindset as you, however. Provided with basic incentives, many, if not most, would not want to leave family, friends, common culture, etc, just for a wealthy life - they just want basic stability.
Companies paying more (offering more at least) would begin to slow this domino effect, which raises standards of living and provides more funding for the government to provide services and ultimately pressure for increases in quality of life by the voting public. Everyone wins.
Ultimately not everyone can emigrate - it's an impossibility. The brain drain will continue to be real, so it's incumbent upon these companies hiring to raise their wages and retain talent.
That was my point. The simplest solutions are sometimes the most effective, as well. Incentivizing brain power to remain the country via pay increases seems like the most basic approach one can take, considering the GP was talking about Romanian studios complaining they can't find/hold talent and they are competing with the rest of the EU.
Step in as a multi-national company, open a software engineering office, pay well over "local market value" and you get good software engineers, at a lower price than moving them elsewhere. They, in turn, get a better quality of life, don't have to emigrate (which for some is a bonus, and for some the opposite (a malus?)) and relatively speaking have more spending power than they would have had, if they moved.
It is really not as easy as go "this is the exchange rate" and simply use that to compare pay in different countries.
The exact same thing is happening in Australia in regards to farmers and fruit picking labour. With international borders shut they are unable to exploit foreign workers and complain that young adults are refusing to relocate for $5 an hour hard labour jobs.
Farming is low margin. Farmers don’t get to set the price of their products, and they aren’t handed billions in capital to burn through or 1000x multiples on their profit in the public markets. When most businesses say they can’t hire someone and the reason is because they don’t pay enough, the underlying cause of their low pay is usually because that’s what the company can afford.
Sure, farmers could pay $10/hr, but unless your Doritos were $10 a bag, they’d lose money.
Being paid $5/hour farming in Australia vs $0.50/hour assembling shoes in Malaysia. Wow, such exploitation.
"Exploitation" is a relative term, some people would rather be exploited in the countryside of a first-world nation rather than the countryside of their homeland.
Which is more exploitative, working for $5/hour in agriculture in a wealthy country, 50¢/hour in an urban factory in a poor country, or $1/day in a rural peasant village somewhere or having no work at all?
> Being paid $5/hour farming in Australia vs $0.50/hour assembling shoes in Malaysia. Wow, such exploitation.
The Australian national minimum wage is $19.84/hr.
Getting paid about a quarter of minimum wage is very clearly exploitation. There may be other benefits for actually working in Australia, but that's not really relevant here when you're trying to attract native Australians to work for you.
In particular, when the wage is low enough you have slavery in all but name. People working for your shitty pay staying in your shitty dormitory eating your shitty food because they can't afford not. The end result people leaving when visas run out with more debt, and headlines about people forced to pick fruit in bikinis and allegations of sexual assault, because that happens when some people gain that sort of power.
To me it always seemed a bit similar to justifications used for the "peculiar institution" - the system can't exist without slavery, therefore slavery must exist. The benefits of cheaper labor was well appreciated; Carnegie called immigration "the golden stream which flows into the country every year."[1]
Doritos aren't going to get more expensive, because like all the real staple crops, corn is completely mechanized, and you have one guy driving an air-conditioned tractor or combine, following GPS-plotted courses.
Not sure that’s the best example to use, since food costs are a trivial fraction of the retail price of a bag of Doritos, and harvesting labor is a trivial fraction of the price of food.
Edit: that was just based on my general memory of non-perishable food economics. But I googled it and found this (sorry, quora) from someone who would know, a 2 oz bag of potato chips costs 15 cents to produce and bring to (retail) market. Of that, a fraction is potato prices.
The reason I buy expensive eggs, milk and cheese is that I know that the supermarkets fuck over the farmers. I never buy the supermarket brand, unless it's the last one on the shelf. The price difference is sometimes 1.5x but unless you're really "broke" broke, you don't notice the difference. It's a few cents a week.
While, like most things in life, it's not a guarantee, doing basic research about the farms and the cooperatives that produce the goods will provide some insight. Almost anything is better than a CAFO (Concentrated animal feeding operation).
I always buy the supermarket brand, even though I can easily afford more. It's not because I want to save a few pennies on each purchase; it's helping to drive prices down on all overpriced goods, helping out those really in need.
Same reason I avoid organic whenever possible. We don't need boutique food.
That doesn't even make sense in neoclassical economics. One person's effects on the market are necessarily tiny, but if buying more of Brand X has any effect it will be to drive up the price of Brand X. You're actually harming "those really in need", and you should feel bad about it.
People have to eat food. Food in aggregate has one of the most inelastic demand curves imaginable. If the price rises because the production costs increase, people will still buy food. They might buy different food, and you'll see a decrease in acreage given over to labor-intensive crops, but that's just the free market at work.
The thing is, if farmers in Australia increase wages and thus their product becomes more expensive, the product on supermarket shelves will not become more expensive. Rather, supermarkets will import food from another country. Food is, with few exceptions, a commodity. The producer does not set the price of the commodity, the market does.
I am telling this as an engineer working in the Canada's oil and gas sector. Since 2014, salaries in our sector have decreased and firms are struggling. "Just raise the price of oil" is not an actionable advice. The price of oil is not ours to set. It's the market's
It just shows most people working in Tech Industry, and specifically software ( or web ) development ( as oppose to Tech / software within Oil Industry or any other industry ) have absolutely zero idea on market or commodity trading.
I consider food security a national security issue. Produce is one of the few areas where I would consider a tarif justified. Every country should be able to feed it's citizens with the food grown there, or they risk hunger when cut off by war or natural disaster.
And countries do that, but for a few strategic ones, not everything. Canada produces lots of wheat and dairy and maple syrup. It even has a strategic maple syrup reserve. But it does not produce a lot of citrus fruits or pistachios or dates. It may be possible for US to produce everything within the country—it is a gigantic country and has just about every climate there is. But it would be an ecological disaster if countries without those attributes do the same. Think of all the heating required to produce warm-climate fruits in Canada.
You're absolutely right that perhaps not everything should be grown everywhere, but I strongly believe that a country should have a local source of all the calories and vitamins / minerals they need. In Canada, for example, maybe they encourage farmers to grow greens, brocolli or potatoes instead of citrus for vitamin c, but they absolutely need a source.
What tariff does is that it moves the price point at which the retailers switch to imports a bit higher. Tariff does not eliminate the ceiling. There still would be a ceiling on price, determined by international commodity price + X% tariff.
So the tariff needs to be high enough to be absolutely ruinous. See, for example, Canada's tariff on dairy. Since Canada runs a cartel on dairy producers, Canada does set the domestic price of milk at $4/gal. This is incompatible with the US subsidy model, where it prices out to around $1.50 a gallon -- so there's a 270% tariff on dairy imports. The great thing about numbers is there's always a bigger one.
Yes, you can do that. It's not something the farmers strapped for workers can do, but the government certainly can. It would not be particularly popular in most political climates, so passing such laws in democracies are always difficult. But as the Canada example shows, not impossible.
Apples and Oranges. If you include global supply, then you have to increase salaries "globally". We all (I assume) should know that demand/supply set the price.
But imported products don't fly right into the country. There is customs, and denying/tariffing them is a policy that countries are very actively using. So at the end of the day, it's a political decision.
Edit: On the other hand, if you are exporting stuff (oil for Canada), then you can't really do that for the external market. Might explain why the government will let some sectors run with illegals (to remain competitive); the alternative being their ultimate demise.
If you ban import X from a country, the country will likely ban import Y from your country in retaliation. Sometimes the net effect would still be in your favour, but often times it would not be. Do you know why American brands dominate the US truck market? It started with a tariff on American chicken by some European countries and it quickly snowballed into much more [0][1].
We reduce production, US, Russia, and Saudi Arabia would be more than happy to fill the gap. Actually Saudi Arabia prefers prices that are lower than the strike point for non-traditional oil, so it can get rid of competition from likes of Canada. How should we convince Russia and Saudi Arabia to take a hit to their exports in order to help us out, when we cannot even convince our biggest ally, the US?
This is offset by imported food. If 100% of American farmers colluded to fix prices, they'd have their lunch eaten(hah) by food importers. Obviously not everything is importable(eg rice), but there is more to the market than the American farmers.
You're right, there is more to the market but not how you describe it. Tariffs are available to any American cartel willing to buy enough politicians. Take a look at sugar, for example. [0]
> complain that young adults are refusing to relocate for $5 an hour hard labour jobs.
USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?
I think people will one day demand higher pay for desk jobs over outdoor work. Looking back on my early life, the hard outdoor work was incredibly valuable to me. Desk jobs sometimes were soul-sucking, depression inducing. Not always and my dev work now is desk work that doesn't suck.
Now, outdoor work with a high injury rate is a different story.
Here in america farmers hire illegals to harvest crops, claiming that no american would do the work. This has always struck me as a really terrible excuse. I'm sorry, you can't simply break the law because you'd have to pay an american $20 / hr to do the work. Sure, it means that your tomatoes are more expensive, but also, now there's a hell of an incentive to apply automation to the problem yes? If you don't like the law, maybe pressure your representatives to reform immigration instead of just hoping everyone turns a blind eye to your exploitation.
A couple of years ago, a serial offender food producer was raided, multiple times. 600+ undocumented workers were found, and all spoke of management knowing their status.
The workers were detained, deported, and so on.
After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.
The point is to punish and cause pain to the immigrants. Regardless of the original justification of a system, the point of a system is what it does or will do. That is what this system does and will do unless changed.
See Postville Raid [1] That was in 2008. The company owners were some prominent Jews, and the political fallout was so severe that immigration raids pretty much stopped after that.
I don't know anything about it, but from the Des Moines Register, it seems like a lot of people felt there was a miscarriage of justice:
"There was false testimony and willful manipulation, and that makes this a shocking case of prosecutorial misconduct."
"An unprecedented letter, recently sent to Kevin Techau, U.S. attorney for the Northern District of Iowa, signed by 107 former high level Department of Justice officials, FBI directors, federal judges and prestigious law professors attests to the extraordinary prosecutorial misconduct which exacted the most possible punitive sentence for Rubashkin"
Trump commuted Sholom's sentence in 2017, apparently.
I'd suggest a different phrase than "prominent Jews"; the family are "ultra-Orthodox Jews of the Lubavitcher hasidic movement" not representatives of some sort of Jewish privilege in general.
Of course it isn't that simple. If your farm's tomatoes are more expensive than your neighbors' tomatoes, then nobody will buy them. You'll quickly go bankrupt.
It is also difficult because of the two tiers of labor. A farmer will try increasing wages by $1/hour, and still struggle to find enough workers. Wages would probably have to be doubled before new workers—Americans—would start to enter the industry.
Or, you know, you let unprofitable endeavors be substituted by profitable ones, and you just import food from different places where it is profitable. Just make sure not to wage war with them all.
That's always an alternative to subsidies and tariffs, but it's not as politically useful.
It's also in general how we end up with monopolies. Well-funded companies operating at a loss until they're the only noteworthy players ain't exactly unheard of.
"gave up" sounds like an unnecessarily ominous take on "delegate". But what's your exact point? Do you believe this is an irremediable situation or that know-how has been lost to time in the age of information?
I'm assuming you're implying a conversation about China and their manufacturing capabilities. But worst case scenario short of nuclear war, assuming US-China burns bridges and somehow for some reason now US is fully unable to import Chinese manufactured goods; how does this eliminate the US and all other countries from investing in an area that has now become more profitable? You could argue that prices would increase, but then again you aren't owed cheap manufacturing, and subsidies and tariffs are paid by either the taxpayer or the consumer (which tends to have a knock-off effect on the rest of economy, anyway). There was never a free lunch.
That's easier said than done. The problem is that in order to reform immigration you have to do one of two things:
1. Start treating people fairly, with the result being higher food prices, which will make some of your constituents very unhhappy
2. Codify the current class structure into law so that you can continue to exploit cheap labor legally. The optics on that are really bad.
Exploiting people off the books is the path of least resistance for everyone, including the workers (manifestly so, or they wouldn't risk life and limb to be here).
If farmers paid enough to get Americans to do seasonal farm labor their produce would cost twice as much as central/south American produce and they would immediately go out of business as everyone buys cheap imported fruit.
People get really pissed off when food gets more expensive. Politicians realize that and will therefore never pass legislation that would cause such drastic price increases on food.
If they enforced the laws already on the books, they'd have to pass new laws to fix things. I see that as a net positive. If you selectively enforce against illegal immigrants, well, that's very much a 'rules for thee, not for me' situation isn't it?
Here in america farmers hire illegals to harvest crops, claiming that no american would do the work
This is a political talking point that has been repeated so often that people believe it is true.
During the recession of 2008, there were stores on local TV news in many places showing unemployed Americans waiting in long lines for the opportunity do do those jobs that the politicians kept saying no American wanted to do for those wages. Meatpacking plants, especially, had more than enough Americans to choose from who were willing to work for low wages.
Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
The more you actually go places and do things and meet people and talk to them, the more you realize that the words coming out of the political organizations on the coasts are mostly for their own benefit, and do not reflect reality.
My somewhat educated guess is that these farmers in the PNW don't share the same politics as farmers in the rural south or midwest.
That and Republicans have grown way more hostile to immigration since 2010, so it's an interesting anecdote but not sure how well it translates across the US, where polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not.
"When you say it's an "educated" guess, educated how? It only seems to reflect the usual bigotry that comes out of the coasts."
I lived first 28 years of my life in the rural south and now live in the midwest.
My entire family and most of my friends growing up are the people you claim I'm bigoted against...
"If you're going to make a statement that sweeping, you really should back it up with credible sources."
The fact that farmers are generally far more conservative and republican than the median voter or average American isn't exactly some shocking new insight.
You seem to have never spent much time around these folks or have done much research on them if this comes as a surprise to you.
I don't think your points really contradict your first statement. First, people tend to drop their standards on the type of work they're willing to do and pay they'll accept when they're desperate. In 2008 a lot of people needed a job. Any job was better than no job. I'm not surprised that jobs that would normally be unpopular had lines of people applying at that time.
Your second point about farmers trying to increase legal immigration in 2010 clearly indicates most Americans won't regularly do this work for the wages being offered. Why else would farmers be looking to hire immigrant workers just 2 years after locals were lining up to get these jobs?
While it would be nice if there was a way to get enough immigrants working legally to fill those jobs that doesn't seem to have happened. Short of another large economic downturn I don't see Americans working those jobs either. That leaves people who are working in the US illegally to fill the gap.
> A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
> Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
Farmers are, for the most part, single issue voters: Whoever has the best plan to support agriculture and small businesses gets their vote.
>Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
How do you determine from TV that the people lining up for meatpacking jobs were not undocumented immigrants?
Like, it is generally accepted there aren't any in "flyover states" or something?
I never really thought of it until this moment, but you know how within 100 miles of any border, your constitutional rights are somewhat compromised so DHS can operate? That ought to mean a pretty substantial incentive for people who are undocumented to migrate inward. So I would bet they do.
About half of US hired farmworkers are illegal immigrants, says the U.S. Department of Agriculture.[1] It was only 12% in 1991, but passed 50% by 2000.
Sounds like exactly the same problem British farmers are having post-Brexit with things like fruit-picking, they're scratching their heads wondering why British workers aren't keen on having half their wages taken for "rent" to live six people to a crappy caravan on-site and do backbreaking labour day in day out. Their only choices are to improve their wages, conditions, both, or go bankrupt.
Well, the problem is this: it might not make economic sense for those specific companies to hire people at a higher rate.
Companies aren't going to say "I guess we're inefficient and should just die." They're going to fight to survive and thrive, which is what you're seeing.
Then again, some companies ARE just cheap. You never know.
I was at a hotel with crappy service and asked about the problems with house keeping. The manager told me everyone in town (Traverse City MI) was trying to bring in foreigners to do housekeeping because they couldn't get local labor.
They said the standard is 30 minutes per room per day. I figure at a very generous $50 per hour (100k / year) it should run $25 to have my room taken care of. Now that doesnt cover the shared costs - doing laundry, cleaning the pool, front desk, etc... but my room was something like $150 per night. How the fuck can they not find people to clean rooms?
Answer: they're too cheap to pay a decent wage. But why? My suspicion is that all these places are actually leveraged far too much and paying rent/interest/dividends so much they can barely operate.
I suspect low interest rates and other incentives to "stimulate" economies are actually causing these problems.
You make no sense. No one will pay $25 to clean a single room in a hotel. Just because your room is $150 it doesn’t mean that “there’s plenty to pay everyone 100k/year.
I assume the cost of a room cleaning (including costs other than direct labor) is closer to $5. That's about what hotels will often give you in some sort of funny money to skip a cleaning.
The service industry might be labor intensive but is hardly the only cost for the owner(s). What about mortgage payments, marketing, integrating with third parties (expedia), gas & electricity, general maintenance / accounting / security etc. I am sure there are hundreds of other costs that I have no visibility into
At college I had a professor who I worked for who grew up in Romania. Apparently the song that kids had back in the day was: "We pretend to work and they pretend to pay us."
Don't forget the whining from bureaucrats when American tech takes over the local market, and ships profits back to it's headquarter where... they employ Romanians expats!
Even if you open a checkbook wide, there's no guarantee that someone remembers the specifics of some kludgy system they worked on years ago. And if things are really snafu-d, there's no guarantee that even someone who does have domain knowledge can recover things.
Let's not forget about the Mythical Man Month. Sure, specific people amd money can fix the problem, but we can't just assume that throwing people and money can fix this.
I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.
Is there another industry where a few employees with "domain knowledge" can sink a company?
Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?
Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
This. I actually think just documentation is, well not completely worthless but close to. Imagine telling someone they could learn to weld airframes by reading. Just no.
For software, it’s the maintenance task that’s helped by documentation not building. They still need to know how to build but it’s a lot easier with documentation. I know that when we use software packages we all rely on the documentation to understand how to use it and code our desired behaviors. It’s obvious that our own code can benefit from documentation in the same way.
documentation is required but not sufficient. also, documentation is usually outdated really fast if the person maintaining the system leaves and/or does not care
The issue with "documentation" is there are never enough resources devoted to it at most companies. When given a schedule it only covers the main project goal(s) and no time for documentation or refactoring or any quality of life improvements.
Significant documentation is also not something easily written in slack time. With any non-trivial project good documentation is a full time effort. It also needs to be updated as the project evolves lest it become out of date and incongruous with the behavior of the system.
There's dreams of in-line documentation but that's only going to cover individual methods and classes and not necessarily whole modules or major subsystems of a project. It's also not going to necessarily cover in-line documentation from other parts of a project in different languages e.g. comments in config files.
There's lots of technical ways to help documentation but there's no replacement for documentation being a high level project goal with resources allotted to it.
Part of a way around this is broadening your definition of "documentation".
For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
A fully scripted build and ideally also deployment (CI/CD pipeline) beats a checklist with tens of items that have to manually followed.
An issue system with good descriptions of changes (including why and what), reproduction steps, expected outcomes, links to related issues, etc also can go a long way.
All of this stuff has the side effect of making day-to-day work simpler, overall, while actually just being a part of that work.
> For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
1. Naming things is hard. It's a bit of a joke but good descriptive yet usable naming is hard. It can get harder after refactoring or adding in features you didn't expect the last time you were in the module.
2. Project structures start out simple. Then you need to add some feature or fix some major issue with a due date of last week. As a project gets bigger the simple structure of yesterday might not serve the needs of today. Resources for refactoring are usually somewhere around the priority level of "Good Documentation".
3. Comments need maintenance like explicit documentation. Tight schedules lead to "# TODO add documentation here". Then you get the assholes that believe comments or in-line documentation is something you put in check in notes.
4. A "unit test" that covers a "real scenario" is not a damned unit test. It's an integration test (or however you label it) and it's not likely something your CI/CD system can run on the regular. If a real scenario is touching a production-load DB or something it's a big deal to run the test. Tests like that need resources to run and non-trivial resources to write. You're not just making sure a method returns a float or throws an error correctly.
I'm not saying your suggestions are bad or shouldn't be used. The issue is management rarely sees any value with documentation, tests, or even code cleanliness. They're rarely incentivized to care about those things so they don't. They get their bonus on feature checkboxes or hitting a deadline. At the end of a project they'll schedule a "documentation" sprint or some dumbass thing that accomplished little useful output. It's worse when you've got a "move fast break things" chanting moron doing the scheduling.
It doesn't help that "Technical Writer" is now an incredibly rare job title, it was one of the first in a long line of specializations that were pushed on to general developers. As you mentioned, they were never allocated the extra time but also don't have the training and experience that technical writers once did. It was also a very poor target for outsourcing.
The specializations common on a team even 20 years ago was much closure to the "Surgical Team" model from the mythical man month than what we have today.
At the highest level, this is a leadership issue. There's a lack of someone at a sufficiently high level (eg: C-suite) that properly understands technology, and that's probably caused by the rest of the leadership team not understanding their limitations.
At a lower level, it's allowing individuals to become knowledge silos (or having a bus factor [1] of 1); lack of documentation; not having+testing disaster recovery plans; neglecting systems, code bases, and platforms; letting technical debt accrue endlessly (or worse: not even realize that's what you're doing).
How to avoid? Make sure there's a strong, competent technology leader with a vested interest in the company succeeding, and empower them to make the changes they need to.
What should that person be doing? Making sure all the stuff I just said doesn't happen.
>Is there another industry where a few employees with "domain knowledge" can sink a company?
I guess any company where leadership got there without progress on the technical side. Law firm partners get there after being lawyers. Consultant partners get there after consulting.
But if you are the CEO of an IT company without ever having to code, then you may never know the things you must take care to prevent a washout.
> Or rather, how does one avoid getting into this situation.
IMO, the problem is technical debt. Most likely, in the majority of these cases, the engineers working on the system knew perfectly well that it was a mess, was very difficult to learn how to maintain, and many parts were dangerous to modify. Most likely they told management multiple times that this was a problem and what the consequences of it would be. They would have refused to budget resources for improving it, so everyone either saw the writing on the wall and quit, or gutted it out and learned how to manage the system eventually. Maybe the system limps along for a while longer, but it's only getting worse and the knowledge needed to maintain it more siloed and specialized. Eventually you lose a key person, something breaks that nobody left can figure out, or maybe the whole thing just implodes like the main story here.
To avoid it, you must understand the pattern, hire good technical people, and give them the resources needed to improve the system in an orderly fashion.
There are a LOT of industries with this problem. Specialist stuff like military shipbuilding is an example, but also things like precision ball bearings. Knowledge gets lost all the time and old timers have to be brought back in after the fact. Or the technology is just lost and a replacement has to be reinvented (famously, the production of tamper material for the US thermonuclear nuclear weapons).
> To anyone outside of the ballpoint pen manufacturing world, it might seem hard to understand what, exactly, is so surprising about this development. China already produces 38 billion ballpoint pens a year, according to China Daily, which is about 80 percent of all ballpoint pens in the world. That's a lot of pens, but there was a catch: China had long been unable to produce a high-quality version of the most important part of the pen, its tip.
> The tip of a ballpoint pen is what makes it a ballpoint pen. At the tip, a freely rotating ball is held in a small socket which connects it to an ink reservoir that allows the pen to write or draw lines. Manufacturing a ballpoint pen tip that can write comfortably for a long period of time requires high-precision machinery and precisely thin steel, but for years China was unable to match those crafted by foreign companies.
> Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
Mentorship and documentation.
You need to set aside time and resources for these two things, and make contributing to them worth something in the performance review.
If all you do is count JIRA tickets closed, you'll forever end up in this situation.
It's slightly more complicated than that, depending on how much time has passed since the creators of the system left, but you're correct in general.
Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.
The person who started the thread added this comment:
> The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..).
Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
Yes. I had an ancient system where the holders of domain knowledge kept either dying or retiring, one by one. I was frantic, but nobody listened to me. The system began to fail on the day I announced my departure. I would later find that one of the previous architects had reached out to my manager to offer assistance, but this was never made known to us. It wouldn't have even cost anything.
Throwaway purely so that I can comment. I've had this happen before.
Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda.
Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due.
My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable.
I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it.
Dougie, if you're reading this I hope your cringing.
Funnily enough it made me sick to my stomach. I manage a small company and the idea of being extorted in such a blasé and psychopathic fashion terrifies me to even think about.
That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee?
I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.
IMO your edit didn't help and you're not getting a lot of sympathy. The point wasn't that the employee is psychopathic as you say, but that management was incompetent by not ensuring a transfer of knowledge and validating before the employee left. You as the buinsess owner should plan for people to not be available and no one person can derail a $10M contract. This is also called the "bus factor".
Also yelling at someone who is the only person who can solve your problem is not a good negotiating strategy, especially when it reminds them of why they left.
Walking away from a toxic situation is not "psychopathic." It's common sense and if we all did it more then the psychopaths who expect us to stick around through it all would finally learn how to treat other people with respect. It's a job, not a marriage.
I mean... what could the other side of the story actually be?
"We tried to hire back an employee to build something that we didn't train or try and figure out ourselves and a deadline forced us to pay a large lump sum to that employee".
No matter how you slice it, that's really shitty planning on the businesses side. They should have spent the time and money learning how to maintain their shit. By not doing it, they paid a huge price.
I imagine if it came down to it they could try to pin it on him and say he didn't provide documentation he said he did, and that's why this entire 10M project now can't be delivered.
Luckily for him they decided it's easier to just pay the fee.
Even if he was a shitty employee that doesn't obligate him to work for a price favorable to the company. Hell, they could have fired him for not documenting shit.
Because here's the thing, even with how weak US labor laws are, one thing they don't allow is for a company to come back and sue your for incompetence. So long as they didn't go out of their way to sabotage things, they are free and clear.
I don't know why you'd make such an assumption about me, please expand.
I think in general all of my comments on this thread have been horribly misunderstood - I was just trying to say that if the company was scummy, instead of paying him the 250k fee, they probably could have sued him instead. I'm not saying that should have done, I'm not saying it would have been the right thing to do, that he deserved it.....literally nothing of the sort. Just pointing out that the company could have made an argument against him in court. As many other people pointed out to me, it would have almost certainly failed. But I don't understand how you can deduct where moral considerations are on my priority list from this?
sure it does. the company would have to prove malicious intent beyond a reasonable doubt and not leaving adequate documentation absolutely does not meet that requirement.
This seems absolutely unrealistic to me. Can you find a precedent? A single precedent of the company suing an engineer (individual contributor, not a manager) for quality of work after he left? I think if they tried they would have been laughed out of court; and out of all their future contracts, too.
Maybe this is my naivety showing, but to my knowledge it's not a common occurrence for companies to sue employees for refusing to work for them, especially if (as the OP notes elsewhere in the thread) there's no actual contractual obligation.
A bit late to the party, but even when threatening with insolvency-via-lawsuit, this would probably not work for them:
- if he can prove the threat, the company will be in an even worse position
- given the rather lacking precedent (as pointed out by the sister comment), there's a good chance the suit would be thrown out pre-court and therefore be rather cheap
- given that the OP seems to be a valued engineer, it is reasonable to assume that he has the f-you-money to fight this out
- even if won, the drawn out battle would take far too long to fix the software in time and, given that they then lost a 10 million contract, they lack the motivation (and maybe money) to continue
- lastly, even if they won after loosing the contract, they'd probably not be able to extract much in damage payments
That won't work in most reasonable countries. A company that is no longer employs someone cannot make that person work. In the situations similar to the ones that are being described, the time is not on the side of the company - they have a dead line, a former employees does not.
The other side of the story is it's fake, fake, fake. But it's a nice story.
Every negotiation is about the best alternative the parties have, and it's generally harder in real life than in fantasy because you underestimate the alternatives to a deal.
Applying maximum leverage has a way of creating "out of the box" alternatives. And vindictiveness.
I think enough of us have been on the same side of this story that we feel fairly comfortable taking it at face value, even if we're only applying it to said personal experience in a wishful fashion.
Yeah, HN at it's finest I suppose. He's like you, he says what you want to hear, therefore you believe him. No need to consider the possibility that maybe there might actually be another point-of-view.
The point of view of managers and business owners is well known. Work is not a new concept. You get what you pay for. Nothing is owed to you. Work? Tough luck, you have to pay for it too. On a free market, no less. I don't understand why business owners think the rules are different for them than the rest.
This is not extortion. The company made choices that left significant leverage in the hands of another business. That business correctly perceived the value of the services it could render and priced them accordingly.
This is exactly the free market at work regarding labor. Have a problem with the free market if you like, many of us do but do not accuse this poster of criminal action equivalent to a protection racket.
The market answer to that is, the seller underpriced their services to begin with, regardless of the personal motivation for increasing the price.
To clarify, the market would likely not allow this specific work to be sold at this price normally. But the buyer wasn't just buying specific work they were buying specific work to be completed by a specific date. The work in isolation does not have this value but the work completed by a certain day does. Companies use this rationale all the time charging more for expedited services.
It's especially common in construction and manufacturing where costs double or even triple when expedited delivery is required.
"Everything is worth what its purchaser will pay for it." That includes labor. The business was free to try and renegotiate their contract with their customer. They were free to delay the release and offer a discount. They were free to turn down the 125k and the 250k payment on principle and deal with the consequences. They found the most economical solution that met their needs was to pay $250,000 to have the work done on time. Was this poster engaging in rent seeking behavior? Absolutely. But that is a consequence of the market and one companies exploit to their own benefit all the time.
> They found the most economical solution that met their needs was to pay $250,000 to have the work done on time.
I'd be inclined to believe that if there were some other providers but as it is told working with OP was the only solution. And OP's first offer was 125 000. Two screams later it's 250 000 ? It has nothing to do with economical theory or free market interpretation.
I find this explanation (from top comment) more in line with how the world spins:
> If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
So, “being a dick” has more to do with price raising than any free market justification (which I am okay with, just don't try to convince me it's the only thing at play. The offer would have been 250k from the beginning then. Or 500K to get 250K, whatever.).
This is such a silly and mundane bit of pedantry you're harping on here. Of course there are emotional factors in price discovery-- it happens all the time, and is a completely natural and rational part of the free market.
Any small or independent contractor in any business will eventually have a blacklist, or a list of troublesome customers who get charged extra. Most of my contractors charge me less because they like me (and I pay on time). My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole; the contractor has decided that it's only worth dealing with the man's unpleasantness if there's more money to be made. Apparently the man has been blacklisted by other HVAC companies, so he might be in a tight spot, but whatever the reason, he ends up paying a lot more money for the same service.
This is a completely market-based mechanism: the troublesome customer has limited options (in this case, maybe only one option), and is thus obviously disadvantaged in negotiating price. Regardless of the myriad motivations involved in price discovery, it all leads to the same place: if the price wasn't worth the service the customer was to receive, they would not have paid it.
Honestly, this isn't mysterious or profound. It's economics 101.
Not even that. It's payback, that's all it is. There's no need to call for a more elaborate explanation for what was at play here. Not that I disagree with it. It's humane nature.
> My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole;
Like you say. No need to even know about market dynamics to (wisely) choose that option. Because the other guy is an asshole.
First, he owed no-one anything. He could set the price to 1 billion for all he cared. His former employees could take it or leave it.
Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational, would have left you with half the money the "irrational" player would get. Seems to me, the "irrational" is actually the rational one, as he netted more profits. Economy and market are not math games in a vacuum, and players are not formless ideas.
> Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational,
The way I see it you are putting words in my mouth and I am definitely not interested in having a discussion like that.
I never insinuated OP was a sentimental fool or wrong to raise the price. I am calling bullshit on the free market and econ 101 as an explanation of why the price got doubled and call it like it is: it's payback from OP because they were dicks to him first, everything else is pedantry and useless justifications.
edit: if anything, OP could be considered a fool since he obviously gave an under the market quote as his first proposal. Had he started with 250K from the beginning he would have gotten more since there likely would have been screams anyway and he would have risen the price all the same.
No one said the free market is "why" the price got doubled. Emo drama ended up being the mechanism of price discovery here. But the market accepted this price. It was not a swindle. It was free trade.
And yes. I completely agree with your edit and said as much elsewhere. Their acceptance of 250K showed that the OP under-priced their services at 125K.
And yes. Free markets produce monopolies. Especially when time is important. As I said above, the critical good here was not the delivery but the delivery within a certain time.
This is literally the definition of price discovery in a free market. If the price was not worth the service the company was to receive, they would not have paid it.
The fact that they did pay it clearly indicates that the first price was far below what the "market" would bear.
"The market" has nothing to do with that because one single trade isn't a market; if you and I exchange 1g of gold for $1000 that doesn't say "gold is fairly priced at $1000/gram". If many people exchange gold for money, each with different information and desires, over time the gold will go to the people who pay more, sellers will generally trend their sale price down because higher prices won't get buyers, buyers will trend their buying price up because lower offers won't get accepted, and a range of "price gold is being traded at" values emerge, the spread of those values highest to lowest narrows and hones in on, and tracks a dynamic "price of gold" where it's "fairly traded" - not fair in some Deity decreed way, but fair in that if you trade at close to that price, you aren't buying unusually high or selling unusually low, you're getting no better or worse deal than anyone else, so it's not unfair in your favour or unfair against you, so if it's not unfair then it must be fair - you couldn't get a significantly better trade with anyone else.
Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
But I disagree with your implication of "the fair price is the one paid" is somehow unfair in an ethical sense. (I also think OP is in no way obliged to be "fair" in any ethical sense to the company, but that's another matter). The company got a $10M contract for $250k, versus losing the contract, is that unfair? If that really was the pivotal factor between losing the contract and not, the company ought to benefit by spending right up to the point where the contract becomes a money loser, to not lose it. Would it be unfair for the company to get an $8M contract by paying $2M? Or a $2M contract by paying $8M? Is OP obliged to help the company make as much profit on the contract as they can, by charging them less for his services?
> Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
This is non-sense. Those conditions specifically define a market and make it distinct from any other. To say a market with any of these constraints is "distorted" is to say all markets are distorted and render the term meaningless.
Also, I never said this one trade signified a market. My point was that this is a consequence of free markets. You can make any single trade "irrelevant" by choosing arbitrary criteria by which to separate it from its market. The company could have certainly hired another engineer in hopes they could complete the work in time while running the risk of blowing the deadline. But it was freely and fairly worth at least 250k to them to guarantee the job could be done correctly and on time.
This kind of thing is such a common part of the free market, that there is a whole aspect of contract law that pertains to ensuring companies don't get into this position. It's why contracting completed work is so important. Leaving things partly done leaves you vulnerable to others eating right up to the tiniest margin of your profit.
Time was a factor. Negotiations in such a situation waste time. Doubling the price associated the cost of wasted time with additional negotiation in a tangible way. In clarifying the situation, thread parent was really helping that former employer. Had that clarification not occurred, the contract might might have been signed too late.
"Lack of planning on your part does not constitute an emergency on my par--wait did you say $250,000?"
If your company were in the hands of incompetents, but still kept going in spite of the incompetents because it had gotten its foot wedged in the door or whatever, and then fell apart in ways you could only scorn, how would you view that sort of situation?
The company can avoid this if they only care to maintain the documentation / wiki left behind by said person
When they failed to do even just basic proper operations, it's their own fault.
I'm a business owner and a consultant, and I too charge more for these kind of asshole clients, because of the extra stress.
But nowadays I just try to avoid them, it's just too damaging for your own sanity. Their kind of stupidity, self-entitlement, and ignorance is on a completely different level.
Citation? By that light, a DVD containing defamatory speech would be libelous, but that's generally not how courts see it.
From the (highly-influential) U.S. Court of Appeals for the Second Circuit, summarizing New York law (which is quite typical): "Defamation, consisting of the twin torts of libel and slander, is the invasion of the interest in a reputation and good name. Generally, spoken defamatory words are slander; written defamatory words are libel. Libel is a method of defamation expressed in writing or print." [0]
There is also "calumny", which is the action of slandering someone (and I assume also making writing libellous statements. After writing that parenthetical, I just googled, and libel is also a verb. So I guess I can say libelling someone)
Business is business. The right to fire people, deny them raises and form binding contracts in regards to their work comes at a price. GP did not act malicious, rather, they acted in the same self-interest most bosses, CEOs and companies act while trying to stand on the moral high ground.
Coding standards, documentation requirements, meetings and code reviews exist for a reason, too. It's not like companies are defenseless.
The employee documented everything, but said documentation was not backed up by the company. Furthermore, the company did not train or hire an adequate replacement for said employee for _six months_.
This is a gross failure of leadership at the company, and the former employee has every right to negotiate compensation for the inconvenience of having to work for them again.
The fact that you'd refer to this as extortion makes me sick. The management/capital class loves the free market except for when it applies to labor, then suddenly it's extortion when supply and demand don't work in your favor.
Then treat your employees better and adhere to best practices and you won't be subject to such "extortion". Once we have elft yoru company, especailly if it was because we were treated poorly, we are under NO compulsion to help you for free. If the idea of paying 1.25% of the value of a deal to make sure the deal happens seems excessive, you're greedy, and that's why his price went to 2.5%.
The idea of /you/ mismanaging your company in such a "blasé and psychopathic fashion" is more what should terrify you. Remember, /you/ manage a small company, not the person you hire. If you end up in such a mismanaged position, the manager who let it get there is you, not the employee.
I owned a small company. After selling it, and giving 2+ more years of my life for the new owners to figure out how to replace me, they didn’t. On my last day I told them that if they ever needed anything my rate is $5,000/day, non negotiable. It’s kept them at bay for now. I have no ethical qualms with my decision. It’s business.
Sounds like your best next step is to become an exemple of software/infrastructure best practices - get everything in the best backup/recovery solution you can, document every detail any time an issue comes up, and above all treat your people like the vaaluable assets they can be.
You may be doing any of that already, since I don't know anything about you except your one comment.
But the answer to your fear of exploitation is avoid making people hate you/your environment enough to want to protect themselves from it.
I think this is just flamebait. Uses emotionally charged words without any other substance. Both the comment and responses to it (including this one) add nothing to the discussion.
You're terrified that you might treat your employees so badly that you would need to pay a lot to resolve a situation? Seems like that's entirely within your control.
Hopefully you wouldn't mismanage your own company and treat your employees like shit. Probably you are a better person than that, and can avoid this ever happening to you.
I see and understand your sentiment but, in my rather insignificant few years in the workforce - carrots, dipped in vanilla chocolate and honey, generally tend to work better than the stick.
The company could have apologised, and do a lot more ego stroking before asking a favour which they can pay for and maybe throw in an all expenses paid weekend with the SO as a thank you. Not only will you get goodwill but future problems are going to be significantly cheaper to solve. Even if you have a bad break up - as a business owner you should understand the value of mending bridges. It's cheaper than losing sleep at night and losing your steak dinner to a bad stomach.
He left detailed instructions how to release the software. The company managed to screw up so bad they lost the manual he made sure to write on their internal wiki.
You forget it's all part of business. If you hired an external contractor you would not say a word for demanding their money. And the price of the contractor grows by a lot relative to how much and how soon you need their work. Remember, you need their help, not the other way around. A contractor on bad terms with you would not help at all. How is a former employee any different? Do you think you become family after paying for their work?
If you do not have the money to pa for someone's services you cannot complain. You say you have a company? Have you ever let a client get your products by paying less than their value or nothing at all? Work itself is a product too.
Yea, this is one of those "I want to believe" stories! It's practically engineer-porn. Everyone who has quit a shitty employer whose technology was a house of cards resting on the point of a needle is cheering OP on. We all have that fantasy that the house will finally fall over, the company will have its comeuppance, and we're the one person in the world with the knowledge to fix it.
The letters stated that I had a contractual obligation to assist. But I noticed at the top they were stamped 'without prejudice'. I asked my friend what that meant, and he said that technically it's not a legal letter - so I ignored them.
For a while, I made it game to light the various communications on fire in interesting ways, but it got boring.
The idea is that you want a trusted third party because either of the two adversarial parties could screw the other over.
The company could say: we’re going to pay you 125 and you could sue us and spend 60k and two years or just settle for that, after he’s already done the work.
The guy could demand the payment directly to his account and then not deliver, and the firm would be in the same situation.
By depositing the funds in escrow, they’re making sure that the money is there and in the hands of people whose only incentive is to give it to the correct party upon fulfillment of the contract.
Look up what "escrow" is. When 1/4 million dollars is on the line, you don't take the company's word that they'll pay you afterward, and they don't take your word for it that you'll actually deliver the service.
Hell. Yes. That's absolutely beautiful, and ethically completely correct.
I have bosses at previous companies that I'd help for free today because they were good to me and we parted on friendly terms. One boss helped me move my house once, and I'm not about to charge him to fix some random issue.
I have other bosses whom I'd require a cleared wire transfer of money before I'd lift a pinky to help them, because they were not nice people and I don't want to associate with them or assist them with anything. If someone in this camp approached me, I'd have a story just like yours afterward.
This is actually not a very rare situation. Unfortunately many of us don't really have the smarts or courage (overcoming guilt) to negotiate a price like the OP did.
I know this sounds like an empowering story, and it may even be true, but don't actually follow this example. If you're at the point where they'd rather send you legal threats than pay you, even escrow is not going to help you much. The escrow process may force them to pay, but there is no guarantee you'll be able to keep that money if they really decide to go after you.
People like that would much rather spend $2.5M on courts and lawyers than let a worker walk home with 250k, or even 25k. Agreeing to a deal like this is an incredible minefield that may blow up on you even years later.
565 comments
[ 2.8 ms ] story [ 315 ms ] threadI know of one case where power failed, the "24 hour" generators kicked in, power company says "We'll need 12 hours" and the outage happened 6 hours later when they ran out of oil.
When I was there this didn't go too well and they couldn't get one their data centres online again - failover to their other centres did work though. This was ~20 years ago in the finance sector.
I was in a building when someone inadvertently powered off the wrong equipment, which had been running for several years, and several of the power supplies failed to come back up. It was 1+1 redundant though, so we could quickly shuffle packs around to bring it back up without redundancy. Then, jogging through the building and asking if anyone had spares, we found a field tech in the lunchroom who had a pile of stuff in his van. Whole thing was back to 100% in less than an hour, and we let the beancounters sort out the field spares being used for office equipment.
If that same failure had happened during the overnight maintenance window (when volatile work was supposed to be performed), there certainly wouldn't have been the same resources around.
[1] https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
Ransomware scenarios come to mind, where the attackers will look for and actively destroy the backups you have if they are remotely accessible with e.g. AD admin credentials, and only trigger the ransomware when the backups are gone. If you have offsite backups in some cloud system, that helps you against natural disasters but not against malicious activity. The need for backups that are not just offsite but also offline is somewhat recent and very, very many companies do not have them.
I doubt any external factors are to blame.
You are, of course, assuming they have a backup and the backup can actually be restored and that the restore contains the required information.
> Our backups are fucked.
So I assume that they have already tried and they didn't work.
No problem they said - we have backups. They had three tapes.
First tape failed.
Second tape failed.
Third tape worked.
In retrospect that was a useful, if rather stressful, lesson.
> a missing $
Bravo!
I've always had anxiety when running a shell script, this is a perfect example of why
Early in my career, we had a storm that caused an electrical surge. In spite of the power protection, it damaged the tape drive on the main server (unknown to us at the time). The entire rest of the system (CPU, RAM, hard drives (RLL!)) were fine.
Listening to the backups being run, I noticed it sounded a little different. Eventually I determined that the tape drive (old QIC, maybe 40MB capacity?) worked, but it would not switch tracks when reading / writing. So the backup ran, but it just overwrite the same track rather than write to subsequent tracks.
A short backup / restore wouldn't have caught the problem, it was necessary to try to read the whole backup to see. This wasn't caught for a while, so all the backups were mostly useless.
> > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there.
Sure sounds like the data is lost to me.
A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.
Those people are probably employed somewhere else and would need very significant incentives. More than likely their current employers would take a dim view of them freelancing to help a competitor, so: enough to take a year off or even to retire on. Basically the idea of hiring back former employees as consultants is a non-starter.
I would be looking at the Internet Archive and Whois too figure out some basic domains and what the registrars are and go from there but of course depends on what business they are in and what their domain names were doing.
So, it probably is that bad...
But the data should still be there, and it's a matter of reverse-engineering how to go from "ERP DB" to "DNS zone file".
-- https://old.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."
Elsewhere, they say that two years ago, they found they were unable to restore from backups...
So their business actually failed a couple years ago, and they have only noticed this month.
But the fact that after this event the management did not get it fixed (and requested that IT demo it, three times, with no failures) is an existential failure.
Its Password123. Adding capital letters makes it more secure.
Password@123
Highlights:
1 Bookings where made at individual cinemas
2 The network was a packet radio system that blocked :-) only Transaction could be in flight (Leicester square had two channels)
3 Core of system was Perl running on three windows NT machines which manipulated a shonky screen based system (Pacer Cats)
4 Trying to sell tickets online for Phantom menace system crashed at midnight (when tickets went on sale) we made 1 transaction at 6am
% And no credentials so I had to research how to crack NT - I did consider setting up or fleet of a dozen or so of our suns as a cluster to crack NT, but not sure if the internal security would have liked that (aka secret squirrels and the bit that Bruce Shenier worked for
Its complicated by at the time the total lock on ticket booking systems cinema and theatre by a particular company - who wanted 1/4 mill plus to update the system.
It was an interesting project I had to get cover from a v senior manger to investigate how to break into NT.
"You need to give us 3 months heads up no matter what, but don't worry, we don't have to give you any if you oversleep for a meeting."
Mind you - if the company is going down the tubes they are highly unlikely to enforce any job contracts.
I didn't break the contract anyway out of some combination of feeling like it was breaking a promise and the worry that Cambridge is a small town.
But this is for when both of them can't find a mutual solution. You can always agree to terminate the contract earlier, if both parties agree.
At least that is how it works in Germany.
It goes both ways of course, my employer is also bound by the 3 month period in case they want to fire me.
There are contractor agencies filling the niche of providing employees on short notice (and usually for a short period), but generally having people begin 3 months later is just something everyone expects and factors into their processes.
The average length of time people spend at a single job is also longer than in the US.
Of course, they would rather it was shorter, but hiring is already a multi-months long process in most cases, so a few more months is inconvenient, but not the end of the world. If you need someone now, you hire contractors.
Also, depending on the country, firing/layoffs can be difficult without cause (and often even with cause) (e.g. you can't layoff someone at will, you might need to prove that you have an economic downturn before downsizing). But this varies way more between countries as far as I know.
But I think it would be better to keep it above board. Present it as a solution, kinda. Like, "we both want the same thing here".
[edited to add : not relevant, they're in Finland: https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu... ]
Source: left voluntarily
Or
If your contract is exclusive they can pay you and prevent you from working for anywhere else (gardening leave).
You can’t (at least in most countries) enforce the contract in such a way that you force someone to work.
But there's some additional factors here. If the employer misses salary payments, then they're in breach of that employment contract, and unless they can't fix the situation really soon, that's grounds for immediate termination.
Also in Finland, salaries earned in a normal employment are backed by the state for up to 3 months of missed payments (the state will then try to collect from the employer). So it's relatively safe for an employee to stick with a sinking company. Just get that CV polished and start warming up the network.
Even if you're not responsible for the problem itself, leaving your employer in the ditch, when there's a major outage - well, it just doesn't look good (barring some scenario where your employer is completely unreasonable - like not paying you).
It's kinda like a firefighter that decides to turn off the hose and go home, in the middle of a raging fire. Who in their right mind would want to hire him/her in the future?
But if they get in a situation where management is yelling at them and demanding they work 20 hours a day until this is fixed somehow and they have no idea how to. Management won't bring in consultants, etc. That ship has already sailed.
If your manager does that, you can say "No". They still can't fire you without paying you your 3 month wages.
The firefighter who recognizes when the building is about to collapse and gets out of there is a good firefighter.
It looks like they could probably sort something out, wouldn't be a bad idea to call up any of the old vets who retired to see if they could come down to the office and contract for a week or so.
Ultimately though a person could not show up for any reason at all (e.g. hit by the proverbial bus) and you need to plan for that. Sounds like a company that just flies by the seat of its pants so they probably have that 3-month clause for that reason.
In that particular case, my experience tells me the guy won't get any credit for finding a solution, I would consider sick leave. Burnout, whatever. Doesn't have any monetary consequences in Europe (at least no harsh ones), and Germany covers up to 6 weeks of salary.
Any way, you can forget about getting goods references. Not that would get them anyway, with an employer like that.
Sure, now you start thinking of backups.
House-of-cards core infrastructure is a sign of negligence and OP should have reconsidered working there a long time ago.
If you aren’t running drills then transparency may not be in everybody’s self interest and someone somewhere is covering up for the fact that they only half know what they’re doing.
Keeping the people who know how the system works is really the last line of defense.
From, the point where such last person left without any compensating mechanism in place, the clock started ticking
There's so many things wrong with the scenario described, that I can't even begin to talk about it.
My experience is that folks never want to talk about backup or DR, as they are expensive, and the idea is that they should never be used.
No one ever has a problem with insurance premiums, though...
Thinking you're fucked is unhelpful, because you start talking about resumes and expecting some deus ex machina to appear and resolve the situation (for good or ill).
Also, make sure everyone knows you are going on vacation once you resolve things.
If anyone happens to know of a career path or company that does that sort of work I would be interested to hear about it. Bonus points if the pay is half decent.
However, not only is not everything a true positive, probably only about 0.001% of things are true positives, among a sea of alerts and reports and dashboards across myriad systems. Some coming from your SIEM, some generated by security appliances and products, some from internal employee reports.
An ideal place will have people who continuously work on trying to reduce alert fatigue and false positive noise - but, in practice, at most big companies it's probably like working at a fire station and getting hundreds of dispatch calls per hour, every hour, every day, each about a potential fire at a different residence. And then you drive up and see they just used the stove for a few minutes or a character said the word "fire" in a TV show they were watching.
But you have to urgently show up every time no matter what because, occasionally, the house actually is engulfed in flames and might be on the verge of igniting the whole town.
And for some reason warehouse managers seem to swear by it, so it still gets a decent number of new customers - meaning those customers need implementers. And since it's a giant pile of hacks, the average deployment needs a whole lot of customizations - so more implementation man-hours, and a steady stream of maintenance man-hours. Thus, I get pinged every once in awhile for some long-term contract. Too bad they didn't ping me when I was actually looking for work last year, else I probably would've accepted one for the hell of it. Still tempted to; would be an interesting side job, albeit probably soul crushing.
It is... interesting. I have been doing this over last 4 years. The biggest problem, as I pointed out before, is the disconnect between what companies claim they would do to fix the problems vs. what they would actually do. So ~90% of selling is repeatedly explaining the same things to different people whose jobs are in the eminent danger of being restructured/deprioritized/eliminated if your services are successfully engaged.
<multiple layers down, Jack, head of IT in charge of backups>: I'm uncomfortable sharing this information with the outside party. I will need to get the approvals for <blah blah blah blah>
...
CEO: Yes, tell them to do this.
...
Jack, head of IT in charge of backups: Ok, you can have this access.
Consultant: Great. <comes back ten minutes later> It says I'm not authorized to perform this operation. It is a blocker, could this be fixed so I could continue?
Jack: This was not in the scope of authorization that I have received. I'm uncomfortable providing this level of access without additional authorization.
<repeat>
When you’re in this role your goal and job is to build relationships and help people. That includes Jack. The job is getting done with or without Jacks help. So he can either jump onboard or he can be removed and someone else can be added to give access.
In the end people like Jack benefit nothing from being gatekeepers. Because the third time I have to ask for permissions to be changed - I’m asking for Jack to be removed from the task and someone else to be put in place.
When you are in a firefighting role, your job is to fix the fire, not appease Jack and build relationship with him, the person whose practices created the situation to begin with. Unfortunately, even in those cases the highest level of stakeholder who engaged your firefighting service may not be willing to tell Jack's manager or Jack that he will do what he is told or he will be removed.
Jack's gatekeeping is what keeps Jack employed. The company's ultimate bosses will need to make decision if they want to remove Jack and solve a problem or if they do not care that much that the problem is quickly solved. In the vast majority of cases no one in the company who can fire Jack wants to be seen as a bad guy.
The guy did very creative reporting though :-)
And the problem was not fixed, of course.
I tried to help him as much as I could, but at some point, the thing is so intricate that you basically give up.
If you are an external consultant (with no prior knowledge of the company's systems/processes/hierarchies) brought in to "fight" a particular fire, how do quickly get ramped up to a level of knowledge where you can analyze the root cause of/recommend a fix for a particular "fire"?
To be fair, the number of fires is finite, new ones are just getting started faster than they are put out.
Sure it is fun, but the stakes can be high and the stress gets piled on.
It starts with congratulations all-round. You are a hero! Good job! Nice save! Your boss's boss's boss's boss comes over and thanks you personally. Take the rest of the week off!
Later, its people asking when will you fix it? Why haven't you got this back working yet? Didn'y you fix the same thing last month? (no) We really need you to fix this before 5pm or the TPS reports wont go out, and the management will be pissed.
Failures become normalised. They get reliant on people doing heroics. People forget that the systems are crap and need investment, and start to rely on you being there to fix it, and if things don't get fixed then it is your fault the TPS reports didn't go out, not tech-debt/lack-of-investment/bad-design/whatever.
"If you do things well enough, nobody will think you did anything at all."
>We can't access the backups. Even if we could, we noticed two years ago when trying a restore from the backups, that it doesn't work. Booting the restored server leads to a kernel panic we couldn't figure out. Management said we don't have enough money to fix any of this.
It's often even worse. Many years ago I was responsible for backup a DEC Unix system. Asked for funds to conduct a disaster recover exercise to find out if our backups worked and was turned down even before we had tried to work out how much it would cost.
I often wonder what proportion of backed up files and systems are actually in a state to be restored and have the necessary tools and expertise to do it in a timely manner.
An even worse problem in some places, including the one I worked at, is a tendency for users to assume that backups are kept forever and for the management to totally neglect archiving of important documents such as specifications, drawings, and design calculations of products that have an expected lifetime of over fifty years.
I had to inform several users that I could not restore the file that they had only just noticed was missing because it was last seen more than two years ago and presumably disappeared longer ago than our longest backup cycle which was one year of monthly full backups.
If you can afford 12 months of backups then you can almost certainly afford >=12 years of yearly backups by buying a new set of monthly tapes once a year and taking a set out of rotation and into archive.
(GDPR and earlier laws, etc.)
The best way to delete user data is to encrypt it all with per-user keys so they can be shredded immediately upon request. Backups of user keys themselves can be very short-lived since the keys are static and the lifetime is scoped to GDPR/other laws. Then the encrypted data can be archived indefinitely, so long as encryption keys are never stored in those archives.
Typically, this kind of decisions are made more or less implicitly during long meetings about priorities, often without any minutes. Managers will presumably try to hold (senior) engineers accountable as they had not made clear enough how important that was.
I obviously don't know who is right here.
Up until now, they didn't know what 'backup not working' meant. Now they know.
Not being bitter -- there was a failure of communication from both parts.
While you can and should be the cool head in the room, it's a route to burnout if you end up being the hero who has to keep saving a dysfunctional organization and management from the consequences of its own failures. By all means do your best fix the immediate problem but get the hell out as soon as you can.
The same dysfunction that led to disaster after disaster is likely to lead to very little reward (if any) for actually being the hero, also.
In war room after war room there has always been at least one person who would be categorized as 'disorganized' by their coworkers. When thrown into a stressful situation they either don't change or become more focused, while the 'organized' people crumple like tissue paper, so Mr Disorganized ends up either fixing the problem, keeping everyone on task until it does get fixed, or insisted on some process/tool improvement that made the problem easier to identify or fix.
https://www.additudemag.com/benefits-of-adhd-crisis/
I've been diagnosed with ADHD and I'm not sure I'd qualify as having that trait. So I don't believe this a general sweeping statement you can make for everyone with that, or if your boss even had ADHD.
But maybe check this out and ask if it rings any bells: https://www.nhs.uk/conditions/attention-deficit-hyperactivit...
I have ADHD and recently started an SRE squad in our company since I tend to be the go-to person to fix any production issues.
I had never really thought of a connection between those two things until now.
But knowing Reddit, I'd give this a solid 50/50 chance of being fake.
Not everyone is AWS. There's a lot of obscure software providers nobody cares when they're down.
The only actual backup is one that has been tested. With the cost of hardware, buy 2 or have an agreement with your vendor to have a machine ready to receive a backup.
All computers except one, which due to a power failure had been offline the whole time.
This computer had the critical DNS information needed to restore the network.
Only problem: it was in Africa, in a country that required Visa which would take weeks to apply for and receive. So someone from the African office had to bring the hard drive to an airport that someone from the London office could travel to and pick up the critical hard drive, then bring it on a plane back to London.
A sweaty and nervous trip, I can imagine.
Whole story here: https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
0: https://www.zdnet.com/article/ransomware-the-key-lesson-maer...
1: https://portswigger.net/daily-swig/when-the-screens-went-bla...
https://darknetdiaries.com/episode/54/
https://malicious.life/episode/episode-106/
Things I would do that I haven't seen mentioned yet:
- Grab images from the relevant servers to have current backups before messing with anything.
- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.
- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.
- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)
- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.
- See if the customer domains can be collected from billing data / some email store of notifications.
(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )
> All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in the ERP which is now busted
I find this to pretty common in many setups where the engineers don't focus on simplicity and removing layers of abstraction. If the workforce is young and inexperienced, over-engineering tends to happen everywhere and you end with situations like this.
I have seen worse in my years in the industry, that's for sure.
https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
> We tried really hard to make management aware and actually succeeded in that, only for management to follow up and say we don't have enough money to fix it, and they understood the possible outcomes.
but don't redo the same errors :)
Everything they list that's an issue seems to revolve around the fact that the employees with domain knowledge for the system have left the company.
Reach out to them and hire them on as contractors. If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
That last part may not be necessary though, as customers will jump ship and the company will probably sink anyway.
Depends, they might have a captive market or a customer base that's non technical and doesn't understand how bad this is.
If their systems are like this, chances are their offering isn't anything groundbreaking and the competition already provides a better service for possibly even cheaper, so the customers who are willing to jump ship would've done so long ago and the fact they're still in business suggests they have a complacent customer base that's likely to stick to them even despite this incident.
The same reason this technical debt was left unchecked for ages applies to customers. The task of migrating to another provider will most likely rot forever in a Jira board somewhere and they will keep paying their bill in the meantime.
I know of a hospital which is still refusing to migrate off of a system which is not supported for a decade. Pay more money? 3 people worked on it: Adam had a stroke, Bob retired with enough money, Charlie left the country.
The reality is company execs like to talk about fixing problems with money but rarely are they actually willing to fix a problem with money. In my career I dealt with the exact three companies where execs decided to throw money at the problem bypassing all the fiefdoms and bureaucracies created to avoid rapidly solving problems using money in favor of slow moving, don't step on Jackson the IT manager's toes, approaches. All three solved their problems, destroying the local fiefdoms in a process.
In context of this case - if they don't have money to fix their backups (https://news.ycombinator.com/item?id=26539988) they likely couldn't drop lots of cash without a serious credit line.
A new system isn't going to do them much good when they don't even know who their customers are. I doubt any customer is going to hang around for a year while you rebuild your system from scratch, ignoring the fact that if they're an MSP the customers may not even have their own copy of the data.
All of that is ignoring the likely contractual obligations they're likely under to maintain the data on their systems.
>Also it's a public hospital - for a fraction of $1M they could get people who's only job is to replicate that system running around with pen and paper - they wouldn't have $1M to just drop on a support issue the same day.
Hospital? He said they're an MSP on reddit:
>Yeah we're an awful MSP. I've been trying to find a new job for quite some time.
https://www.reddit.com/r/sysadmin/comments/ma4mwl/the_absolu...
This thread is about a different case.
The ERP system was a nightmare, though, and the company was difficult to work with (key, long term employees). It was a system I inherited, built out of garbage, that I tried to improve. Before I left, I found a very experienced contractor to take over, on-boarded him, and stayed as a support resource for a couple months. Six months later, he's retired and I'm getting panicked phone calls. I valued (and still do) my current job far more, even though I would probably have done better financially in the short term with the alternative.
A few years ago, I heard they migrated to a new ERP at long last.
After 25 years, I'm an expert at making small, efficient software tools to fix local IT workflows. God knows, in the Fortune 250's I've worked at, there are endless opportunities. And, while I've written many solutions, I've seen this political game of thrones thwart my efforts many times. This is how you get the kind of 20-year legacy mess described in the original post: People sacrificing long-term success of the company for short-term success in their careers.
I guess there are 2 kinds of people. Those that think this is just how the world works, and find this to be a perfectly-cromulent way of getting through it, and those, like me, who find it odious. Luckily, I now work for people like me. We do what we can.
He had retired, but took another 12 or 18 month contract flying around the country servicing these things. Apparently they asked, and not wanting to do it he quoted what he thought was an insane "go away and leave me alone" fee, but then they said yes. (I have no idea what that fee was or what multiplier he applied, though it definitely brings to mind patio11's standard advice [1]).
[1] https://www.kalzumeus.com/2006/08/14/you-can-probably-stand-...
If the long-gone ex-employees hated their employer and would be happy to see if go out of business then you're in quite a bit more trouble.
Sorry, I though I was on the Romania subthread.
That's certainly a simple solution.
The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
> The problem with simple solutions is that they are near guaranteed to not actually be feasible or realistic.
This. There is no amount of money anyone sane can be given to stay in Romania. Most recently i’ve read they (hospitals) were tying covid patients to their beds.
Couldn't help but comment on this as some of things i discovered are shocking for an eu member country (or any country for that matter), such as EU leading human trafficking, police corruption, collapsing health care, and the list goes on.
But sure, the simple solution is to give people money, not the constant abuse taking place at work and outside that makes people want to flee that place.
Companies paying more (offering more at least) would begin to slow this domino effect, which raises standards of living and provides more funding for the government to provide services and ultimately pressure for increases in quality of life by the voting public. Everyone wins.
Ultimately not everyone can emigrate - it's an impossibility. The brain drain will continue to be real, so it's incumbent upon these companies hiring to raise their wages and retain talent.
It is really not as easy as go "this is the exchange rate" and simply use that to compare pay in different countries.
Sure, farmers could pay $10/hr, but unless your Doritos were $10 a bag, they’d lose money.
If nobody wants to buy $10 doritos, that means the product can't exist in a market without cheap labor.
"Exploitation" is a relative term, some people would rather be exploited in the countryside of a first-world nation rather than the countryside of their homeland.
It’s all relative.
The Australian national minimum wage is $19.84/hr.
Getting paid about a quarter of minimum wage is very clearly exploitation. There may be other benefits for actually working in Australia, but that's not really relevant here when you're trying to attract native Australians to work for you.
[1]https://www.ebooksread.com/authors-eng/andrew-carnegie/trium...
And it's not just cheap labor, it's labor exploited under pain of deportation.
Edit: that was just based on my general memory of non-perishable food economics. But I googled it and found this (sorry, quora) from someone who would know, a 2 oz bag of potato chips costs 15 cents to produce and bring to (retail) market. Of that, a fraction is potato prices.
https://www.quora.com/How-much-does-a-bag-of-57g-of-Lays-act...
Edit: fix Verizon math.
Same reason I avoid organic whenever possible. We don't need boutique food.
I am telling this as an engineer working in the Canada's oil and gas sector. Since 2014, salaries in our sector have decreased and firms are struggling. "Just raise the price of oil" is not an actionable advice. The price of oil is not ours to set. It's the market's
https://www.lowtechmagazine.com/2020/04/fruit-trenches-culti...
You're absolutely right that perhaps not everything should be grown everywhere, but I strongly believe that a country should have a local source of all the calories and vitamins / minerals they need. In Canada, for example, maybe they encourage farmers to grow greens, brocolli or potatoes instead of citrus for vitamin c, but they absolutely need a source.
Australia does not use this approach, and instead seems to have pushed other countries to eliminate their own tariffs for the TPP.
But imported products don't fly right into the country. There is customs, and denying/tariffing them is a policy that countries are very actively using. So at the end of the day, it's a political decision.
Edit: On the other hand, if you are exporting stuff (oil for Canada), then you can't really do that for the external market. Might explain why the government will let some sectors run with illegals (to remain competitive); the alternative being their ultimate demise.
[0] https://en.wikipedia.org/wiki/Chicken_tax
[1] https://www.npr.org/sections/money/2017/01/25/511663527/epis...
Of course, you might get pilloried for the artificial scarcity, but hey... nobody ever likes the capitalist :)
Actually farmers could set any price if they colluded to manipulate the market. People will buy food at any price and everyday.
[0]https://www.cato.org/policy-analysis/candy-coated-cartel-tim...
USD $5/hour is what I earned as a teenager working the fields by hand (picking corn, beans, and fruit for sale same-day at the farmers small stands throughout the area) in the early 1990s... That's insane. Is that USD $5 or AD?
Now, outdoor work with a high injury rate is a different story.
What I've seen first-hand in some farming communities was closer to $10AUD/hr, but that is still far below our actual minimum wage of $19.84AUD/hr.
The workers were detained, deported, and so on.
After several raids, not so much as a misdemeanor was levied against a single person, nor a fine to the company in general.
[1] https://en.wikipedia.org/wiki/Postville_raid
"There was false testimony and willful manipulation, and that makes this a shocking case of prosecutorial misconduct."
"An unprecedented letter, recently sent to Kevin Techau, U.S. attorney for the Northern District of Iowa, signed by 107 former high level Department of Justice officials, FBI directors, federal judges and prestigious law professors attests to the extraordinary prosecutorial misconduct which exacted the most possible punitive sentence for Rubashkin"
https://www.desmoinesregister.com/story/opinion/columnists/i...
Trump commuted Sholom's sentence in 2017, apparently.
I'd suggest a different phrase than "prominent Jews"; the family are "ultra-Orthodox Jews of the Lubavitcher hasidic movement" not representatives of some sort of Jewish privilege in general.
It is also difficult because of the two tiers of labor. A farmer will try increasing wages by $1/hour, and still struggle to find enough workers. Wages would probably have to be doubled before new workers—Americans—would start to enter the industry.
I'm assuming you're implying a conversation about China and their manufacturing capabilities. But worst case scenario short of nuclear war, assuming US-China burns bridges and somehow for some reason now US is fully unable to import Chinese manufactured goods; how does this eliminate the US and all other countries from investing in an area that has now become more profitable? You could argue that prices would increase, but then again you aren't owed cheap manufacturing, and subsidies and tariffs are paid by either the taxpayer or the consumer (which tends to have a knock-off effect on the rest of economy, anyway). There was never a free lunch.
That's easier said than done. The problem is that in order to reform immigration you have to do one of two things:
1. Start treating people fairly, with the result being higher food prices, which will make some of your constituents very unhhappy
2. Codify the current class structure into law so that you can continue to exploit cheap labor legally. The optics on that are really bad.
Exploiting people off the books is the path of least resistance for everyone, including the workers (manifestly so, or they wouldn't risk life and limb to be here).
This is a political talking point that has been repeated so often that people believe it is true.
During the recession of 2008, there were stores on local TV news in many places showing unemployed Americans waiting in long lines for the opportunity do do those jobs that the politicians kept saying no American wanted to do for those wages. Meatpacking plants, especially, had more than enough Americans to choose from who were willing to work for low wages.
Because these things happened in "flyover" states, they were only very rarely shown on television on the coasts, and so the meme stuck.
A similar oft-repeated lie is that American farmers are all a bunch of Republican hillbillies who hate brown immigrants.
Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
The more you actually go places and do things and meet people and talk to them, the more you realize that the words coming out of the political organizations on the coasts are mostly for their own benefit, and do not reflect reality.
That and Republicans have grown way more hostile to immigration since 2010, so it's an interesting anecdote but not sure how well it translates across the US, where polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not.
source?
When you say it's an "educated" guess, educated how? It only seems to reflect the usual bigotry that comes out of the coasts.
polling shows farmers are overwhelmingly Republican who tend also tend to hate immigration legal or not
If you're going to make a statement that sweeping, you really should back it up with credible sources.
I lived first 28 years of my life in the rural south and now live in the midwest.
My entire family and most of my friends growing up are the people you claim I'm bigoted against...
"If you're going to make a statement that sweeping, you really should back it up with credible sources."
The fact that farmers are generally far more conservative and republican than the median voter or average American isn't exactly some shocking new insight.
You seem to have never spent much time around these folks or have done much research on them if this comes as a surprise to you.
https://www.washingtonpost.com/news/wonk/wp/2015/06/03/why-y...
Are these just bigoted facts "out of the coasts" too:?
Your second point about farmers trying to increase legal immigration in 2010 clearly indicates most Americans won't regularly do this work for the wages being offered. Why else would farmers be looking to hire immigrant workers just 2 years after locals were lining up to get these jobs?
While it would be nice if there was a way to get enough immigrants working legally to fill those jobs that doesn't seem to have happened. Short of another large economic downturn I don't see Americans working those jobs either. That leaves people who are working in the US illegally to fill the gap.
> Around 2010, I attended a conference in Seattle for farmers in the Pacific Northwest. The main topic of discussion was how to increase legal immigration from Mexico so that they could have more farmworkers to pick the cherries and apples and mint and whatnot. These people were talking about goals of making people from Mexico and Latin America U.S. citizens in staggering numbers.
Farmers are, for the most part, single issue voters: Whoever has the best plan to support agriculture and small businesses gets their vote.
How do you determine from TV that the people lining up for meatpacking jobs were not undocumented immigrants?
Like, it is generally accepted there aren't any in "flyover states" or something?
I never really thought of it until this moment, but you know how within 100 miles of any border, your constitutional rights are somewhat compromised so DHS can operate? That ought to mean a pretty substantial incentive for people who are undocumented to migrate inward. So I would bet they do.
[1] https://www.ers.usda.gov/topics/farm-economy/farm-labor
Companies aren't going to say "I guess we're inefficient and should just die." They're going to fight to survive and thrive, which is what you're seeing.
Then again, some companies ARE just cheap. You never know.
They said the standard is 30 minutes per room per day. I figure at a very generous $50 per hour (100k / year) it should run $25 to have my room taken care of. Now that doesnt cover the shared costs - doing laundry, cleaning the pool, front desk, etc... but my room was something like $150 per night. How the fuck can they not find people to clean rooms?
Answer: they're too cheap to pay a decent wage. But why? My suspicion is that all these places are actually leveraged far too much and paying rent/interest/dividends so much they can barely operate.
I suspect low interest rates and other incentives to "stimulate" economies are actually causing these problems.
sorry for pun. But that's not always a solution. Sometimes employer is greedy. Sometimes his business just does not allow to pay enough salary.
Engineers not make money.
You'll continually be amazed at the human tendency to double down instead of admitting a mistake.
I would anticipate a mix of: the people who know where the right bodies are buried, the right incentive (yes, that includes money), but also people to look over the entire program, a designer to make sure it is designed correctly, an organized person to make sure it is properly documented, and someone to make sure it is adhered to in the future.
Asking because, while, yes, throwing money at the problem might be a solution, I think the obvious bigger issue is how does one allow themselves to get into this situation?
Or rather, how does one avoid getting into this situation. Because it seems to me that the software industry of late has become so mired in esoteric layers of abstraction that this feels less like an outlier scenario for a mismanaged company but more like an inevitable one for many otherwise well managed companies in the industry.
Documentation?
you handle this by having and exercising a disaster recovery plan and having (and listening to) competent people that would spot this issue in a jiffy.
Significant documentation is also not something easily written in slack time. With any non-trivial project good documentation is a full time effort. It also needs to be updated as the project evolves lest it become out of date and incongruous with the behavior of the system.
There's dreams of in-line documentation but that's only going to cover individual methods and classes and not necessarily whole modules or major subsystems of a project. It's also not going to necessarily cover in-line documentation from other parts of a project in different languages e.g. comments in config files.
There's lots of technical ways to help documentation but there's no replacement for documentation being a high level project goal with resources allotted to it.
For example, in code: good naming, simple structure, unit tests, and useful comments all can count. A set of easy-to-understand unit tests that cover real scenarios is much better than even the greatest documentation ever written.
A fully scripted build and ideally also deployment (CI/CD pipeline) beats a checklist with tens of items that have to manually followed.
An issue system with good descriptions of changes (including why and what), reproduction steps, expected outcomes, links to related issues, etc also can go a long way.
All of this stuff has the side effect of making day-to-day work simpler, overall, while actually just being a part of that work.
1. Naming things is hard. It's a bit of a joke but good descriptive yet usable naming is hard. It can get harder after refactoring or adding in features you didn't expect the last time you were in the module.
2. Project structures start out simple. Then you need to add some feature or fix some major issue with a due date of last week. As a project gets bigger the simple structure of yesterday might not serve the needs of today. Resources for refactoring are usually somewhere around the priority level of "Good Documentation".
3. Comments need maintenance like explicit documentation. Tight schedules lead to "# TODO add documentation here". Then you get the assholes that believe comments or in-line documentation is something you put in check in notes.
4. A "unit test" that covers a "real scenario" is not a damned unit test. It's an integration test (or however you label it) and it's not likely something your CI/CD system can run on the regular. If a real scenario is touching a production-load DB or something it's a big deal to run the test. Tests like that need resources to run and non-trivial resources to write. You're not just making sure a method returns a float or throws an error correctly.
I'm not saying your suggestions are bad or shouldn't be used. The issue is management rarely sees any value with documentation, tests, or even code cleanliness. They're rarely incentivized to care about those things so they don't. They get their bonus on feature checkboxes or hitting a deadline. At the end of a project they'll schedule a "documentation" sprint or some dumbass thing that accomplished little useful output. It's worse when you've got a "move fast break things" chanting moron doing the scheduling.
The specializations common on a team even 20 years ago was much closure to the "Surgical Team" model from the mythical man month than what we have today.
At a lower level, it's allowing individuals to become knowledge silos (or having a bus factor [1] of 1); lack of documentation; not having+testing disaster recovery plans; neglecting systems, code bases, and platforms; letting technical debt accrue endlessly (or worse: not even realize that's what you're doing).
How to avoid? Make sure there's a strong, competent technology leader with a vested interest in the company succeeding, and empower them to make the changes they need to.
What should that person be doing? Making sure all the stuff I just said doesn't happen.
[1] https://en.m.wikipedia.org/wiki/Bus_factor
I guess any company where leadership got there without progress on the technical side. Law firm partners get there after being lawyers. Consultant partners get there after consulting.
But if you are the CEO of an IT company without ever having to code, then you may never know the things you must take care to prevent a washout.
IMO, the problem is technical debt. Most likely, in the majority of these cases, the engineers working on the system knew perfectly well that it was a mess, was very difficult to learn how to maintain, and many parts were dangerous to modify. Most likely they told management multiple times that this was a problem and what the consequences of it would be. They would have refused to budget resources for improving it, so everyone either saw the writing on the wall and quit, or gutted it out and learned how to manage the system eventually. Maybe the system limps along for a while longer, but it's only getting worse and the knowledge needed to maintain it more siloed and specialized. Eventually you lose a key person, something breaks that nobody left can figure out, or maybe the whole thing just implodes like the main story here.
To avoid it, you must understand the pattern, hire good technical people, and give them the resources needed to improve the system in an orderly fashion.
The relevant part of the article:
> To anyone outside of the ballpoint pen manufacturing world, it might seem hard to understand what, exactly, is so surprising about this development. China already produces 38 billion ballpoint pens a year, according to China Daily, which is about 80 percent of all ballpoint pens in the world. That's a lot of pens, but there was a catch: China had long been unable to produce a high-quality version of the most important part of the pen, its tip.
> The tip of a ballpoint pen is what makes it a ballpoint pen. At the tip, a freely rotating ball is held in a small socket which connects it to an ink reservoir that allows the pen to write or draw lines. Manufacturing a ballpoint pen tip that can write comfortably for a long period of time requires high-precision machinery and precisely thin steel, but for years China was unable to match those crafted by foreign companies.
Mentorship and documentation.
You need to set aside time and resources for these two things, and make contributing to them worth something in the performance review.
If all you do is count JIRA tickets closed, you'll forever end up in this situation.
Usually, since admitting that they were stupid to let all the knowledge of the system walk out the door requires more emotional and intellectual flexibility than managers who let this sort of thing happen possess, the biggest obstacle isn't even the money... it's getting the managers to admit they're idiots.
> The company that built this ERP solution went bankrupt 6 years ago, and we don't have the source code. It uses PostgreSQL and MySQL, and also has a built-in key-value database for which none of us has any credentials. We don't have the source code for this software (it's built in C and delivered to us as compiled packages..).
Sounds like finding their old employees wouldn't help. They need to find old employees of the company that wrote the system and hope they kept a copy of the source code.
Which from a legal perspective might be fraught with peril, at least in theory. In practice, if the company folded then it'd depend entirely on the litigiousness of the current owner of the copyright, which could range from "nobody" to "some holding company consisting almost entirely of lawyers chomping at the bit to eat you for dinner".
Anything but pride, I guess.
Years ago I left a rather awful company - I left on great terms with my boss, though I hated the company. I was the last person who knew how to release X product. While I documented everything, after I left, the wiki lost its mind, backups apparently weren't a thing, and the director of IT moved to Bermuda.
Fast forward 6 months.. a contractual obligation existed to release the software. They still hadn't successfully built (yup autoconf, make and friends), let alone know where to build it. I got the call you always dream about, 30 days prior to the release being due.
My boss asked me to do him a favour - I reminded him that I quit due to a lack of promotion and raises. He asked what I want, and I asked for 125K, which clearly he scoffed at. Then I reminded him that it was a 10M deal they'd signed, because we had an all hands. The CEO got involved and screamed at me for 'torpedoing the company'. I walked out and calmly informed them both along the way the price was now 250K, non negotiable.
I ignored every single mail (and legal letter) sent me way. With 10 days left, they agreed. I had the company deposit a cheque in escrow with lawyers of my choosing. We signed contracts, and I did my thing. Four days of work. It was well worth it.
Dougie, if you're reading this I hope your cringing.
That an overworked employee who had already left the company would stick up for himself? Or that a company would go to such efforts to try to manipulate a former employee?
I realize my tone doesn't hide my feelings, but even if we disagree I'd still be curious to hear more.
https://en.wikipedia.org/wiki/Bus_factor
Walking away from a toxic situation is not "psychopathic." It's common sense and if we all did it more then the psychopaths who expect us to stick around through it all would finally learn how to treat other people with respect. It's a job, not a marriage.
Sadly, I've seen this. Not first hand, but it happened to a friend and I have no reason to doubt his version of the story.
"We tried to hire back an employee to build something that we didn't train or try and figure out ourselves and a deadline forced us to pay a large lump sum to that employee".
No matter how you slice it, that's really shitty planning on the businesses side. They should have spent the time and money learning how to maintain their shit. By not doing it, they paid a huge price.
Luckily for him they decided it's easier to just pay the fee.
Because here's the thing, even with how weak US labor laws are, one thing they don't allow is for a company to come back and sue your for incompetence. So long as they didn't go out of their way to sabotage things, they are free and clear.
I think in general all of my comments on this thread have been horribly misunderstood - I was just trying to say that if the company was scummy, instead of paying him the 250k fee, they probably could have sued him instead. I'm not saying that should have done, I'm not saying it would have been the right thing to do, that he deserved it.....literally nothing of the sort. Just pointing out that the company could have made an argument against him in court. As many other people pointed out to me, it would have almost certainly failed. But I don't understand how you can deduct where moral considerations are on my priority list from this?
Especially ex-employees who've been gone 6 months already.
Shouldn't be hard to find a no win no fee ambulance chaser to happily take them on and get costs awarded and damages as well...
- if he can prove the threat, the company will be in an even worse position
- given the rather lacking precedent (as pointed out by the sister comment), there's a good chance the suit would be thrown out pre-court and therefore be rather cheap
- given that the OP seems to be a valued engineer, it is reasonable to assume that he has the f-you-money to fight this out
- even if won, the drawn out battle would take far too long to fix the software in time and, given that they then lost a 10 million contract, they lack the motivation (and maybe money) to continue
- lastly, even if they won after loosing the contract, they'd probably not be able to extract much in damage payments
So it's a rather empty threat.
Every negotiation is about the best alternative the parties have, and it's generally harder in real life than in fantasy because you underestimate the alternatives to a deal.
Applying maximum leverage has a way of creating "out of the box" alternatives. And vindictiveness.
I have no need to judge it if it were true.
This is exactly the free market at work regarding labor. Have a problem with the free market if you like, many of us do but do not accuse this poster of criminal action equivalent to a protection racket.
To clarify, the market would likely not allow this specific work to be sold at this price normally. But the buyer wasn't just buying specific work they were buying specific work to be completed by a specific date. The work in isolation does not have this value but the work completed by a certain day does. Companies use this rationale all the time charging more for expedited services.
It's especially common in construction and manufacturing where costs double or even triple when expedited delivery is required.
The way the story is told the price doubled after and because of the CEO screaming.
The market has nothing to do with that except the good old "The fair price is the one that god paid, uh.” I suppose.
I'd be inclined to believe that if there were some other providers but as it is told working with OP was the only solution. And OP's first offer was 125 000. Two screams later it's 250 000 ? It has nothing to do with economical theory or free market interpretation.
And yelling, made the prise jump closer to market value.
I find this explanation (from top comment) more in line with how the world spins:
> If they left under bad terms because the business was a bunch of dicks, expect to pay 10x market rate. If this is truly "fix this or the business is out of business" - then it shouldn't be a tough decision to make.
So, “being a dick” has more to do with price raising than any free market justification (which I am okay with, just don't try to convince me it's the only thing at play. The offer would have been 250k from the beginning then. Or 500K to get 250K, whatever.).
Any small or independent contractor in any business will eventually have a blacklist, or a list of troublesome customers who get charged extra. Most of my contractors charge me less because they like me (and I pay on time). My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole; the contractor has decided that it's only worth dealing with the man's unpleasantness if there's more money to be made. Apparently the man has been blacklisted by other HVAC companies, so he might be in a tight spot, but whatever the reason, he ends up paying a lot more money for the same service.
This is a completely market-based mechanism: the troublesome customer has limited options (in this case, maybe only one option), and is thus obviously disadvantaged in negotiating price. Regardless of the myriad motivations involved in price discovery, it all leads to the same place: if the price wasn't worth the service the customer was to receive, they would not have paid it.
Honestly, this isn't mysterious or profound. It's economics 101.
Yes.
> It's economics 101.
Not even that. It's payback, that's all it is. There's no need to call for a more elaborate explanation for what was at play here. Not that I disagree with it. It's humane nature.
> My HVAC contractor charges a particular customer double just because the customer is a rich entitled asshole;
Like you say. No need to even know about market dynamics to (wisely) choose that option. Because the other guy is an asshole.
Second, you speak as if his claims were irrational. The way I see it, however, you would set for 125k (probably much less judging from what you write) while he, the sentimental fool, scooped up double the money. Therefore, your decisions, which you personally claim are rational, would have left you with half the money the "irrational" player would get. Seems to me, the "irrational" is actually the rational one, as he netted more profits. Economy and market are not math games in a vacuum, and players are not formless ideas.
The way I see it you are putting words in my mouth and I am definitely not interested in having a discussion like that.
I never insinuated OP was a sentimental fool or wrong to raise the price. I am calling bullshit on the free market and econ 101 as an explanation of why the price got doubled and call it like it is: it's payback from OP because they were dicks to him first, everything else is pedantry and useless justifications.
edit: if anything, OP could be considered a fool since he obviously gave an under the market quote as his first proposal. Had he started with 250K from the beginning he would have gotten more since there likely would have been screams anyway and he would have risen the price all the same.
And yes. I completely agree with your edit and said as much elsewhere. Their acceptance of 250K showed that the OP under-priced their services at 125K.
And yes. Free markets produce monopolies. Especially when time is important. As I said above, the critical good here was not the delivery but the delivery within a certain time.
This is literally the definition of price discovery in a free market. If the price was not worth the service the company was to receive, they would not have paid it.
The fact that they did pay it clearly indicates that the first price was far below what the "market" would bear.
[0]https://www.youtube.com/watch?v=ojTKkfgvwvU
I think so too, however maybe better play safe I guess, and be content with 250? The negotiation partner seemed a bit irrational to me
Anything distorting those conditions - e.g. there's only one seller, only one buyer, a time constraint, a skill constraint, an information disadvantage, will distort the pricing as well. One trade isn't enough information for a market to exist or to settle on a fair price. I agree there that the market has nothing to do with it.
But I disagree with your implication of "the fair price is the one paid" is somehow unfair in an ethical sense. (I also think OP is in no way obliged to be "fair" in any ethical sense to the company, but that's another matter). The company got a $10M contract for $250k, versus losing the contract, is that unfair? If that really was the pivotal factor between losing the contract and not, the company ought to benefit by spending right up to the point where the contract becomes a money loser, to not lose it. Would it be unfair for the company to get an $8M contract by paying $2M? Or a $2M contract by paying $8M? Is OP obliged to help the company make as much profit on the contract as they can, by charging them less for his services?
This is non-sense. Those conditions specifically define a market and make it distinct from any other. To say a market with any of these constraints is "distorted" is to say all markets are distorted and render the term meaningless.
Also, I never said this one trade signified a market. My point was that this is a consequence of free markets. You can make any single trade "irrelevant" by choosing arbitrary criteria by which to separate it from its market. The company could have certainly hired another engineer in hopes they could complete the work in time while running the risk of blowing the deadline. But it was freely and fairly worth at least 250k to them to guarantee the job could be done correctly and on time.
This kind of thing is such a common part of the free market, that there is a whole aspect of contract law that pertains to ensuring companies don't get into this position. It's why contracting completed work is so important. Leaving things partly done leaves you vulnerable to others eating right up to the tiniest margin of your profit.
This person left the company, and documented everything. What should they do, come in and save the day for $15/hr?
If your company were in the hands of incompetents, but still kept going in spite of the incompetents because it had gotten its foot wedged in the door or whatever, and then fell apart in ways you could only scorn, how would you view that sort of situation?
This is pure slander
The company can avoid this if they only care to maintain the documentation / wiki left behind by said person
When they failed to do even just basic proper operations, it's their own fault.
I'm a business owner and a consultant, and I too charge more for these kind of asshole clients, because of the extra stress.
But nowadays I just try to avoid them, it's just too damaging for your own sanity. Their kind of stupidity, self-entitlement, and ignorance is on a completely different level.
<pedantry>It's libel (written) not slander (oral); "defamation" is the generic noun. </pedantry>
It seems that paying to skywrite (you know, small plane making artificial clouds) could be slander but not libel
Citation? By that light, a DVD containing defamatory speech would be libelous, but that's generally not how courts see it.
From the (highly-influential) U.S. Court of Appeals for the Second Circuit, summarizing New York law (which is quite typical): "Defamation, consisting of the twin torts of libel and slander, is the invasion of the interest in a reputation and good name. Generally, spoken defamatory words are slander; written defamatory words are libel. Libel is a method of defamation expressed in writing or print." [0]
https://scholar.google.com/scholar_case?case=601727562995661...
Coding standards, documentation requirements, meetings and code reviews exist for a reason, too. It's not like companies are defenseless.
This is a gross failure of leadership at the company, and the former employee has every right to negotiate compensation for the inconvenience of having to work for them again.
And you identify with this band of miscreants? That's embarrassing.
You may be doing any of that already, since I don't know anything about you except your one comment.
But the answer to your fear of exploitation is avoid making people hate you/your environment enough to want to protect themselves from it.
That's probably what they deserve, if they come asking for a favor and then start getting abusive.
The company could have apologised, and do a lot more ego stroking before asking a favour which they can pay for and maybe throw in an all expenses paid weekend with the SO as a thank you. Not only will you get goodwill but future problems are going to be significantly cheaper to solve. Even if you have a bad break up - as a business owner you should understand the value of mending bridges. It's cheaper than losing sleep at night and losing your steak dinner to a bad stomach.
He left detailed instructions how to release the software. The company managed to screw up so bad they lost the manual he made sure to write on their internal wiki.
I feel for them but they failed to protect themselves.
If you do not have the money to pa for someone's services you cannot complain. You say you have a company? Have you ever let a client get your products by paying less than their value or nothing at all? Work itself is a product too.
For a while, I made it game to light the various communications on fire in interesting ways, but it got boring.
Did they have a legal reason to think so?
The company could say: we’re going to pay you 125 and you could sue us and spend 60k and two years or just settle for that, after he’s already done the work.
The guy could demand the payment directly to his account and then not deliver, and the firm would be in the same situation.
By depositing the funds in escrow, they’re making sure that the money is there and in the hands of people whose only incentive is to give it to the correct party upon fulfillment of the contract.
The contractor knows that the money has been paid by the company and won't be withheld after the job is done.
The company knows that the contractor doesn't have the money until the job is done.
I have bosses at previous companies that I'd help for free today because they were good to me and we parted on friendly terms. One boss helped me move my house once, and I'm not about to charge him to fix some random issue.
I have other bosses whom I'd require a cleared wire transfer of money before I'd lift a pinky to help them, because they were not nice people and I don't want to associate with them or assist them with anything. If someone in this camp approached me, I'd have a story just like yours afterward.
People like that would much rather spend $2.5M on courts and lawyers than let a worker walk home with 250k, or even 25k. Agreeing to a deal like this is an incredible minefield that may blow up on you even years later.