Ask HN: If I Close My Data Centers, What About the People/Jobs Lost?
(throwaway account)
I have a chance to basically migrate 90% of my F50's data centers to commercial providers. The cost savings are awesome, but what about the people currently doing legacy stuff? theyre doing stuff like manual config, and future state is everythign automated as much as possible in the cloud and shutter the data centers.
I think the harsh truth is that many wont have jobs post move, but maybe I'm being cynical. maybe theres a chance to retrain the employees, but even doing that automation generally reduces the workforce.
whats your take? what do I tell current employees? how many can I realistically be able to save from being jobless?
149 comments
[ 2.6 ms ] story [ 236 ms ] threadIt’s a horrible thing to have to let people go but the alternative may be far worse.
With that assumption in mind, if you do migrate and realize that you now don’t need 80% of the people you needed before, it could mean one of two things:
- you are now able to pocket whatever savings you were able get from the migration and either save it or invest in something to potentially keep that money in the economy, just in a different way
- you now have 80% of your people free to do whatever other things you didn’t do because they had to maintain datacenters. That is huge in terms of efficiency gains and allows you to redirect that money to increase the value provided by your company either by repurposing the current workforce or getting rid of them and bringing people you could previously not bring on board due to existing costs
Either way I believe that everybody is better off After the changes.
Efficiency gains such as the one you are describing is what makes it possible for economies to grow and keep moving forward.
Can you imagine if you had to handle cows, pigs and farms to have a BLT? It’s very good that such things are outsourced because that allows for improvement that would otherwise not be possible.
Have you already virtualized your own server and applications in the DC you have?
And realistically, you need most of those people to maintain and configure your cloud service.
It's hard to imagine someone Fortune 50 scale gaining any cost efficiency from going to a cloud service, is this a temporary pricing promotion you're looking to take advantage of or have you done all your homework on the savings? How confident are you on what your cloud expenses will look like?
Google, AWS, Azure would all love a f50 client and I don’t think they’d care too much about breaking even on the costs of the service or even losing some money because of the amount of social capital they earn from the “partnership”.
There are a lot of benefits to migrating to the cloud, in my experience, cost isn’t one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I’m not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.
This is one of the other risks of moving to a cloud provider: You add a new dependency on a single company. If you don't have a multi-cloud strategy that allows you to force Google, Amazon, and Microsoft to fight for your business each time you expand, you're going to eventually end up paying top dollar.
That’s one of the things tools like Terraform, Kubernetes or Mesos were designed for.
If the lowest common denominator is K8S/Mesos, while not using any of the cloud services, you’re doing it wrong.
However, you do bring up an important point, which is a criticism I've always had for the term "infrastructure as code": Unless the underlying hardware is identical in performance, that code isn't really describing/commanding anything deterministic.
This is especially true for databases or anything I/O-sensitive like message queues or stream processing.
It would certainly make "cloud agnosticism" a much less trivial exercise.
At the risk of sounding ad-hominem, I generally find that it is software engineers whose experience is primarily at higher levels of abstraction who under-estimate how much it matter (or, conversely, over-estimate how much software or "code" can just solve such problems).
See also https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
Some cloud providers will be better than others. AWS is still very expensive for I/O intensive workloads. The tendency to revert to mainframe area commitments because a vendor provides a short term discount without factoring in true cost required to maintain the same QoS provided by on-prem is common for executives.
A new layer of abstraction is a major benefit of the cloud.
I see two possible explanations (both of which may apply):
(1) You're getting the cloud servers for a remarkable discount, which, as other commenters mentioned, may not last if you don't put in the engineering effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost as otherwise implied).
(2) You're grossly over-paying for your current environment. Given what I know about the purchasing habits over merely large (nowhere near the size of F50, mind you) companies and, more importantly, the pricing of "enterprise" hardware and support, this seems likely.
If the second is true, then I expect you could gain the same savings, if not more, by implementing your own "private cloud" by emulating the purchasing and ops habits of larger-scale startups or established tech companies (but not yet so huge as to be too custom) who run their own hardware.
You'd still have to retrain and put an end to manual configuration, but I expect that would ultimately be welcomed. Some might be able to keep their jobs almost as-is.
What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. You'd probably have to hire someone who knows how to be frugal like that from an engineering/ops standpoint for architecture/purchasing and is willing to work for such a large company and isn't already employed by FAAMG.
1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later,
2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams.
It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling.
I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.
I would be surprised if this isn't the vast majority of it, with the rest being the over-provisioning you mention later. Mostly, it's tough to gauge, because each effect is multiplicative.
I'll go so far as to say that, somewhere between 10 and 15 years ago, that any hardware support you've been paying to vendors can be considered as complete waste [1]. Worse, it's often a percentage (i.e. multiplier) of original purchase that goes one indefinitely, even though the value of that gear falls rapidly.
Also, (for that 90% you believe you can move to the cloud), if you have any proprietary hardware, you're paying an eye-watering multiple of its commodity equivalent (even considering additional labor, if any).
The above is easiest to observe in storage. There's not all that much mystery around how much the components cost and how reliable they are, so cost estimation for an equivalent commodity solution is easy enough. Today, even commodity/free no longer means "no frills" in terms of software features, but I expect that's less relevant when comparing to cloud providers, anyway.
> you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs
I very much would. Just remember you can smaller VMs yourself, cheaper than cloud providers, bringing your staff up to speed on more modern skills, so long as you don't fall into a vendor's trap.
VMWare's model where the software ends up cost is the same order of magnitude as the hardware is such a trap, but so is software that's free but requires expensive professional services to set up and maintain (OpenStack, historically, though I'm not sure if that's still true).
> and autoscaling.
This, actually, is the rare case where cloud providers can save really save money, if your duty cycle (for lack of a better term) is under 10% of so. Other comments in the thread mentioned this, as well.
Unfortunately, if you never autoscale to (approximately) zero, then you're grossly over-paying for that base load, which can wipe out those savings.
The "hybrid cloud" model addresses this, as well as avoiding all-or-nothing thinking.
Ultimately, if you're optimizing (not necessarily exclusively, of course) for cost and want it to last, you'll have to change the whole organization's way of thinking. Avoiding vendor lock-in is key, be it proprietary hardware, software, or cloud providers. Also, encourage good engineering practice/culture, without getting overly focused on the latest fashionable tool. (A trite example being saving on labor through automation but buggy automation causing an eye-watering AWS bill).
[1] There's some non-zero value to any actual replacement hardware, but that's negligible if considering what you actually got, net of any markup for being propreitary.
The first thing I'm reminded of is https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-.... It's from 2009 but this image jumped out at me: https://www.backblaze.com/blog/wp-content/uploads/2009/08/co...
Let me guess... you're using kit from the vendor at the bottom of that list? :)
No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.
The commentator your/this reply is to points out an interesting aspect:
> What likely would not be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect.
This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.
Well... the landscape is indeed changing, as clarified by the 9-year old article above.
So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity-hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.
(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)
And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, you can build your own servers and network kit. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.
If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy boring, particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depend...
I suspect it's this kind of over-reach that ends up scaring decision-makers off from any in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.
Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is hard. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?
I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.
What we don't know, however, is if those backplanes were actually a commodity yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).
To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.
Hmmm. Good point. (Maybe I scared OP away, heh.)
> Is there an example of any large company, other than Google, who has mastered this?
Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s (2015).
Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].
Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)
As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on https://about.gitlab.com/ (the one from Forrester) doesn't show GitHub :D
Yeah, building own hardware is sadly still within the realm of over-specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.
I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.
I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).
Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.
My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.
> Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior
That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower-end/frugal market.
> I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively
Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly massive scale.
Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a smaller VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.
> (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc
That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.
At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.
Networking gear isn't quite commoditized, but we're close, with open-source/SDN type platforms.
One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).
[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.
a) get the severance package and retire
b) get retrained and stay
c) get retrained and then leave because you really hate cloud ops
d) get the severance package and move to a similar job
We offer a free edX course and then a paid one that prepares you for the Certified Kubernetes Administrator exam.
Disclosure: I am executive director of the CNCF and helped create these courses.
P.S. No job will be around forever. You are not doing you or them any favors by trying to maintain them in inefficient roles. But you can invest the resources to bring them with you to the new world.
They might have to retrain and if they live in a small town they might have to move/commute to find something new however. This can be hard enough for some people that they won't be happy about it.
In the end, I think it's hard to argue that we should keep being inefficient just to save jobs. History haven't been nice to that kind of companies.
However, you don't need to be callous in your pursuit of efficiency. You should try to give these people an opportunity to update their skills and get a new position. You might not be in a position to do that however.
They're going to have tangential skills to the new platform.
Get your network team to learn all about VPCs and networks and security groups and peering and VPNs, train them to be your in-house experts to deal with changes and expansions and to help trouble-shoot issues.
Chances are you're still going to have systems, your systems engineers can be repurposed to better tie in your monitoring tools, your security tools, etc. You're going to need authentication and identity management, and probably have internal people handling it.
If they're motivated to keep their jobs, all it should cost you is a $5-20K/head in training.
That's peanuts compared to the turmoil of everyone thinking they're going to lose their jobs, or the costs of hiring new people w/ pre-existing cloud skills. It is cheaper than the stop-gap of paying outside consultants till you can bring in the interior staff. Also, these people _know_ your existing systems. They know your business logic, they know what needs to be up and they know what can't break on Thanksgiving weekend. That is valuable knowledge that is a hidden cost in rehiring the work force.
You're going to find new challenges in the new system. If your employees are willing to transition, then transition them. It is your best bet.
If you have a subset of employees that can't/refuse to transition, put them on performance improvement plans. No rumor mill - if you're on a PiP you need to shape up or get out; if you're not on a PiP keep doing well, your job is safe.
Perhaps saying something like, "Are you sure the cost savings are really there, and you won't face unexpected hidden costs in the future? I have experience with X,Y,Z, and we found that while the cloud is optimized for x64 deployment, we had hidden costs A, B, C, crop up after 9 months."
As it is, it's only added noise and made you look bad.
Perhaps using http://www.paulgraham.com/disagree.html and avoid dismissing the content because you don't like the way they said it (DH2, responding to tone).
Those will remain where they are and, eventually be ported or replaced (or kept alive - IBM mainframes have been legacy-friendly for over 50 years now) by newer systems as they are phased out.
The job of the people is to translate the business requirements into stuff actually happening, and that doesn’t go away with cloud. If anything you need to do more of it (someone has to take care of auto-scaling, a thing that wasn’t possible in your legacy DC, for example). The worst thing you can do is sack the people then realize you need consultants at 5x the cost to do anything now...
For example, if the company needs 8 fewer people to install and set up servers, but 10 more people to work on robust automated code deployments to support business growth, and both jobs need Unix skills.
Hence, the setting-up-servers department budget shows cost savings, the overall technology budget grows by 2 employees instead of by 10, and business growth means overall profit increases.
Are some of them close to retirement age?
Those are absolutely the last people on Earth you should ask, because their advice will always be, fire all your own staff and outsource it to us. Or if you're too cowardly, for an extra fee you can TUPE them over and we'll fire them for you.
Kudos for being switched on enough to cut them out of the loop.
Once you let those people to you will be losing their knowledge in this area. Not sure if it matters.
You could retrain them. You could make them work fewer hours without reducing wages if the same surplus value can be produced with fewer labor hours.
But, if, as it seems, your goal is to reduce labor costs, then there is no way to do that without costing wage earners their livelihoods. At that point hand wringing about people losing their jobs is pretty meaningless. They're probably better off leaving and looking at other options than they are staying and having their wages cut, but either way they're getting screwed.
A former Fortune 500 employer of mine went through the outsourcing boomerang. Now I understand that they have big plans to move everything to the cloud within the next two years. Knowing their systems like I do, and knowing that the CEO who is driving this is fairly noobish (note that I said CEO rather than CIO), I don't think this plan is going to work out too well for them. But given that since I left they've had to file bankruptcy, and in doing so involuntarily dropped to about half of what they were before (employees, assets, etc.), I guess this is just par for the course for them now.
As others said, you could retrain them.
Loyal tech workers are invaluable right now. But then again I'm not the one who has to make that choice.
Technology employees should embrace new tools; that's part of the job that ensures compounding effects over time.
I have a feeling that whatever decisions you finally take would take into consideration the wellbeing of your employees.
I would assume that the folks who are doing the legacy stuff have the necessary aptitude to pick up the skills that are needed to automate for the future. Can their existing data center skills be translated into learning about AWS/Azure, VMs, Containerization and the necessary scripting and DSLs around those toolsets?
Obviously there will be reductions in headcount but give them the option to retool for the future. Even if they are not all retained but if you retrain them then they will still have options elsewhere with relevant skillsets.
You are finally running a business after all where revenue and profit are the drivers.
Good luck and I would say that your employees are luckier than those who consider them as mere resources and not people with lives and loved ones to take care of.
I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins. Most of them left to go safe companies where they can stick to their old ways.
Yes, devs are often enthusiastic because they think they can ditch those boring old operations people who always stop them having fun.
Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff...
The truth is the need for ops people doesn't evaporate when you go cloud, be wary of anyone who says it does. This is the same dynamic that played out with MongoDB.
Who can get away with a job that just requires them to open a Java IDE? Those jobs are just as extinct as a pure ops role.
I think it's more subtle than that. (Also, I think the parent was trolling, but you knew that). It's that there's a certain class of programmers (and/or managers with exclusively programming backgrounds) that truly believe all (technology) problems can be solved with software.
If one assumes that belief, then the hostility to someone who believes differently is arguably inevitable.
It also explains why, to some, "Devops" is code for "wannabe dev" or some other coding-first role.
>Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff... >The truth is the need for ops people doesn't evaporate when you go cloud
That explains the dozens of seemingly-identical job postings from tech startups for "Devops" engineers with both experience in the above latest tools and operational experience. Fortunately, they only ask for 1-3 years, which is realistic.
> This is the same dynamic that played out with MongoDB.
Ultimately, though, did it matter? If we sometimes got fail whales instead of 140 characters, we still didn't get flying cars.
Alternatively, for whatever reason (freely flowing VC? low interest rates?), there's too little incentive for low cost or high reliability ops right now.
You have much more knowledge to give those answers than we do. What to tell employees, and who can/not be saved from losing their job depends on those answers.
I will say that open communication, with complete answers of what people should expect, will help the change to happen without losing the trust of those who do get to stick around. You will have some harsh truths to share. If I worked for you, I would want to hear them as soon as there are enough answers to not leave me in limbo on how it affects me, personally. Not sooner, not later.
As you might expect, replacing top notch techies is quite hard. So managing the transition and making sure those that won’t be kept are better off is not only a good heartened approach but important to the success of the project.
Management consultants usually get a bed rep but that’s something they’re pretty good at. If you’re an F50, you have the budget to hire a top one.
Retraining can be done as part of the project, but you need to figure out what the project and sub projects looks like first. Then you an know roughly how you’ll move people around at various phases. It takes time to build new systems and time to train. Line those up and you can take advantage of them.
Be careful with costs. They will blow out if you aren’t mindful.
It’s going to be messy for a LONG time, which causes a huge amount of pain in the org. Be ready for this.
Moving to cloud brings a whole new set of problems you haven’t thought of yet. For example, cleanup becomes more critical (old resources cost money, but can’t be “seen” because they’re not physical and just a hidden line item somewhere).
There’s more. My email is in my profile if you wish to reach out.
The parent's comment use of the word "cycle" implies, by association with phrases like "cycle of violence", that there might be some equality or that both parties may be victims to some degree.
In the face of such assymmetry, such a claim would be extraordinary, requiring a strong argument, more than just an implication.