While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs? Not to mention all the other issues with operating your own service like this.
That comparison isn't apples to apples, though. Internal solutions are frequently less complex than cloud systems, and have fewer and more understandable failure modes as a result. I could throw a rock at my last job and hit mail servers that had better uptime than Gmail, to use one prosaic example.
Keep in mind, when comparing it to your experience, is that not all customers are necessarily down when there's a problem.
I'm not dumping on Gmail, running a giant system like that has got to be extremely challenging to get right.
But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware.
> But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware.
This doesn't jive with the tech industry hype train though, hence, it being "myopic" to consider running your own services internally.
Of course it's possible to achieve better uptime yourself. The question is not: is it theoretically possible to be more available than gmail. The question is is it cost-effective to do so, once all the relevant factors are considered.
I have to agree with this. We recently replaced a homegrown webapp with its MS Dyanmics GP equivalent and headaches have increased exponentially. It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls... when one guy churned out the original in a day, and ran for a year-and-a-half with only minor technical (and no major organizational) hiccups.
> It's as if the system was designed to be so complex as to require a team of people to operate, plus a manager to supervise, plus a support person to field phone calls
This is exactly it. Even if not intentionally designed, it ends up that way. As andrewstuart2 mentioned above, "[customers] also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer". This is great I guess, but if your home-grown solution doesn't use, say, Oracle, to start out with, you're going to eliminate a heck of a lot of necessary engineering expertise when things go wrong. Break up systems even more and suddenly just one generalist can do it all on the side...
A classic build vs. buy situation where the purchase seems like the solution and in reality, it is a purchase plus development plus support. In many cases the internally designed, function specific, application is better because of its constrained functionality. Many companies, mine included, have made the mistake of purchasing a silver bullet rather than forging one with our own time, talent, and treasure.
For many companies, especially smaller ones, the cost of setting up and managing infrastructure that can support high uptimes just isn't manageable. It's not just the hardware, it's also the price of hiring or contracting people with the expertise to do it.
Case in point: It's informative that you refer to "mail servers". The plural makes it clear that the company in question isn't in the same league, size-wise, as the target market for most SaaS vendors.
Depends. If you're hoping for high uptimes but aren't forced to guarantee them, you can get awfully far with a good Lenovo box running RAID1, Qmail and judicious application of uninterruptible power supplies.
If you're just after high uptimes for your email server, and your email server doesn't need to be anything more than an email server, and you already happen to have a need for a Unix server and a qualified Unix admin, then sure.
Typical companies, though, are mostly Windows shops and really are looking to get something more along the lines of an integrated communication/collaboration/scheduling tool like Microsoft Exchange or GMail.
I've also seen internal systems become far more complicated because there was no check on what a project owner could request and cloud systems drive simplicity because e.g. you couldn't try to force Amazon to implement some weird custom behaviour one of your legacy app development teams thought they needed.
The general trend I follow is that you're good at things you do regularly. That means that internal systems tend to be much worse at recovering from failure (i.e. the time to recover can be days or even weeks) unless they are careful to design with redundancy in mind and practice it regularly. That's a time-honored tradition in large IT shops but it's also one which often requires constant effort to justify the "wasteful" expense even in places which should know better.
Ya that's a fair point. I still think that in general you're likely to have worse uptime, or at best just comparable uptime to large cloud services in most cases.
Cloud services just have to be equal to internally run ones in this regard, since they save so much in other areas. So I guess people have to decide: is 20 hours of downtime every few years worth running your own services internally and paying people full time to manage them?
Agreed. The following statement the author makes is too generalized in my opinion:
> What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer.
This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products.
In this case, the customers of #NA14 did experience downtime. But they also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer, as well as having likely already mitigated instances of this problem before. These prior mitigations, of course, are met with much less fanfare than an outage. It's hard to appreciate things you don't experience.
> > What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer.
> This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products.
Does all the "intellectual capital" (read: aws/etc runs a highly complex system made up of various parts that sometimes all just shit themselves when a single low level thing hiccups) make it somehow not someone else's computer?
The point of "it's just someone else's computer" isn't that it's just a computer like you or I have, it's that it's not some magical, mythical beast.
The greatest achievement of aws/et-al is convincing developers they will need "web scale" infrastructure at the drop of a hat.
What I'm saying is that you can't justifiably say that it's "just someone else's computer" (emphasis mine). Just someone else's computer would be ssh root access to a physical machine in some data center. No tenancy separation, no dynamic allocation, no exposure of failure tolerances (e.g. AWS Availability Zones), etc.
To illustrate this point succinctly, there are probably thousands of "somebody else's computers" sitting in a dell warehouse somewhere that are no good to me. The cloud is only useful if there are additional services to help me consume that computer.
I'm not saying it doesn't run on somebody else's computer. I could, though, since private clouds are in fact "clouds" that are not someone else's computer.
Again, what I'm saying is that it is not a sufficiently specific generalization to accurately describe "cloud." The definition of "cloud" is fundamentally more complex than "just somebody else's computer."
> While events like this are not great, isn't it a little myopic to think the solution is to run it yourself? 20 hours of downtime sucks to be sure, but is your own uptime rate really going to be better than theirs?
It sounds very much like you're assuming the answer is no.
In my entire 10+ year career I've never had a service I was working on have a 20 hour outage.
neither did I. but on average, I'm sceptical if my track record is better than theirs, especially on the more complex installations.
I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.
I think his point is that when comparing running SalesForce on the cloud for every users SalesForce has, vs running SalesForce locally for your users, there will a lot less challenge to keep it running locally than on the cloud.
The point isn't to serve the volume that SalesForce has, the point is to serve yourself.
If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else.
the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
> I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.
So what?
If your business is built around a system and that system goes down, would you rather that system serve billions of customers and go down for 20 hours, or serve only you and go down for 20 minutes?
Your homegrown service is unlikely to have a 20 hour outage, by being much simpler. Whereas if you use any cloud solution at all, you become a part of a huge and complex system that is likely serving orders of magnitude more requests than your service. And that can go down by reaching situations that would never be seen in your service.
The flip side is, your service is probably peanuts to them. Should you grow using your own tech, you'll hit the same pain points that they have, in their very beginning, probably even during the MVP. They have already taken care of that so you won't have to.
> In my entire 10+ year career I've never had a service I was working on have a 20 hour outage.
Through luck or careful planning and good resources?
Once you have enough hardware to handle load, it all really comes down to risk mitigation.
Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line.
Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?
I won't deny the role of luck, but I find that the more careful I am to keep my systems redundant, flexible, and rapidly deployable, the luckier I am.
I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of:
1. Calling the hosting provider and ascertaining the problem.
2. Pulling the latest code onto the fallback server (from a different hosting provider).
3. Restoring the database from the backup server into a newly-created database server.
4. Changing the configs.
2 and 4 were able to happen while 3 was running.
As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.
I was thinking the same thing. Just like the article said, the cloud is someone else's computer. So why do you think your computer woudln't have these issues?
Throughtout my career the interesting thing about the level of scrutiny the "cloud" receives is that people rarely ever applied that scrutiny to themselves.
Is it secure? Hm..I'm sure Salesforce does more work around security than the vast majority of their customers would even be capable on their own.
Similarly, these types of catastrophic events will happen and I am just as confident in saying that Salesforce was able to diagnose and fix this problem better than the vast majority of their customers would have been able to do on their own systems. I've seen JIRA on prem go down several times.
I've worked building enterprise applications my entire career. Unless you have an extremely lean and engineering oriented organization, it's almost always better to buy than to build in house. There are exceptions, but for the most part this holds.
Having said all that, every single place that I've worked at has had it's own data lake/warehouse and having your own backup of your data is critical.
> I'm sure Salesforce does more work around security than the vast majority of their customers would even be capable on their own
This is a fallacy that people like to believe because they are not doing the work themselves. The fact is SalesForce hires the same type of developer/admin that any company would hire and that person is trying to do their best like everyone else. Because it's behind a curtain people feel its more secure, when in reality it's probably the same level of competence/incompetence that anyone can do hosted on-prem.
I understand what you're saying, but most companies (not all) don't have dedicated staff to this sort of thing.
They usually take other precautions like VPNs, etc.
I'm not saying it's perfect, but for MOST people, this is a better alternative.
Does that make sense? If companies made a concerted effort to hire for this and take it seriously, it makes sense. But large enterprise customers with the resources to do this aren't the majority of saleforce users.
But giving this service is the main product of salesforce, therefore intrinsic quality measures (like security) are more important to their management (for their well understood need for customer trust), than if you were developing an in-house product.
In the latter case intrinsic quality of a biproduct of your operation would not matter to your management, because they primarily think in order to give the best quality to the customers of the company's product, and the intrinsic quality of that product matters to them (eg. to use BPA free materials for the rubber ducks they manufacture). This is why they will feel spending on security of the in-house development unjustified, until a major security breach happens.
> therefore intrinsic quality measures (like security) are more important to their management
Right, I agree, but just because it is important doesn't mean they are good at it. And there is the fact that there is now likely 1 or 2 people attending to 1000 customers, than 1 or 2 people in house tending to 1 customer.
Except that security engineer : customer ratio doesn't make sense as a metric. Those 1000 customers all need the same thing - for their data in the cloud to be secure. They don't have differing security requirements that pull the engineers apart in different directions.
So it really is a comparison of 1-2 security engineers vs 100-200 security engineers.
This is completely not true, many customers often need specific custom tailored security for business specific needs (This is something most startup SaaS developers dont realize or care about because they are usually just starting development and care about growth and base product). This is where the problem lies, you paint with a broad brush and get the 80%, the other 20% is a nightmare usually. Most the time the 80% has holes in it too, like we saw here with SalesForce.
Whenever Im architecting a system, I always tell the execs you can either pay for it now (going on-prem) or pay for it later (with Saas) because at the end of 5 years you always end up paying just about the same, its just a matter of where. With payroll, downtime, customer losses, hardware, DR, wasted time with project managers from your SaaS providers.
Id always rather have control and know my system than depend on someone else's hire who behind a curtain of three, four or five nines of marketing jargon. It's always a pay me now or pay me later situation.
That doesn't entirely work. Most companies that use Salesforce or other cloud services (thus would be the ones doing on-prem instead of SaaS) aren't hiring huge security teams geared around cloud security. Even if they are hiring security teams, they're both smaller and probably focused around internal security issues (enterprise security teams).
And I also don't accept the premise that it's the same type of engineers that the companies would hire. The security teams at cloud companies are made up of very specialized engineers that are vastly different from the average dev you'd hire to run your internal on-prem servers.
Security is one of the things that sets SaaS (by the big players) apart. Running something on-prem will not have nearly the same security resources looking at it.
Hiring technical people is one of the hardest problems in technology and you're just going to blithely assume that a software company will do no better at it than a non-technical business with an IT department? Even setting aside the difficulties of screening and interviewing, more top-tier technical workers will prefer to work for a software or SaaS company in SF that pampers its employees than for a paper company in Duluth, Michigan.
S3 guarantees three nines of uptime. Thus, you should be prepared for 8+ hours of downtime at any moment.
If Salesforce gives you 99.8% vs. Amazon's 99.9% and this shocks you into writing a reactive web post, it's a good time to review some basic systems theory.
No surprise an article like this comes from an IT vendor that sells services to people that still own their own hardware.
Which brings up the real question. Can you afford to not run on someone else's infrastructure? Yes, Salesforce had an issue. How much would you need to spend on your own company's infrastructure to do better? Is it worth it for your business?
You need to own the gear, hire the internal people, pay for a host of enterprise software that is probably more expensive than services, etc.
> Can you afford to not run on someone else's infrastructure?
Most businesses do, so I would say "Yes".
We'll go through this whole decade long period of how wonderful the "cloud" is, and then we can go back through the next phase where everyone is getting paid to move services back locally because of how "easy it is to manage with Docker/Kubernetes/$flavor_of_the_week_cloud_orchestration".
I think the economies of scale will eventually draw most to using hosted services, even if it's as simple as better bottom line for the IT supply chain.
It's a little like managing your own email or website server on site - the overhead ends up being more resource consuming than farming it out, and we haven't really seen people switching back to their own servers.
"This was where the real problems began – the file discrepancies responsible for the database failure had been replicated back to the Washington data center while at the same time, the backup in Chicago had not completed in time and, as such, could not be used as a restore point either."
Ah, another case of "To make error is human. To propagate error to all server in automatic way is #devops."
Id rather think its a failure in planning.From AWSs playbook the conventional wisdom is that things fail all the time and you have to plan for these eventualities.See the Netflix models and Googles multihoming approach to spreading the service offering.Its always going to be the case that a cloud provider can fall over however if you are simultaneously running on different providers , all you have to worry about is the speed of rerouting traffic to your still operational site.Short of the entire internet keeling over from a backhoe incident you should be sitting pretty.
A single circuit breaker shouldn't cause an outage if your servers are powered using dual power supplies from separate circuits or you are load-balancing across servers powered by separate circuits. This appears like mere incompetence.
I know this is not a productive comment but I hate editorial or commentary from companies that chase crisis for more business. I used to work in hosting and these things happened and then all our competitors would jump in on the bashwagon... it got to a point where everyone was chasing each others tails and not productive.
Plus I don't care who you are or how big you are. If something goes wrong there's a possibility your app will go down. H.A. notwithstanding is a best practice but some of the most robust systems in the world still blow up. They were invented by people ergo they are inherently flawed. That's the beautiful thing about them.
How many smaller businesses that use salesforce would have lost all their data with hardware failures over the years? Most smaller businesses have absolutely no backups. Remember HN is the exception as were a technical crowd ;)
Zerto makes DR solutions for VMWare based private clouds. IMHO - the title is sensationalist and probably an attempt to drive interest in private clouds over public clouds.
That being said the firmware bug could have happened in a private cloud as well - impacts would be localized to a company though.
Either way - public clouds are here to stay unless someone needs extreme scale in which case they'll use a private cloud (e.g. Dropbox moving out of AWS). But then they most likely may not use VMWare for their virtualization solution.
The main issue is with how Salesforce is running its multi-tenanted cloud infrastructure.
Modern techniques and solutions already exist to minimize these kinds of events from happening using a multi-cloud approach, containers, geo failover, etc.
So while this sucks for Salesforce, it doesn't mean every SaaS platform or cloud infrastructure provider suffers from the same legacy issues as Salesforce.
If anything the author should be arguing the flaws of multi-tenancy, not that SaaS is dead because of a power outage of a single SaaS provider data center.
68 comments
[ 3.0 ms ] story [ 145 ms ] threadhttp://www.pcworld.com/article/160153/google.html
http://news.softpedia.com/news/Longest-Gmail-Meltdown-in-His...
http://arstechnica.com/information-technology/2012/12/why-gm...
http://www.computerworld.com/article/2486914/web-apps/update...
Keep in mind, when comparing it to your experience, is that not all customers are necessarily down when there's a problem.
I'm not dumping on Gmail, running a giant system like that has got to be extremely challenging to get right.
But let's not wear rose-tinted glasses about the cloud. It can be great, but for some applications you can get better reliability by competent people deploying reliable software on reliable hardware.
This doesn't jive with the tech industry hype train though, hence, it being "myopic" to consider running your own services internally.
Is there a list of outages on other 'major' services of this magnitude?
http://m.huffpost.com/us/news/gmail-down
This is exactly it. Even if not intentionally designed, it ends up that way. As andrewstuart2 mentioned above, "[customers] also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer". This is great I guess, but if your home-grown solution doesn't use, say, Oracle, to start out with, you're going to eliminate a heck of a lot of necessary engineering expertise when things go wrong. Break up systems even more and suddenly just one generalist can do it all on the side...
Case in point: It's informative that you refer to "mail servers". The plural makes it clear that the company in question isn't in the same league, size-wise, as the target market for most SaaS vendors.
Typical companies, though, are mostly Windows shops and really are looking to get something more along the lines of an integrated communication/collaboration/scheduling tool like Microsoft Exchange or GMail.
The general trend I follow is that you're good at things you do regularly. That means that internal systems tend to be much worse at recovering from failure (i.e. the time to recover can be days or even weeks) unless they are careful to design with redundancy in mind and practice it regularly. That's a time-honored tradition in large IT shops but it's also one which often requires constant effort to justify the "wasteful" expense even in places which should know better.
Cloud services just have to be equal to internally run ones in this regard, since they save so much in other areas. So I guess people have to decide: is 20 hours of downtime every few years worth running your own services internally and paying people full time to manage them?
> What we seem to forget all too easily is that the cloud is, in its simplest form, just someone else’s computer.
This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products.
In this case, the customers of #NA14 did experience downtime. But they also got the full expertise of the salesforce engineering organization (which clearly is far from incompetent) being brought to bear on the problem, at no extra charge to them as a customer, as well as having likely already mitigated instances of this problem before. These prior mitigations, of course, are met with much less fanfare than an outage. It's hard to appreciate things you don't experience.
> This isn't even actually true for Infrastructure as a Sevice (IaaS), in which case there's significant intellectual capital that's been invested in the automated provisioning of resources for customers. This is much more true with SaaS products.
Does all the "intellectual capital" (read: aws/etc runs a highly complex system made up of various parts that sometimes all just shit themselves when a single low level thing hiccups) make it somehow not someone else's computer?
The point of "it's just someone else's computer" isn't that it's just a computer like you or I have, it's that it's not some magical, mythical beast.
The greatest achievement of aws/et-al is convincing developers they will need "web scale" infrastructure at the drop of a hat.
To illustrate this point succinctly, there are probably thousands of "somebody else's computers" sitting in a dell warehouse somewhere that are no good to me. The cloud is only useful if there are additional services to help me consume that computer.
I'm not saying it doesn't run on somebody else's computer. I could, though, since private clouds are in fact "clouds" that are not someone else's computer.
Again, what I'm saying is that it is not a sufficiently specific generalization to accurately describe "cloud." The definition of "cloud" is fundamentally more complex than "just somebody else's computer."
It sounds very much like you're assuming the answer is no.
In my entire 10+ year career I've never had a service I was working on have a 20 hour outage.
So, probably, the answer is yes.
I've seen cases where a single downed router chopped of a whole building of business people who - since everything was on network drives - couldn't do anything better than play solitaire a whole business day long.
How arrogant. I seriously doubt the services you've managed have even served a fraction of the data and customers SalesForce has.
If you have experience enough to prevent downtime or to handle it when it happens, then perhaps that is better than risking that to someone else.
the 20 hours of downtime was probably BECAUSE of the amount of data/customers to handle. whereas if there is 1 customer, recovering isn't nearly as difficult or time consuming.
So what?
If your business is built around a system and that system goes down, would you rather that system serve billions of customers and go down for 20 hours, or serve only you and go down for 20 minutes?
Your homegrown service is unlikely to have a 20 hour outage, by being much simpler. Whereas if you use any cloud solution at all, you become a part of a huge and complex system that is likely serving orders of magnitude more requests than your service. And that can go down by reaching situations that would never be seen in your service.
The flip side is, your service is probably peanuts to them. Should you grow using your own tech, you'll hit the same pain points that they have, in their very beginning, probably even during the MVP. They have already taken care of that so you won't have to.
Through luck or careful planning and good resources?
Once you have enough hardware to handle load, it all really comes down to risk mitigation.
Assuming it handles load, your service could probably run on a single server, with a single drive, on a DSL connection for years without any issue. I would say if that happens, it has less to do with your skill as an operator and more to do with the luck of not having a power outage, drive failure or backhoe operator dig up a line.
Can your service withstand your office/datacenter burning down? Or your datacenter being shutdown with all assets seized by the SEC (true story)? Or a massive earthquake or flood or hurricane that disrupts the entire city?
I did have an issue once where a very large power outage that took out a data center. Total downtime was about 50 minutes, which consisted of:
1. Calling the hosting provider and ascertaining the problem.
2. Pulling the latest code onto the fallback server (from a different hosting provider).
3. Restoring the database from the backup server into a newly-created database server.
4. Changing the configs.
2 and 4 were able to happen while 3 was running.
As an aside, this is largely why specialized cloud services like AWS or AppEngine that promote vendor lock-in seem crazy to me. AWS can and does go down, and if your infrastructure is built around their tooling, you can't just provision a new box on a new provider.
Is it secure? Hm..I'm sure Salesforce does more work around security than the vast majority of their customers would even be capable on their own.
Similarly, these types of catastrophic events will happen and I am just as confident in saying that Salesforce was able to diagnose and fix this problem better than the vast majority of their customers would have been able to do on their own systems. I've seen JIRA on prem go down several times.
I've worked building enterprise applications my entire career. Unless you have an extremely lean and engineering oriented organization, it's almost always better to buy than to build in house. There are exceptions, but for the most part this holds.
Having said all that, every single place that I've worked at has had it's own data lake/warehouse and having your own backup of your data is critical.
This is a fallacy that people like to believe because they are not doing the work themselves. The fact is SalesForce hires the same type of developer/admin that any company would hire and that person is trying to do their best like everyone else. Because it's behind a curtain people feel its more secure, when in reality it's probably the same level of competence/incompetence that anyone can do hosted on-prem.
They usually take other precautions like VPNs, etc.
I'm not saying it's perfect, but for MOST people, this is a better alternative.
Does that make sense? If companies made a concerted effort to hire for this and take it seriously, it makes sense. But large enterprise customers with the resources to do this aren't the majority of saleforce users.
In the latter case intrinsic quality of a biproduct of your operation would not matter to your management, because they primarily think in order to give the best quality to the customers of the company's product, and the intrinsic quality of that product matters to them (eg. to use BPA free materials for the rubber ducks they manufacture). This is why they will feel spending on security of the in-house development unjustified, until a major security breach happens.
Right, I agree, but just because it is important doesn't mean they are good at it. And there is the fact that there is now likely 1 or 2 people attending to 1000 customers, than 1 or 2 people in house tending to 1 customer.
So it really is a comparison of 1-2 security engineers vs 100-200 security engineers.
This is completely not true, many customers often need specific custom tailored security for business specific needs (This is something most startup SaaS developers dont realize or care about because they are usually just starting development and care about growth and base product). This is where the problem lies, you paint with a broad brush and get the 80%, the other 20% is a nightmare usually. Most the time the 80% has holes in it too, like we saw here with SalesForce.
Whenever Im architecting a system, I always tell the execs you can either pay for it now (going on-prem) or pay for it later (with Saas) because at the end of 5 years you always end up paying just about the same, its just a matter of where. With payroll, downtime, customer losses, hardware, DR, wasted time with project managers from your SaaS providers.
Id always rather have control and know my system than depend on someone else's hire who behind a curtain of three, four or five nines of marketing jargon. It's always a pay me now or pay me later situation.
And I also don't accept the premise that it's the same type of engineers that the companies would hire. The security teams at cloud companies are made up of very specialized engineers that are vastly different from the average dev you'd hire to run your internal on-prem servers.
Security is one of the things that sets SaaS (by the big players) apart. Running something on-prem will not have nearly the same security resources looking at it.
The article's contents is still worth reading though.
Yes.
If Salesforce gives you 99.8% vs. Amazon's 99.9% and this shocks you into writing a reactive web post, it's a good time to review some basic systems theory.
https://en.wikipedia.org/wiki/High_availability
It's 99.9% per month, so that's ~40 minutes of downtime not 8+ hours.
Which brings up the real question. Can you afford to not run on someone else's infrastructure? Yes, Salesforce had an issue. How much would you need to spend on your own company's infrastructure to do better? Is it worth it for your business?
You need to own the gear, hire the internal people, pay for a host of enterprise software that is probably more expensive than services, etc.
Most businesses do, so I would say "Yes".
We'll go through this whole decade long period of how wonderful the "cloud" is, and then we can go back through the next phase where everyone is getting paid to move services back locally because of how "easy it is to manage with Docker/Kubernetes/$flavor_of_the_week_cloud_orchestration".
I'm old. I've seen this before.
I think the economies of scale will eventually draw most to using hosted services, even if it's as simple as better bottom line for the IT supply chain.
It's a little like managing your own email or website server on site - the overhead ends up being more resource consuming than farming it out, and we haven't really seen people switching back to their own servers.
Mainframes->commodity off the shelf x86->"cloud" (virtualized) computing.
Super Computer -> Personal Computer -> Virtual Computer -> ???
Ah, another case of "To make error is human. To propagate error to all server in automatic way is #devops."
Zerto, you have great technology, but chasing ambulances is a poor marketing strategy.
Plus I don't care who you are or how big you are. If something goes wrong there's a possibility your app will go down. H.A. notwithstanding is a best practice but some of the most robust systems in the world still blow up. They were invented by people ergo they are inherently flawed. That's the beautiful thing about them.
That being said the firmware bug could have happened in a private cloud as well - impacts would be localized to a company though.
Either way - public clouds are here to stay unless someone needs extreme scale in which case they'll use a private cloud (e.g. Dropbox moving out of AWS). But then they most likely may not use VMWare for their virtualization solution.
Modern techniques and solutions already exist to minimize these kinds of events from happening using a multi-cloud approach, containers, geo failover, etc.
So while this sucks for Salesforce, it doesn't mean every SaaS platform or cloud infrastructure provider suffers from the same legacy issues as Salesforce.
If anything the author should be arguing the flaws of multi-tenancy, not that SaaS is dead because of a power outage of a single SaaS provider data center.