My wife worked at Travelport in the late 2010s. They ran a lot of Delta’s systems but Delta of course has their own infrastructure. Some of the history of Delta’s teams is intertwined with the companies that were merged together to form Travelport.
From what I could tell from her talking about work it wasn’t particularly far off the state of the art at the time. They had a fancy new AWS hire to help lead them into AWS from mainframes. That didn’t fully work out but they still had a bunch of containerized .Net and Java running.
Mostly this whole back and forth argument sounds like a ball of pain with everyone upset at each other and very little of it mattering unless lawsuits are filed. If Delta is using Windows at IBM it doesn’t make them “ancient.”
Gross. If that's the alternative I would rather stick with mainframes. People forget that it's perfectly fine and even better to just run code directly in many cases.
This is from 2015, and it improved from there (e.g. the daemon-less podman matured, NAT is no longer needed). That said, I'd happily take even the performance impact of VMs and order a few more servers, if it means that I won't have to deal with dotnet/java runtimes being shared by multiple applications.
Yes those langs have large runtimes. But even for compiled native languages, if you have a lot of dependencies you will require containers sooner or later. Go doesn't really have the huge dynamic-linking ecosystem. But, if you do and then project X and project Y diverge on versions... you have a problem.
Also containers can automate deployment easier. Like the line from bare server -> full deployment is much faster if you use containers.
from what ive read microsoft is partially liable for allowing crowdstrike to make updates the way they did, forcing updates and then not detecting and disabling the affected driver. maybe im wrong. seems like theyre finger pointing and that makes me think theyre guilty even more.
Windows did not disable the driver because Crowdstrike marked their driver as critical for boot. Had they not done that, Windows would disable the driver automatically.
Most severs have a management microcontroller in them that is capable of administoring the main system, typically including (but not limited to) full keyboard video and mouse access.
Typically this is exposed through a dedicated ethernet port, allowing administrators to remotely manage their servers.
What's your plan for if an attacker gets access to your IPMI system and uses it to put rootkits on all your machines, including the ones that you image other machines from?
> CrowdStrike followed up with Delta on the offer for onsite support and was told that the onsite resources were not needed
> Even though Microsoft's software had not caused the CrowdStrike incident, Microsoft immediately jumped in and offered to assist Delta at no charge following the July 19 outage," the letter said. "Each day that followed from July 19 through July 23, Microsoft employees repeated their offers to help Delta. Each time, Delta turned down Microsoft's offers to help, even though Microsoft would not have charged Delta for this assistance
Two vendors offer you free on-site help, during the biggest outage in history, and you turn it down?
This isn't the CIA, it's Delta Airlines. Bring those overpriced security monkeys in, make em sign an NDA, and have them sit there and watch you type.
To not do that... even if they were useless... I can't find a justification ....... other than being able to later claim it wasn't Delta's fault it was down so long, and nobody having evidence to the contrary.
"Security" software has always been like that. What's bonkers is the state of regulations and auditing where putting this kind of crap on your systems is somehow seen as increasing your security rather than decreasing it.
The larger the org the slower they are with updates. Some global companies skip one or two versions of Windows because of the time it takes them to upgrade their systems. Microsoft is trying to deflect the blame, but it is their policy of maintaining backward compatibility that bit them. If you tell enterprise they can run their software on the latest version of your OS there is no incentive to upgrade, because they can always do it later. Steve Jobs knew how enterprise clients think from his time at NeXT and killed Apple's enterprise division when he came back. When large customers pushed him to create enterprise sales and support team he told them to go the nearest Apple Store, because he did not want them to limit his ability to change the OS.
> Linux distros make changes and they ate Microsoft’s lunch in the server room.
This statement is a little funny considering the only reason the CrowdStrike situation was such a big deal was because of how many high-profile companies are using Windows Server.
Noone noticed because most Linux servers does not run Crowdstrike Falcon.
If the same number of Linux servers as Windows servers were affected, most of the Internet would have been down.
The price was secondary. The ecosystem for windows was just garbage for web, VM, container, etc stacks. Much like they missed the boat on mobile, they missed the boat on transitioning to cattle from pets.
Case in point, Linux ate Microsoft in the university datacenters where Microsoft was literally giving it away for free.
Your offshoring partners are ignorant. There is no equivalent partner representing open source.
You’re living in a denial zone where you think something is winning because there is no competitor making more money. The reality is people realized they didn’t need to pay at all and moved on with 10-100x the scale outside of the garden of stupidity.
How is that not a case in point? Cost is irrelevant and people chucked that shit to the side.
Also, you’re mixing up the early days of UNIX with Linux, which were essentially decades apart. I watched Linux eat the entire campuses of universities and replace windows in the early 2000s. The UNIX and BSD life support cycle you are talking about is irrelevant shit from computer science departments where windows was never welcome but the scale was constrained to the department.
That was secondary to startups using Linux, because they could not afford Microsoft and because at the time IIS was as tight as a sieve. Demand for people with knowledge of Linux drove Linux adoption at universities.
It was overpriced and didn’t have a lot of software.
That's not how I remember it at all. Apple XServers cost about the same as the competition and the XServer RAID was a bargain compared to similar hardware offerings at the time. And although I never used the software, based on the demos I saw from their sales team, it looked really nice and I never heard anybody complain about it. The aversion wasn't price or software, it was just that people didn't really trust Apple support and service.
My point is that I worked at a couple of different places that where considering XServer RAID setups, and as part of that I read everything I could find, attended live presentations of the hardware and talked to people online who where using XServers. And in that process found no reason to write off the hardware of purely software grounds. If you worked extensively with the software and found it a deal breaker, then I'll defer to your experience, but at the time I didn't run into anybody with anything overly bad to say about the software.
They tried to enter that market a couple of times with MacOS and AIX, but ultimately abandoned it. They are happy making their hardware pluggable into corporate networks. It's your choice, if you want it.
Huh? I don't see how MS has any blame in any of this. Client chooses not to upgrade systems. Client chooses to use crowdstrike as a vendor. Client chooses not to accept free support. At what point was MS supposed to do something here? There are loads of things to complain about or take issue with when it comes to MS, but I don't see how this is one of them.
Microsoft wants you to upgrade and support ends for old version for precisely that reason, to get the customers to pay (like SAP's model). I don't think MS will ever tell a customer go ahead use a unsupported older version, from both a liability perspective and wanting to make the sale perspective. If Delta wanted to cheap out, don't blame the vendor
On the other hand, Microsoft will sell you “extended security updates” for your out of support OS. Third hand information, but I was told the cost was something like $100/seat year one, $200/seat year two, etc.
Actually NeXT flubbed enterprise sales because they didn't ensure that their OS was fully POSIX-compliant. POSIX-compliant OSes didn't need special waivers in order for the Federal govt to buy them; that is why the CIA and NSA were NeXT's main customers, because no one requires spy agencies to buy COTS software.
If Jobs had spent just 2 engineer-years or less of engineering time to fix POSIX compliance, their market would have been many times larger; something HP's workstation division and Sun were very happy about, I am sure!
For NeXT, as Apple, the POSIX/UNIX story has always been the same as on Windows, a means to bring the ecosystem into the platform, not to make software there and deploy it elsewhere.
There is even a surviving video from a NeXT meeting, where Steve Jobs talks about this strategy.
If backward compatibility actually was that great and clients actually believed it, why would they wait around for ages before upgrading? It should be a no-brainer.
Instead, what I've seen in practice, is that people are very much afraid their random crap will break. They absolutely don't believe the backwards compatibility is a thing, at least not without "testing".
The company I work for has a bunch of software that the providers only "support" on versions of windows server that are at least one or two steps behind current. Hell, I've even seen this for things which seemed pretty basic (think random web app) running inside a JVM.
I don't personally use windows all that much, so don't really have first-hand experience, but my impression is that people are very much afraid of upgrades, even when running pretty "basic" things (as in "office software", not things which manipulate low-level, old interfaces or similar)
The article doesn't really explain anything to an extend that would all us to conclude anything regarding the age of those systems.
The comment "because it runs on those providers' systems" is also a bit weird. Do they run on AS/400? Or is it simply because IBM would let Microsoft employees touch any IBM owned and operated infrastructure, probably the later. In my experience dealing getting access to infrastructure run by IBM (formerly SoftLayer), then yes, things might be ancient. We required VM with Windows XP to even run their VPN solution and that barely worked anyway.
"Ancient" would be fine. At this point, we're pretty much back to a 3270 displaying data from a mainframe anyway. Use DOS and an 8514/A graphics card to access the "cloud" and we'll be better, faster and more secure than anything Javascript based.
What is NOT fine is the lack of a disaster recovery plan. Your disaster recovery plan should be able to cold start yourself within 24 to 48 hours. It's clear that Delta simply didn't have anything resembling a disaster recovery plan.
Pure speculation: The part where the Delta team didn't want Microsoft's help feels like it is probably that they knew that their systems were a mess of old, unpatched Windows clients and servers and were embarrassed about it (and that their legal teams probably knew that it was better to keep everyone out so they could sue later on)
63 comments
[ 3.4 ms ] story [ 107 ms ] threadFrom what I could tell from her talking about work it wasn’t particularly far off the state of the art at the time. They had a fancy new AWS hire to help lead them into AWS from mainframes. That didn’t fully work out but they still had a bunch of containerized .Net and Java running.
Mostly this whole back and forth argument sounds like a ball of pain with everyone upset at each other and very little of it mattering unless lawsuits are filed. If Delta is using Windows at IBM it doesn’t make them “ancient.”
Gross. If that's the alternative I would rather stick with mainframes. People forget that it's perfectly fine and even better to just run code directly in many cases.
https://web.archive.org/web/20150218172511/http://domino.res...
This is from 2015, and it improved from there (e.g. the daemon-less podman matured, NAT is no longer needed). That said, I'd happily take even the performance impact of VMs and order a few more servers, if it means that I won't have to deal with dotnet/java runtimes being shared by multiple applications.
Also containers can automate deployment easier. Like the line from bare server -> full deployment is much faster if you use containers.
This is the EU's fault :-P
Most severs have a management microcontroller in them that is capable of administoring the main system, typically including (but not limited to) full keyboard video and mouse access.
Typically this is exposed through a dedicated ethernet port, allowing administrators to remotely manage their servers.
Since IPMI is usually on servers, did you mean Intel vPro / AMD DASH for PC clients?
[1] https://www.heise.de/news/Mobile-Device-Management-Einbreche...
> Even though Microsoft's software had not caused the CrowdStrike incident, Microsoft immediately jumped in and offered to assist Delta at no charge following the July 19 outage," the letter said. "Each day that followed from July 19 through July 23, Microsoft employees repeated their offers to help Delta. Each time, Delta turned down Microsoft's offers to help, even though Microsoft would not have charged Delta for this assistance
Two vendors offer you free on-site help, during the biggest outage in history, and you turn it down?
This isn't the CIA, it's Delta Airlines. Bring those overpriced security monkeys in, make em sign an NDA, and have them sit there and watch you type.
To not do that... even if they were useless... I can't find a justification ....... other than being able to later claim it wasn't Delta's fault it was down so long, and nobody having evidence to the contrary.
You need to trust the guard implicitly.
Linux distros make changes and they ate Microsoft’s lunch in the server room.
This statement is a little funny considering the only reason the CrowdStrike situation was such a big deal was because of how many high-profile companies are using Windows Server.
https://www.theregister.com/2024/07/21/crowdstrike_linux_cra...
The AWS stats are very clear, Microsoft is way behind in the server world.
Which is quite a different point.
Naturally Apple's UNIX flavour came with Apple's tax, and not free beer one, as an additional bump, besides the software policies.
Also note that the chaos on the desktop from was caused by 1% of the Windows desktop market share.
The Year of Desktop Linux with its 2% is nowhere closer to be used in such critical infrastructure.
Case in point, Linux ate Microsoft in the university datacenters where Microsoft was literally giving it away for free.
You’re living in a denial zone where you think something is winning because there is no competitor making more money. The reality is people realized they didn’t need to pay at all and moved on with 10-100x the scale outside of the garden of stupidity.
Also, you’re mixing up the early days of UNIX with Linux, which were essentially decades apart. I watched Linux eat the entire campuses of universities and replace windows in the early 2000s. The UNIX and BSD life support cycle you are talking about is irrelevant shit from computer science departments where windows was never welcome but the scale was constrained to the department.
That's not how I remember it at all. Apple XServers cost about the same as the competition and the XServer RAID was a bargain compared to similar hardware offerings at the time. And although I never used the software, based on the demos I saw from their sales team, it looked really nice and I never heard anybody complain about it. The aversion wasn't price or software, it was just that people didn't really trust Apple support and service.
If Jobs had spent just 2 engineer-years or less of engineering time to fix POSIX compliance, their market would have been many times larger; something HP's workstation division and Sun were very happy about, I am sure!
There is even a surviving video from a NeXT meeting, where Steve Jobs talks about this strategy.
If backward compatibility actually was that great and clients actually believed it, why would they wait around for ages before upgrading? It should be a no-brainer.
Instead, what I've seen in practice, is that people are very much afraid their random crap will break. They absolutely don't believe the backwards compatibility is a thing, at least not without "testing".
The company I work for has a bunch of software that the providers only "support" on versions of windows server that are at least one or two steps behind current. Hell, I've even seen this for things which seemed pretty basic (think random web app) running inside a JVM.
I don't personally use windows all that much, so don't really have first-hand experience, but my impression is that people are very much afraid of upgrades, even when running pretty "basic" things (as in "office software", not things which manipulate low-level, old interfaces or similar)
The comment "because it runs on those providers' systems" is also a bit weird. Do they run on AS/400? Or is it simply because IBM would let Microsoft employees touch any IBM owned and operated infrastructure, probably the later. In my experience dealing getting access to infrastructure run by IBM (formerly SoftLayer), then yes, things might be ancient. We required VM with Windows XP to even run their VPN solution and that barely worked anyway.
What is NOT fine is the lack of a disaster recovery plan. Your disaster recovery plan should be able to cold start yourself within 24 to 48 hours. It's clear that Delta simply didn't have anything resembling a disaster recovery plan.
Also, wouldn't that contractually be IBM's issue to solve?
Another cute quote by M/S. I hope this goes to trial with a Jury. Then the truth will come out.
Crowdfare crashed only Windows Systems, not IBM Mainframes or AS/400 or AIX or Linux or BSD.
The blame is on Crowdfare and Windows only, not IBM systems.