32 comments

[ 4.2 ms ] story [ 88.5 ms ] thread
This all being in one location with little redundancy ... somehow not surprised, but still, really?
> [IBM]

Anything the customer doesn't think to ask for, the customer doesn't get.

> A network card used in the carrier's backup systems was not working

It should have been redundant, the problem is that the redundancy didn't work and I guess nobody noticed until it was actually needed.

It's the homeopathic backup problem — they make sure to have a backup because they know that's best practice, but they don't have any checks in place to make sure the backup works.
To be fair to the IT people, the failover system couldn't have been any more redundant.
Details are sketchy at this moment, but Aer Lingus runs lots of AS400 and Mainframe, MVS/ESA, IMS, DB2 and all that stuff. From the linked description on one of the articles

"...construction work damaged a fiber optic cable, wiping out the carrier’s check-in and boarding systems. Issues compounded after it was found that a network card used in the airline’s backup systems was not functioning, leaving Aer Lingus without passenger information for up to ten hours..."

So this looks like they had a backup location, but the new Kyndryl ( IBM ) team had not tested failover in a while... They are probably just trying to recover standard contract fees from Kyndryl specified in their SLA's

Edit: They sent this message at the time to their customers

https://img2.thejournal.ie/inline/5868785/original/?width=63...

I was on a 4 hour delayed transatlantic flight recently - which meant EU legislated delay compensation. It worked out as €600 per person - more than we paid for the return trip. In a 200 seat aircraft that's something like €120,000 - multiply that by 10 hours of delayed aircraft departures and the cost to the airline is huge! I can see why they'd want to pass that on to someone if they could.
Wait what ? I was delayed by 24 hours and missed out on connecting flights because of an Aer Lingus server infrastructure issue which caused them to cancel all flights for that day.

They rebooked my flights but I was paid 100 EUR, which was barely enough to cover the cost of extra taxis, meals and stay for that night.

I'm not an EU citizen so I'm not sure how that affected my compensation.

The compensation applies to all flights to and/or from the EU, regardless of citizenship. You should look into filing a claim, I've heard it's not very time-intensive. There's even companies that take care of filing for you for a fee. The fee is taken out of the claim compensation, no upfront costs!
They try and fob you off with a little compensation (ie. €100) in the hope that that makes you not claim what the law says you're entitled to.
I would love to see how this plays out, considering it was the fault of another 3rd party for the damage of the cable, or if they had drass
If Kyndryl breached their contract with Aer Lingus, then they're liable.

If Kyndryl wants to sue a third party to see if they can recoup some of that cost, that's their business.

What was the other third party? I assume the argument will be the airline paid for redundant data centers and IBM didn’t live up to that portion of the contract.
I haven't been able to find the name of who exactly, most articles only mention "unrelated construction work", which is why I made the previous comment, but as the other commenter answered, i guess that would be a separate lawsuit.
Depending on the size of the construction company, any claim made against them will just result in them shutting down and going out of business if their insurance won’t cover it. This is a pretty common occurrence in construction.
This looks like a great opportunity to learn a lesson and hire competent people to run their infrastructure, along with the associated cost of being redundant and highly available.
But that's a "cost center" according to what the execs learned in business school.
'They' = Kyndryl, Aer Lingus or both?
This might sound weird - but does anyone have a story where "just trust IBM for something important" actually ended well? (Maybe limit it to the past 10 or 15 years.)
Looks like someone might get fired for choosing IBM.

Oh how times have changed

They didn't choose AWS.

Times haven't really changed, the perceived "safe" options have changed.

No IT system is 100% reliable.

They should have practiced what to do incase of a 100% IT outage.

And I think for an airline that would involve just letting anyone into the plane who claims to have a ticket.

If anyone wants to buy a ticket, you take their name and address and bill them later, then hand write a ticket.

Sure, you might get fined for not meeting some laws about id verification, or a bit of revenue loss from people who find out and travel on fake tickets, but overall you're going to make more profit and have more happy customers.

The security checkpoints are staffed by government employees. They aren't going to let passengers through without valid tickets. And what about checked bags?
I believe the only reason for the ticket check at security checkpoints is to prevent non-flying family members from attempting to go to the gate (clogging up the security checkpoint unnecessarily). There are websites which let you generate a 'boarding pass barcode' which will let you through security.

Checked bags can just have a hand written label if needed. Or you have a bunch of machine-readable labels pre-printed in preparation. Or you do what smaller airports do outside the western-sphere, and tell people that they need to take their checked bags with them to the gate and leave them on the tarmac.

None of that is even remotely feasible or legal at commercial airports in developed countries. Ask anyone who works in airport management, or for the TSA. There is zero chance that they would agree to compromise their security control and baggage tracking systems just because one airline happens to be having temporary operational problems. Not going to happen.
Is this speculation? TSA would still do the normal security theater.
The "security" ticket + ID check is a free service to airlines to help airlines stop frequent flier mileage fraud.

In USA I flew a couple times in past decade after I had misplaced my ID. A few TSA questions and then on my way. The person in TSA above the TSA drone checking IDs has considerable authority. Be nice and they let you through quickly.

Aer Dingus more like it...

good name for a dildo app: AirDingus

I think the most interesting part of this article is that Ryanair was able to immediately offer affected fliers $100 tickets to their destination. Being a primarily US flier, I am amazed that their systems and flight vacancies allowed them to react so quickly to the opportunity.
I don't understand the article:

Posted October 07, 2022

Subtitle:

  An IT outage hit the airline's check-in and boarding systems \*last month\*
First sentence:

  Aer Lingus is seeking compensation after an outage to its IT systems led the airline to cancel more than 60 flights \*last night\*.
Which is it? Are there two separate incidents?

Then later:

  The flight cancellations occurred on September 10
One outage: Sep 10

> Aer Lingus is seeking compensation after an outage to its IT systems led the airline to cancel more than 60 flights *last night*.

Data Center News reporters get tired too you know, it's a lonesome job.. :-)