Ironically I tried to close my TSB account on Saturday morning because I'd just switched to Monzo Bank and was told they couldn't close my account due to the issues they were having.
I tried Monzo and they're awesome, but ended up ultimately moving to Starling. Reason being they have a full FSCS license, so your balance is guaranteed by the government up to around £85k.
Apart from that they're both much the same, and both leagues better than a normal bank. It's amazing buying stuff or getting cash out and the notification comes through on the phone before the ATM tells you the cash is coming!
Another good thing both have - you can switch off cards instantly. I thought i'd lost my Starling card when out last week, instead of cancelling it I just temporarily halted it until it was found.
What I do love about Monzo is their approach to development - they blog everything, even the outages. Good to see the process involved in making a new bank from scratch. I'm keeping the account as a scratch account but have already moved the direct debits from my old Halifax account to Starling.
Also using it full-time, moved over my direct debits and it's been great. Ironically we went with TSB over a challenger business bank thinking they might be more reliable... Will have to find a branch this afternoon so we can pay our team as online is down
Monzo!! I was wondering when they will become a full bank. I remember they were offering prepaid cards. I am shocked that in the UK you cannot get an SMS alert when making a purchase or cannot see "fast/live" when you make an expense.
I am most definitely joining them (but keeping my "main" bank)
As for the TSB, it would be interesting to see what went wrong and how they fixed it.
> I am most definitely joining them (but keeping my "main" bank)
This seems to be a very common pattern, and I wonder if they are having issues because of it. I was a Monzo early adopter and even I haven't moved over entirely.
I think a lot of users _liked_ the prepaid cards, to help them budget spending money, etc. I suspect that a lot of people haven't upgraded to a full bank account with them as a result.
Basically you can put money in your account into separately named isolated areas in your account that can't be spent using your card normally. In layman's terms as many saving accounts as you want with more flexibility.
You can still kind of use it as a prepaid card by just topping it up from your bank when you want. It's just better as you can now use Android Pay etc. and have proper banking functions like having an actual sort number/account number, direct debits, standing orders etc.
Why didn't they roll back the update as soon as that was discovered? Would've caused another downtime of a few hours but better than the current chaos.
I have no inside information for TSB, but I have worked on other large migrations before at $DAYJOB.
Based on the original planned 50-hour outage, I suspect that TSB chose a "big bang" migration: all-or-nothing, no rollback possible. These are technically easier (so cheaper) to develop, but far far riskier than taking a phased approach.
I have successfully argued against such approaches in the past, due to the high risk of catastrophic failure if it goes wrong (i.e., exactly what has happened to TSB).
I regard TSB's failure as an in-the-making textbook example of "how not to do it".
Often, these kinds of migrations are the only option due to previous cost cutting/money saving decisions - likely due to pressure from the business and nothing to do with any technical reason.
I remember one migration which involved lots of internal services which were all tightly coupled, meaning updates across the whole backend, not just the part that needed it and no way to do a phased rollout. It was made worse by the fact that the update was to the platform and that depended on a DB upgrade, but that upgrade was incompatible with the old version which meant everything had to be done at once.
They had no disaster recovery plan. They got it done, but I'm sure a lot of people involved went grey early thanks to that nightmare.
Of course the business had no idea of the utter mess that caused all of this and continued to make harmful decisions in the name of saving money and further underfunding the IT department.
Even with tightly coupled systems, ultimately it's all bits on the wire. You can always develop shims or intermediaries to perform a phased migration with rollback options. Of course that takes time and costs money, which is why some people take the damn the torpedoes approach.
The problem with core banking systems is that they're nearly important to test. There's no distributed system, you always need one single source of truth for the whole bank (the ledger). Moving part of that is only possible if you keep shadow accounts in the old system. That's also why live migrations aren't possible and need at least one night downtime.
Can't you set up fake accounts with fake data? Random numbers? I do that all the time is a different context. I have a bunch of scripts I run that populate a database with nonsense. So not shadow accounts, but fake accounts.
This is exactly what I was asking my bank, TD Bank, after their recent upgrade was also a disaster. It looked like a fairly straighforward error from the client side- the login script was looking in the wrong place for my email and couldn't find it (they had my email, because they were still emailing me).
Tech support switched me over to a part of the site that had not been updated, and that worked for a while. Then it didn't. Back to the bank I go, with a new customer service representative, who really didn't grok the issues (new to the company?), and who became quite snippy.
Not so much an upgrade, as moving away from Lloyds system (they were previously merged with Lloyds as Lloyds TSB) and jumping onto a completely new system built for them.
I logged in to my TSB account yesterday and found angular placeholders and bits of un-rendered html all over the place.
The previous version was all server side rendered and worked perfectly.
The app-before-this-one (before the server went down) was a reasonably competent bit of kit, basically a lift and shift of Lloyds, which when retired late last year became something with flashy animation on everything (animated log in place holder, 2FA selectors floating in from off screen - WHY!) but less functionality.
The teams that have participated in migration have proven to have the skills and knowledge required, which they have significantly developed and honed during the course of this project. Banco Sabadell’s proposal to be an agent of change has become evident, with an advanced technological proposition, not only in the domestic market but also on an international scale. With this project Banco Sabadell has created a competitive and value-added platform for the TSB franchise.
This appears to be an IT failure comparable to the 2012 and 2015 IT failures at the Royal Bank of Scotland. These involved multi-day / multi-week full/partial outages, and multi-million pound fines.
Background: TSB is a UK bank. They had a planned 50-hour maintenance window over the weekend (Fri 16:00 - Sun 18:00), to move away from Lloyds's systems (following the "un-merger" in 2013-2014). This went wrong, in what is likely to become a textbook example of an IT failure.
From a letter from Nicky Morgan, chair of the UK Parliament's Treasury committee, to TSB CEO Paul Pester:
> The reports of unauthorised transactions, access to other customers’ accounts, and failures of in-branch services have all the hallmarks of an IT meltdown. This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption.
The IT systems of these companies are held together loosely with gaffa tape. This kind of thing could happen at any time. This is what happens when you deliver your project due to pressure from non technical people even though you know it's not properly finished. Why do people not listen to the professionals when it comes to IT?
In my current job I have repeatedly told non technical superiors that reverse engineering an outdated API is not a proper solution and could fall apart at any time. They do not care. It doesn't even register. Do civil engineers have this problem if they realise there might be a structural weakness?
As far as I know civil engineers can go to prison if the construction they took part in collapses - no such provision is afforded to software engineers.
In the US civil engineers must go through a multi-year certification process that involves a significant amount of on the job work in addition to standardized examination before they are labelled a Professional Engineer (PE).
As a PE, they can face significant legal repercussions for errors in work they approve.
The benefit to the PE is that many Civil Engineering projects are legally required to have a PE design/approve them. The PE literally has a physical stamp they use to stamp blueprints and designs when they approve it. Work won't proceed without that stamp.
So, if a PE refused to approve a bridge design, presumably it would be very difficult to find another PE who would approve it (especially given the record of the first PE rejecting the plans) and the work would not proceed.
If software engineers could potentially face legal repercussions or jail time for serious errors, unsound technical judgment, or bad ethical decisions, it would enable a lot more push-back on these poorly-planned projects and corner cutting. I've always thought it was pretty sad that lawyers are held to higher professional and ethical standards than software engineers.
> I've always thought it was pretty sad that lawyers are held to higher professional and ethical standards than software engineers.
I think that's a bit silly. The vast majority of software is trivial stuff and no real harm is caused when something goes sideways. Few of us work on safety critical systems. When a lawyer screws up it often does a lot of harm and they are also inherently bound by our legal systems. Corruption and negligence/incompetence in this area lessens the public's faith in those systems.
Lawyers and civil engineer inherently take on far greater responsibility than your average Joe Schmoe programmer writing yet another CRUD app.
That's true, but what about the software engineers who do build world-changing stuff? I think that we should definitely hold them to a higher standard than we currently do.
>That's true, but what about the software engineers who do build world-changing stuff?
Yeah I can get behind that. I work in biotech/cancer diagnostics/prognostics and we could definitely afford a higher standard, especially around the core stuff (e.g. the CV/algorithms people, but also the typical IT end of things.)
First you pass the FE exam, then you work for several years under a certified PE, then you take the PE exam. It's an 8-hour open book test, though IIRC for structural engineers it might actually be a 16-hour affair over 2 days.
If you pass, you get a stamp, which lets you approve drawings as a certified PE.
If you believe something is not designed to adequate standards, as a PE it is on you to not put your seal on the drawings. They can't build the bridge without an engineer doing this.
If you're taking about options when some other engineer has already approved the design, I'm not sure what the best course is. Maybe bring your concerns to the engineer in question and work from there.
Or if the design is fine but a contractor is cutting corners in construction and not building to the specifications, you could potentially engineer a new design to reinforce the problems, or you could make them take it out and build it right.
Yeah the problem is what we acknowledge as screwing up. When a company leaks the personal data of dozen of millions of users, let private content be accessed by third party etc.. Somehow we haven't accepted that as a major failure.
And I'm not even talking about Facebook, more about things like cloud bleed. How do you even try to regulate something like this? It's not like they intended it, but they still majorly screwed up.
In this particular case, the issue is that TSB split apart from a bank that was known as Lloyds TSB. However, they continued to rent their entire IT infrastructure from Lloyds for a while after that happened. This "upgrade" was actually replacing enormous parts of their branch- and customer-facing infrastructure with an entirely new system, and it's obviously gone horribly wrong.
It's got so bad that when my partner went into our local branch they were handing out cash and writing it down in a little book because none of their in branch systems were working either.
I'm actually old enough to remember when that was normal. You could also phone your branch and the security was basically someone who recognised your voice.
57 comments
[ 2.3 ms ] story [ 110 ms ] threadStill no access to my old account this morning...
Apart from that they're both much the same, and both leagues better than a normal bank. It's amazing buying stuff or getting cash out and the notification comes through on the phone before the ATM tells you the cash is coming!
Another good thing both have - you can switch off cards instantly. I thought i'd lost my Starling card when out last week, instead of cancelling it I just temporarily halted it until it was found.
What I do love about Monzo is their approach to development - they blog everything, even the outages. Good to see the process involved in making a new bank from scratch. I'm keeping the account as a scratch account but have already moved the direct debits from my old Halifax account to Starling.
https://www.youtube.com/c/cloudnativefdn
I am most definitely joining them (but keeping my "main" bank)
As for the TSB, it would be interesting to see what went wrong and how they fixed it.
This seems to be a very common pattern, and I wonder if they are having issues because of it. I was a Monzo early adopter and even I haven't moved over entirely.
I think a lot of users _liked_ the prepaid cards, to help them budget spending money, etc. I suspect that a lot of people haven't upgraded to a full bank account with them as a result.
Basically you can put money in your account into separately named isolated areas in your account that can't be spent using your card normally. In layman's terms as many saving accounts as you want with more flexibility.
You can still kind of use it as a prepaid card by just topping it up from your bank when you want. It's just better as you can now use Android Pay etc. and have proper banking functions like having an actual sort number/account number, direct debits, standing orders etc.
My Amex is linked and I get a notification of all purchases as they go through regardless of whether I use my phone or card.
If it fails because of a failed upgrade, hacker attack, AWS not being available, etc... what is your plan B ?
Based on the original planned 50-hour outage, I suspect that TSB chose a "big bang" migration: all-or-nothing, no rollback possible. These are technically easier (so cheaper) to develop, but far far riskier than taking a phased approach.
I have successfully argued against such approaches in the past, due to the high risk of catastrophic failure if it goes wrong (i.e., exactly what has happened to TSB).
I regard TSB's failure as an in-the-making textbook example of "how not to do it".
I remember one migration which involved lots of internal services which were all tightly coupled, meaning updates across the whole backend, not just the part that needed it and no way to do a phased rollout. It was made worse by the fact that the update was to the platform and that depended on a DB upgrade, but that upgrade was incompatible with the old version which meant everything had to be done at once.
They had no disaster recovery plan. They got it done, but I'm sure a lot of people involved went grey early thanks to that nightmare.
Of course the business had no idea of the utter mess that caused all of this and continued to make harmful decisions in the name of saving money and further underfunding the IT department.
In the specific case of TSB, I have my suspicions about the reasons for cutting corners in this migration, given: https://www.thetimes.co.uk/article/missed-deadlines-leave-1m...
Tech support switched me over to a part of the site that had not been updated, and that worked for a while. Then it didn't. Back to the bank I go, with a new customer service representative, who really didn't grok the issues (new to the company?), and who became quite snippy.
Very aggravating.
Really? I'm with TSB since 2011 and it never worked nice for me. When I login I always get
Greetings, ???NAME??
Sabadell warned about ‘high risks’ in IT integration of TSB https://www.ft.com/content/c5157c1e-20ab-11e5-aa5a-398b2169c...
http://prensa.bancsabadell.com/en/News/2018/04/banco-sabadel...
Backup 1: http://archive.is/WnwiB
Backup 2: http://webcache.googleusercontent.com/search?q=cache%3Ahttp%...
Funny and poignant
This appears to be an IT failure comparable to the 2012 and 2015 IT failures at the Royal Bank of Scotland. These involved multi-day / multi-week full/partial outages, and multi-million pound fines.
FT: https://www.ft.com/content/9178ef60-46d9-11e8-8ee8-cae73aab7....
Background: TSB is a UK bank. They had a planned 50-hour maintenance window over the weekend (Fri 16:00 - Sun 18:00), to move away from Lloyds's systems (following the "un-merger" in 2013-2014). This went wrong, in what is likely to become a textbook example of an IT failure.
From a letter from Nicky Morgan, chair of the UK Parliament's Treasury committee, to TSB CEO Paul Pester:
> The reports of unauthorised transactions, access to other customers’ accounts, and failures of in-branch services have all the hallmarks of an IT meltdown. This is yet another addition to the litany of failures of banking IT systems. Potentially millions of customers could be affected by uncertainty and disruption.
Some interesting screenshots of people seeing debugging information in the front-end: https://twitter.com/bcleeve/status/988333991676796928 https://twitter.com/apphancer/status/988698182170357760 https://twitter.com/thejackthomson_/status/98856435451268710...
Ironically, TSB's parent company issued a press release (in error?) claiming that the migration has been a "success"! http://press.bancsabadell.com/en/News/2018/04/banco-sabadell...
In my current job I have repeatedly told non technical superiors that reverse engineering an outdated API is not a proper solution and could fall apart at any time. They do not care. It doesn't even register. Do civil engineers have this problem if they realise there might be a structural weakness?
This could involve going to the press, to a standards body in the country, to the government, to their union or industry representative body.
In the UK there are laws protecting whistleblowers who take actions like this.
As a PE, they can face significant legal repercussions for errors in work they approve.
The benefit to the PE is that many Civil Engineering projects are legally required to have a PE design/approve them. The PE literally has a physical stamp they use to stamp blueprints and designs when they approve it. Work won't proceed without that stamp.
So, if a PE refused to approve a bridge design, presumably it would be very difficult to find another PE who would approve it (especially given the record of the first PE rejecting the plans) and the work would not proceed.
I think that's a bit silly. The vast majority of software is trivial stuff and no real harm is caused when something goes sideways. Few of us work on safety critical systems. When a lawyer screws up it often does a lot of harm and they are also inherently bound by our legal systems. Corruption and negligence/incompetence in this area lessens the public's faith in those systems.
Lawyers and civil engineer inherently take on far greater responsibility than your average Joe Schmoe programmer writing yet another CRUD app.
Yeah I can get behind that. I work in biotech/cancer diagnostics/prognostics and we could definitely afford a higher standard, especially around the core stuff (e.g. the CV/algorithms people, but also the typical IT end of things.)
If you pass, you get a stamp, which lets you approve drawings as a certified PE.
If you believe something is not designed to adequate standards, as a PE it is on you to not put your seal on the drawings. They can't build the bridge without an engineer doing this.
If you're taking about options when some other engineer has already approved the design, I'm not sure what the best course is. Maybe bring your concerns to the engineer in question and work from there.
Or if the design is fine but a contractor is cutting corners in construction and not building to the specifications, you could potentially engineer a new design to reinforce the problems, or you could make them take it out and build it right.
(Context: I am the Head of Engineering at a bank.)
And I'm not even talking about Facebook, more about things like cloud bleed. How do you even try to regulate something like this? It's not like they intended it, but they still majorly screwed up.
EDIT: Sorry correction, looks like they sold the stake in 2015.