TLS means in transit interception is not easy and SPF that the address is not spoofed, but without DKIM I'm not able to verify that the content is not forged. As such, for example, my email provider is able to read those emails, and probably to modify them without me noticing.
If something doesn't grant confidentiality, data integrity and authentication, it's not encryption. I don't think TLS alone is enough to qualify an email as an "encrypted channel".
In general, you can't control how an email will travel from the original sender's system to your own, or from your own to its final recipient. Such is the nature of SMTP. A lot of modern email systems will be encrypting their connections on various hops, but it's not something you can guarantee for mail that leaves your own organisation, and even if all of the hops were encrypting the data in motion, there are still plenty of vulnerabilities while the data is at rest anywhere along the way.
In short, banks sending out PDF statements by email would be dangerous, as sillysaurus3 originally implied.
In that case, you have my sympathies. Far too many organisations, particularly those in sensitive fields like finance, medical care and (surprisingly, perhaps) law are cavalier about sending things via email that really shouldn't be. I think a lot of them genuinely don't understand that it's not a secure or appropriate medium for sending confidential information, and actually what we need to improve that situation is more like a public awareness campaign than theoretical technical solutions that most people aren't going to use anyway or legal/regulatory solutions that just punish a small proportion of the culprits who didn't know they were doing anything wrong anyway.
I tried doing that, but then I realized that I can't even select-copy-paste the data from PDF, after which I decided to continue with paper statements... there's no real advantage to PDF if I can't even export the data!
TSB's parent company had appointed an offshore 3rd party to implement the work. The CEO of TSB had hired IBM because he did not trust the offshore company's ability to deliver anymore. I agree it's likely going to cost even more in technical debt and/or setbacks if they bring in fresh eyes at this point!
Lots of working people don't feel they have the luxury to take time away from their busy lives in order to plan their finances in any great detail. I feel bad for the those now on TSB who were originally Lloyds customers even though, like you, I have several accounts and cards spread across different banks.
I haven't gone quite as far as that, but I agree with the principle.
As much as reasonably possible, I keep facilities like personal banking, personal credit cards (always at least two), any mortgage or other secured loan, and any business financial services with different organisations that aren't part of the same group. There's often small print in bank terms and conditions about being allowed to grab money from anything of yours they have access to if there is a problem with anything else, and frankly I don't trust them not to abuse that, particularly if they've already made a mistake that caused a serious problem in the first place. Also, the FSCS compensation limits tend to be per person per firm, so dividing assets can sometimes increase the amount covered if anything really bad happens to the financial firms.
And yes, I do also have some emergency cash hidden in safe places. I don't get crazy paranoid about it, but it's a substantial amount, certainly enough to fill my tank and buy essential provisions for a while. I would be concerned about becoming a truly cashless society as well, for this among other reasons.
I feel bad for the rank-and-file developers (the ones without decision-making power) who are stuck trying to fix all this in what are probably insanely bad working conditions right now.
Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions.
A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.
Sure, this definitely is true, but when your website or app is throwing an IndexOutOfBoundsException left and a InvalidBeanException right, your code quality also has issues. Security auditors (good or bad as they may be) would likely have your head for any leaked exception, for all they know the application may also be leaking other info.
I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great.
For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on.
A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick.
The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
I've worked with the UK banks. From experience they're typically large outsourcing firms that contract their work out to the large consultancies that are more interested in appearance and winning bun fights than delivering quality.
I just left one of these orgs about a month ago where the program CTO of the bank (title inflation) wanted to hire me direct because they believe they need to have a core competency that isn't aligned with external political agendas, but largely the other senior managers just see having their own techs as a risk because they can't outsource the blame.
That and directly answering questions or questioning process upsets a lot of people because they have fiedoms. Like someone asks you for all your IP address you have on cloud and what server they're associated with. They have no ability to handle the dynamic nature of these changing but their job is to write them down so you need to fall inline sorta deal...
Signs of a QA team that only unit tests or only is allowed to unit test to a change management system that is too easily bypassed.
so while I can too place some blame on developers it only takes one rogue and a bad change management system to get code out that should have stayed home
I agree with you entirely, but even then someone somewhere overruled those security auditors, right?
And if this job was falling behind, they may have contracted additional developers who came in to do work in a hurry, which can lead to an even worse and misaligned codebase, etc..
It seems possible that the deadline could not be adjusted - they were running on infrastructure that was not their own but rented for a limited time from their old owner, who is now their competitor (and has no interest in their success). As the time agreed during the restructuring runs out, they likely don't have the option of continuing as before for some time more, either they switch to something new, no matter how bad it is, or shut down.
Meh, I don't believe this.. A deal can always be made.. The old owner surely had a price (because everyone has a price) that could have been met to keep the existing systems up and running.
TSB just chose to screw their customers, and make them pay for the problems, rather than spend the money to pay for the problems themselves.
I expect that, when the dust settles, we'll see an insider account of how this all went wrong, including inexperienced developers, management ignoring numerous warnings, and an intractable disaster of a codebase.
British Banks have been playing "fast and loose" with their IT systems for a long time now. TSB won't be the last time we see such a catastrophe. It is not the first in recent memory either.
"Fast and Loose" may mean rushing changes through with unrealistic timelines, outsourcing the software development or/and infrastructure management to the cheapest location possible, sacking onshore experts to save money, etc. The UK Government is fully complicit, as their support for RBS (owned by the Government themselves) during their ruthless cost-cutting showed.
Banks are of systemic importance. People suffer if they can't make payments on time or can't access money that's needed for something essential. But the systemic importance is not reflected in the legislation: we have a situation where Banks outsource their IT to Indian service companies and do their upmost to move the "heavy lifting" of software development to the cheapest location they can find, while the social networks and search engines are recruiting the top talent.
Like the Civil Service, the senior ranks are selected by where their parents sent them to school, and how well they learned to conjugate Latin verbs while they were there. Technical skill is not only viewed with contempt it is actually a bar to promotion.
Investment banking is actually a lot saner - Thatcher’s reforms in the 80s went a long way towards destroying the Old Boys Network in that industry niche.
I worked for one of the biggest names in the business in ~ 2007 and was fairly shocked at how poor quality the IT systems were (and i wasn't particularly experienced then). Probably the worst job I have had. Crap work (might as well have been working in a factory for the creativity it and genuine problem solving that it involved) aggressive style management seemed to be what got promoted. And high turnover - I don't think anyone in my team lasted much over a year.
The thing is they had the money to throw at the problem. Just keep on hiring young devs at a bit above market rates. Th name would look good on your CV.
Banks also have lots of money to buy political influence. Underregulating an industry with systemic importance invites disaster when component failure becomes cascading failure. But sane regulations, difficult to achieve for any industry, are especially challenging for banking, and then even more difficult to maintain.
I am a TSB customers and the service has been insane.
Here's some of the highlights
- Login screen regularly throws a message from the backend saying 'java.lang.NullPointerException' (so now I know what the backend written in)
- My balance was totalled incorrectly, which made me appear to be very wealthy.
- Payments had gone missing from my statement (how do I know they were missing, one of them was my pay)
- When I did login I tried to transfer some funds and was met with 'Payment successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment worked? Who knows
- They have a fundamental scaling issue.
I've never seen anything like it. I do not get the impression this will be fixed quickly.
Although, as the article mentions, TSB are owned by a Spanish bank, Sabadell, and it was the move to Sabadell's platform, developed in Spain, that caused the trouble!
Well before the system migration they rented their online services from LLoyds, and it it was great. Also if you go into a TSB branch they're really good. So until now, I've been happy.
I have a lot of sympathy for TSB's branch staff right now. Previously I'd say they had a good reputation compared to a lot of other high street banks here, but right now they're on the front line dealing with customers who are suffering due to a catastrophic failure, and in reality there is probably very little they can say or do to help. Presumably some of those customers are very angry by now and some of them won't be very polite about it, but it's not the front-line staff's fault this happened any more than it's that customer's.
Had some friends worked for LloydsTSB a few years before it was split up, working on COBOL (sorry, brainfart). I imagine there's a lot of legacy code to interface with.
There is no way in hell I'd be attempting a transfer when the bank is in this state. I'd borrow cash from friends, take out a payday loan, sell the family jewels, anything else.
This is the crazy thing. It's all very well TSB executives making positive noises about not penny-pinching and making sure customers don't lose out, but usually when banks say these kinds of things after an outage, they are talking about things like waiving their own fees or making good any resulting late payment fees or missing interest. Obviously that sort of response would be nowhere near sufficient in this case if, for example, someone gets the utilities at their home cut off for non-payment, or a business gets into trouble for not being able to file its tax returns on time, or a small trader can't work because they can't buy materials or accept payments from customers. The true financial cost of an entire bank being out of service for a week is going to be staggering.
Even though TSB will pay you back with interest the fact that you've used the payday loan will sit on your credit file and damage your history in the eyes of some companies.
The point of a payday loan is they DON'T report to credit bureaus... I'm sorry but this is complete malarky. Payday loans are terrible terrible financial instruments and should never be used as credit...
but I have used a payday loan in the past for an emergency when my wife needed to extend her job hunt by 3 weeks and we had credit that was about to go to collections and my guaranteed annual bonus was a month out.
There are some cases where a payday loan can be used effectively to PREVENT damage to your credit history.
Since ever internal pages like your accounts, transfers etc had suffix .jsp so that's not a secret.
It's interesting to see that credit cards work fine, but debit cards and online statuses don't. I'm also lucky to have emergency funds on Monzo and in cryptocurrencies.
Jokes on you, you can't do a run on a bank where you can't login or take any money out.
More seriously, I would definitely move my money somewhere else if I had an account there. Not because I would worry that it will disappear (government insurance & all that), but just because this looks like the kind of problem which will take a long time to fix, and which could strike back.
Exactly.. you have clearly under-tested code that is in debug mode printing out all sorts of detailed error messages and systems team that is overloaded with errors and problems to triage. Talk about a ripe target.
That is stuff of nightmares
You definitely can do a run on a bank where you can't log in - you come to the office (or mail a registered letter) with written instructions to transfer all your money elsewhere and close the accounts, and legally they have a quite limited time to comply, I believe one or two business days at most, until various EU consumer protection laws for financial products kick in.
Furthermore, there's a lot of process for switching banks that can be done by the customer through the bank they're switching to. While your savings may be stuck in TSC for some days, you don't need to go to TSC at all if you'd choose to switch your wages and recurring bill payments to somewhere else, you just go to the new bank.
Like what? You can't take cash out from ATMs or from bank offices, you can't login so you can't make a transfer, some people can't use debit cards... so yeah, run.
This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this.
As far as I know Monzo is using microservices (a bank is probably big enough for that to be the right choice) to try and limit the damage of anything insane like this. The way they handled Prepaid to Current Accounts was absolutely seamless https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payment...
This has nothing to do with microservices and everything to do with lack of skill, proper engineering and chaos in IT departments of almost all banks. You can write and deploy (don't forget about the fact developers aren't skilled in ops at all and that's a common cause for these things) good software in microservices and monolith architectures, you just need to know what are you doing.
Monzo/Mondo had issues on their own, and in fact they have very frequent issues with their payment processors.
They had very frequent issues with their payment processors.
The current account (now the only account as the prepaid beta programme has ended) runs on their internally developed payment processor, and has so far had minimal issues.
> This kind of thing gives me confidence in "challenger banks" like Monzo who are building their systems in-house and I hope with a lot more testing than this.
I know someone who interviewed at Monzo and said the interview process and technical test (write a web crawler in Go) was generally a pretty good process. However they "noped!" out of there when the interviewer said that they don't do unit/automated testing, and it's the developer's responsibility to make sure their code works.
This was a while ago, so hopefully things have changed or hopefully the interviewer was not telling the truth (perhaps looking for a reaction?). However if it is the case then they're about as "modern" a bank as all the rest of these legacy banks, or at least will be in a few years time when their technical debt starts to catch up with them.
It is at odds with some of the blogs they have been posting[1], but those seem to focus on more of the frontend / app side of things. Having worked for a PSP and integrated with several of the big UK banks, I really really really hope they are actually covering the backend side as well.
Ironically, before they merged in the 90s, Lloyds had a terrible computer system, and TSB was market-leading in major bank IT customer systems, at least from a customer POV. Then they didn't learn and continue with TSB's IT systems and management, but became more lloyds instead. Sadly, this new TSB is likely only in name anything like the old one.
The same thing happened with Tangerine.ca when they launched their new website late last year. Tangerine is owned by one of Canada's "big 5" banks, Scotiabank.
For at least a week after the launch, users were experiencing login errors, timeouts, incorrect balances and features not working. Backend code exceptions would bubble up to the surface. It was ugly.
But even with the backend issues ironed out weeks later, users continue to lament about the horrible new UI, which over 6 months later, is still a UX nightmare on desktop web browsers.
Reading the 400+ Facebook comments on their "launch day post (they since removed the post from the timeline) is quite an eye opener.
Apparently they did minimal testing and disregarded all user feedback during the beta phase.
I'm a Lloyds customer, and I'm not suggesting that this is in any way related but my current account today showed a planned 0% overdraft limit of £2 shy of £10bn.
95 comments
[ 3.2 ms ] story [ 143 ms ] threadEDIT: Nope, I'm wrong. That would involve sending sensitive data over unencrypted channels, which is a no-go.
Personally I wouldn't care and I'd want that, but that's irrelevant.
Why do you think that? My inbox is plenty of (unsigned) PDFs coming from mx<n°>.<mybank>.eu ...
Headers show they're relayed from <redacted>.internal.<mybank>.eu (private IP included), that spf authentication is ok, but dkim is not even provided.
The GP was talking about encryption: can anyone else involved with any system your email passed through read your intimate financial details?
TLS means in transit interception is not easy and SPF that the address is not spoofed, but without DKIM I'm not able to verify that the content is not forged. As such, for example, my email provider is able to read those emails, and probably to modify them without me noticing.
If something doesn't grant confidentiality, data integrity and authentication, it's not encryption. I don't think TLS alone is enough to qualify an email as an "encrypted channel".
In short, banks sending out PDF statements by email would be dangerous, as sillysaurus3 originally implied.
I'm not trying to argue that it's ok, I'm saying that in my experience bank employees use email for confidential communications.
It’s lower effort for me to take an envelope and put it in a “2018 statements” folder than to go download statements.
It’s also easier to throw out that folder in 2020 than to sort out digital documents.
It doesn’t appear they were involved in the original migration.
I can't understand people with one account, one card, and 10 pounds in their pockets.
See also recent Swedish fears that their cashless society is very vulnerable to a digital attack.
As much as reasonably possible, I keep facilities like personal banking, personal credit cards (always at least two), any mortgage or other secured loan, and any business financial services with different organisations that aren't part of the same group. There's often small print in bank terms and conditions about being allowed to grab money from anything of yours they have access to if there is a problem with anything else, and frankly I don't trust them not to abuse that, particularly if they've already made a mistake that caused a serious problem in the first place. Also, the FSCS compensation limits tend to be per person per firm, so dividing assets can sometimes increase the amount covered if anything really bad happens to the financial firms.
And yes, I do also have some emergency cash hidden in safe places. I don't get crazy paranoid about it, but it's a substantial amount, certainly enough to fill my tank and buy essential provisions for a while. I would be concerned about becoming a truly cashless society as well, for this among other reasons.
Yeah yeah "get a different job" or "you made the mess, you fix it" but massive failures like this aren't orchestrated by the front-line developers, they happen because people many steps removed from the work make bad executive decisions.
A deadline was probably set long ago, and no one adjusted it when unexpected problems and complexities arose.
I have limited experience with financial systems (zero directly working with banks), but I've noticed a trend where institutions are willing to take huge gambles with the integrity and security of their systems if the risk isn't that great.
For them, risk was losing money, and while you'd think that having severe downtime or awful security practices would result in losing money, I didn't meet a single manager that believed this. They were happy to cut corners wherever because it got the job done faster, and under budget, and that's what they were judged on.
A mate of mine worked for a large UK-based bank, and pretty much confirmed this for me when we asked about what it was like to build web services for a huge bank. From the outside, their devs were the cream of the crop. Oxbridge educated, some ex Google or Microsoft guys, and years of experience in building huge systems, but a management culture that saw them as code monkeys, and wanted work delivered that was an inch off of negligent if it meant it was done quick.
The risk to the managers was their job, but from an upper management perspective all that mattered was whether it would directly affect the bottom line in an obvious (non-tech) way. Bugs, security breaches, or poor availability didn't affect this, and were worthwhile risks.
I just left one of these orgs about a month ago where the program CTO of the bank (title inflation) wanted to hire me direct because they believe they need to have a core competency that isn't aligned with external political agendas, but largely the other senior managers just see having their own techs as a risk because they can't outsource the blame.
That and directly answering questions or questioning process upsets a lot of people because they have fiedoms. Like someone asks you for all your IP address you have on cloud and what server they're associated with. They have no ability to handle the dynamic nature of these changing but their job is to write them down so you need to fall inline sorta deal...
so while I can too place some blame on developers it only takes one rogue and a bad change management system to get code out that should have stayed home
And if this job was falling behind, they may have contracted additional developers who came in to do work in a hurry, which can lead to an even worse and misaligned codebase, etc..
TSB just chose to screw their customers, and make them pay for the problems, rather than spend the money to pay for the problems themselves.
Edit: Turns out I missed the most important factor. According to https://news.ycombinator.com/item?id=16939848, they had brought in IBM.
"Fast and Loose" may mean rushing changes through with unrealistic timelines, outsourcing the software development or/and infrastructure management to the cheapest location possible, sacking onshore experts to save money, etc. The UK Government is fully complicit, as their support for RBS (owned by the Government themselves) during their ruthless cost-cutting showed.
Banks are of systemic importance. People suffer if they can't make payments on time or can't access money that's needed for something essential. But the systemic importance is not reflected in the legislation: we have a situation where Banks outsource their IT to Indian service companies and do their upmost to move the "heavy lifting" of software development to the cheapest location they can find, while the social networks and search engines are recruiting the top talent.
Investment banking is actually a lot saner - Thatcher’s reforms in the 80s went a long way towards destroying the Old Boys Network in that industry niche.
The thing is they had the money to throw at the problem. Just keep on hiring young devs at a bit above market rates. Th name would look good on your CV.
Banks also have lots of money to buy political influence. Underregulating an industry with systemic importance invites disaster when component failure becomes cascading failure. But sane regulations, difficult to achieve for any industry, are especially challenging for banking, and then even more difficult to maintain.
Here's some of the highlights
- Login screen regularly throws a message from the backend saying 'java.lang.NullPointerException' (so now I know what the backend written in)
- My balance was totalled incorrectly, which made me appear to be very wealthy.
- Payments had gone missing from my statement (how do I know they were missing, one of them was my pay)
- When I did login I tried to transfer some funds and was met with 'Payment successful' followed by a java.lang.IndexOutOfBoundsException. Had the payment worked? Who knows
- They have a fundamental scaling issue.
I've never seen anything like it. I do not get the impression this will be fixed quickly.
I'm sure there are plenty of people waiting to do just that.
Edit: I'm prone to idiocy.
There is no way in hell I'd be attempting a transfer when the bank is in this state. I'd borrow cash from friends, take out a payday loan, sell the family jewels, anything else.
Even though TSB will pay you back with interest the fact that you've used the payday loan will sit on your credit file and damage your history in the eyes of some companies.
but I have used a payday loan in the past for an emergency when my wife needed to extend her job hunt by 3 weeks and we had credit that was about to go to collections and my guaranteed annual bonus was a month out.
There are some cases where a payday loan can be used effectively to PREVENT damage to your credit history.
And I've also been told (By financial industry credit checks) it being on a report is seen unfavourably.
Since ever internal pages like your accounts, transfers etc had suffix .jsp so that's not a secret.
It's interesting to see that credit cards work fine, but debit cards and online statuses don't. I'm also lucky to have emergency funds on Monzo and in cryptocurrencies.
More seriously, I would definitely move my money somewhere else if I had an account there. Not because I would worry that it will disappear (government insurance & all that), but just because this looks like the kind of problem which will take a long time to fix, and which could strike back.
Furthermore, there's a lot of process for switching banks that can be done by the customer through the bank they're switching to. While your savings may be stuck in TSC for some days, you don't need to go to TSC at all if you'd choose to switch your wages and recurring bill payments to somewhere else, you just go to the new bank.
I've switched banks over far far less. I don't know how TSB has any customers left when this is done.
Like what? You can't take cash out from ATMs or from bank offices, you can't login so you can't make a transfer, some people can't use debit cards... so yeah, run.
As far as I know Monzo is using microservices (a bank is probably big enough for that to be the right choice) to try and limit the damage of anything insane like this. The way they handled Prepaid to Current Accounts was absolutely seamless https://monzo.com/blog/2018/04/05/how-monzo-to-monzo-payment...
Monzo/Mondo had issues on their own, and in fact they have very frequent issues with their payment processors.
The current account (now the only account as the prepaid beta programme has ended) runs on their internally developed payment processor, and has so far had minimal issues.
More information here: https://monzo.com/blog/2018/04/04/ending-prepaid/
I know someone who interviewed at Monzo and said the interview process and technical test (write a web crawler in Go) was generally a pretty good process. However they "noped!" out of there when the interviewer said that they don't do unit/automated testing, and it's the developer's responsibility to make sure their code works.
This was a while ago, so hopefully things have changed or hopefully the interviewer was not telling the truth (perhaps looking for a reaction?). However if it is the case then they're about as "modern" a bank as all the rest of these legacy banks, or at least will be in a few years time when their technical debt starts to catch up with them.
[1] https://monzo.com/blog/2016/04/26/automated-testing/
User friendly error message of the year!
[https://twitter.com/thejackthomson_/status/98856435451268710...]
The same thing happened with Tangerine.ca when they launched their new website late last year. Tangerine is owned by one of Canada's "big 5" banks, Scotiabank.
For at least a week after the launch, users were experiencing login errors, timeouts, incorrect balances and features not working. Backend code exceptions would bubble up to the surface. It was ugly.
But even with the backend issues ironed out weeks later, users continue to lament about the horrible new UI, which over 6 months later, is still a UX nightmare on desktop web browsers.
Reading the 400+ Facebook comments on their "launch day post (they since removed the post from the timeline) is quite an eye opener.
Apparently they did minimal testing and disregarded all user feedback during the beta phase.
The comments are still accessible from the post photo here: https://www.facebook.com/TangerineBank/photos/pb.46798899664...
Are they using IBM technologies for their stack?
https://news.ycombinator.com/item?id=16910947
- 'TSB gave me access to someone's £35,000' (bbc.co.uk)
- TSB Bank: Botched upgrade has left customers unable to access their accounts
- this
- Warning signs for TSB's IT meltdown were clear a year ago, according to insider