As a CEO of a new/small company I want to own problems like this guy does. Crappy problem for hisbteam, but, i can still respect a person/company that puts it out there after.
In my personal experience, it's not always a good idea to do it like this. I've been quick to offer sincere apologies for what were big problems from my own perspective. That unfortunately turned out to affect only a small amount of my customers, confusing and annoying many others.
I still appreciate the intention to strive for transparency, but you need to do it right. If I ever do it again, I'll need to make sure that I know for sure how many people it's affecting, and how much.
What examples can you provide of "being transparent and sincere in trying to do better" back-firing? Did massive clickbait articles get reported, blowing up the issue?
I have a tendency to be over impulsive, in where I get personally emotional over customer emails sent to me, and then I work my ass off to fix the bugs, reports, complaints etc reported in the mails..
Then I figure out that I can't fix everyone's problems by fixing bugs only, get emotional, write a weird and over-emotional post, and just leave lots of customers behind at the end of the process.
It's still a small-scale issue if you ask me, as it is mostly personal, but it's an issue for sure.
I'd steered clear of this company to date, and was about to snort at the "apology" but this:
> If you were forced to use an alternative payment method and therefore incurred fees, we will fully reimburse you for this amount.
does make me waver, they're obviously trying to do the right thing.
I'm still confused about whether they provide the same guarantees other organizations do under the FSCS. I wouldn't feel comfortable without this. Responses by their team to questions about this suggest they protect your deposits under some other regulation... but since it's different I don't understand it yet and trust it less.
What does it mean when Nikolay uses the title and language,
>No excuses — we let you down
>An apology from the Founder & CEO of Revolut
>The last 24 hours have been painful to say the least. Painful for our engineers who have been working around the clock to resolve the issue, but even more painful for our customers who have been unable use their accounts.
and
>You will get no excuses from me.
and
>We have let you down and I take full responsibility for what happened.
for what is essentially a very minor brief issue. He says:
>I would like to reassure you that all personal data and customer funds were completely secure at all times.
So I would like your help interpreting why he uses such completely inappropriate and over-the-top apology language for what amounts to brief [Edit: 1 day of] downtime.
It’s the Californian English style of expression, which is all about superlatives and deals only in extremes. With the exploding cultural influence of California in tech, it has spread and you now hear it from many people working in tech, regardless of whether they are Californian or not.
Edit: I guess some Californians got upset and decided to downvote me? It’s not negative in any way, just a different style of expression. You certainly will never hear a Japanese or Romanian person exclaim “this is so amazing!” on a regular basis.
I didn't downvote, but suspect some of them are because you offered an unlikely or minor reason behind the superlatives.
Also, I'm unsure of your exposure to Japanese, but superlatives are used frequently in daily conversation. "Sugoi"/"Suge" ("great/amazing!") or " kawaii" ("cute!") are a couple examples. There's some nuance and loss in translation, but the general usage and frequency is similar to American English from my experience with my Japanese in-laws, friends, and employees.
I do agree that Californians and our sun-fried brains seem more likely to take it to an extreme.
I am one of those people. If I couldn't access funds due to some service problem, I would stop using the service unless I saw a good admission of the problem, and explanation of cause, and a description of what the company will do to avoid further issues.
This reply contained all three of those. Kudos to the team for owning up to the problem and assuring their users that they take their money seriously.
For 14 hours or so people couldn't access their funds. That's the sort of error that can easily be enough to sever ones relationship to a company, depending on individual context.
The business risks of being overly-apologetic are minimal; those who were not affected, or didn't care, will dismiss the statement and carry on.
However, the reverse is not true; an insufficiently-apologetic email would only serve to enrage customers who were already upset. These users would then be exponentially more likely to sever their relationships with the company.
Although I agree that this approach is a little off-putting, I think it may only read that way to someone not in that second group, which is really who the message is targeting.
Revolut positions itself as a full service bank, including account number and sort code - they will have users that understandably use their product as their primary account. There will be regular users that will be unable to pay for lunch, and business users that will be unable to pay employees and suppliers (don't forget that it was last Friday of the month, a common payday in the UK). To let their customers go without access to their funds for an entire working day is not a 'brief downtime', it's completely catastrophic.
I'm also not surprised that user's did not hear back from support. I was formally a Revolut premium user that used the 'premium support', and they frequently had support response times >4 hours (one time over >6 hours). It's no surprise that when something serious happens their support infrastructure completely collapsed.
I want to give them the benefit of the doubt over their downtime, but I struggle to understand how it took them an entire working day to realise that an application server was playing up, and why a single server was able to take down up to 75% of their transaction processing?
So for why I needed help understanding the tone, I think I read too many crypto exchange announcements, where this kind of apology usually accompanies "we lost everyone's money. we're shutting down." That's why it seemed strange to me.
The title "there is NO excuse for 1 day of downtime" just seems so superlative to me.
It just seems to me that this is just an inconvenience. You yourself use the word "frequently" to describe the frustration you felt. a singe occurrence once doesn't seem like a catastrophe to me, though another respondent told me that it was.
For me, I don't consider it catastrophic if I can't access banking for 1 day. At any rate, thanks for the responses. It's clear some people consider this catastrophic.
This is why exercising your backup systems in production is critical. our company regularly switches to our secondary data center regularly and runs production traffic on it to ensure they failover actually works in reality, not just theory.
It seems to be often the main database that on failure catches startups on the wrong foot.
The GitLab incident comes to mind.
The easiest way to run a database is to set it up as a SPoF.
You want to be serious and set everything up with redundancy, but then in case of problems it is just too easy to mess something up by doing the wrong thing when switching back or promoting a stale replica or something.
I don't even know if there are proper hosted replicated SQL-ACID databases-as-a-service available one would consider using for financial services and similar use cases.
There seems to be merit in paying DBAs proper money to handle these things.
Why aren't the cloud offerings from Google, AWS, etc good enough? My understanding is that their infrastructure is PCI compliant, and there are a few banks already on this infrastructure. Is this simply a case of NIH syndrome, or are there regulations that prevent this?
ING uses Cassandra, which has no single point of failure, has been around for almost a decade, and would have made gitlab’s mistake a nonissue (and almost certainly would have made this one a nonissue as well - single instance failures don’t matter and linear scalability is a thing)
1. there's no monitoring in place
2. fail over is manual
3. there are no standby servers
4. nothing auto scales
5. backups are not tested
6. they lack sufficient manpower
7. there are no clusters
8. the service is one error away from an outage
It sounds like they completely ignored the operations side of running a service, and finally defaulted on technical debt. Yikes!
I hope the best for them, but they have to start taking operations seriously.
Even lower, if my experience with clients is any clue.
Most people I've seen running companies are either clueless in general in IT matters (business or engineering people), or, "older" gents who grew up long before IT security and robust backups were "the norm" and stressed in every class and conference. If there's no direct money to be made, it doesn't matter.
Only in the last 3 years or so have I seen ANYONE of my clients care about potential hackers and breaches. And one of them only cared... because they were hacked by the Chinese.
You are making a lot of assumptions based on that blog post. Unless you have other information, I'm confident you don't have it straight and that you are wrong.
None of those points are confirmed by the post. Maybe apart from the sufficient manpower to handle support during emergencies. (But who has that, honestly?)
Some of your points are almost dangerous. Not automatically failing over something handling large number of financial transactions may be a good decision, unless you can guarantee consistency between environments. Same goes for autoscaling databases (if you can even autoscale them in that environment - I'd expect them to be write-heavy and transactional, which is a good reason for scaling up, not out (or sharding or other things, not the point))
Maybe that sort of contrition works in relationship arguments or parole hearings, but I hardly find it appropriate here.
"At around 07:00 BST on Friday morning, our transaction database began to malfunction. Naturally, we followed procedure and switched to a backup server. Unfortunately, the backup server began to drastically slow down and was struggling to process live transactions."
That is very vague and sounds like a euphemism for "we don't know who changed the config file but it broke production and so we had somebody spin up a new instance on their laptop".
I any case I was in a foreign country when my card (and those of my travelling companions) stopped working without warning or explanation. I don't want him to be sorry, I want a rationale for why I shouldn't switch to an equivalent competitor like Monzo (on the assumption they have better reliability).
It's not a post from engineering. It's an announcement to customers. Most will neither understand nor care about the details of that infrastructure failing. What you want is an engineering post mortem which is a completely different kind of post. Don't get me wrong, I want them to publish one and would love to read the details. But their engineering team has (hopefully) much more important things to do at the moment.
You guys are great. The largest Indian bank is offline almost every night and they are ok with it. Is there a place to publicly shame these kind of entities ?
To me the worst offender in revolut is that the ui is not blocked when is waiting for server responses (second is the inability to edit my preferred contacts list). It never caused me problems but I always felt uneasy (I use it more sparirily now so perhaps they fixed something). All the other problems that I care about have been slowly resolved. About the issue at hand this is not the first time the service is down but who knows, perhaps they're starting to learn their lesson. I used it for around 6 months intensively
Their UI is a mess. You'll enter the pass code, the screen will wipe out of view, revealing... Another pass code screen. Contacts that use Revolut will never be cached, and will instead always be looked up every time without any indication (I guess that's what the GP refers to).
For all the funding they have, I'd assume they could fix these, but no, they don't.
I had no idea what the product is so I clicked the logo. It went to the blog which doesn't explain what it is. Instead it should go to the main site. There's no obvious link in the blog so I had to edit the address.
That doesn't tell me much. My bank's app already provide those (except maybe the cutoff sentence) so I wanted to know more. What banks or countries is it compatible with, etc. That link doesn't lead me to the answer.
edit: After watching the video it seems it's an international online bank that allows you to have and exchange among a bunch of currencies and pay with card at the cheapest rate possible.
It doesn't interest me because I don't plan on going out of the EU, where I can pay with just the phone in the vast majority of stores.
This is a bizarrely endemic problem on official company blogs, and I've often wondered why.
I always guessed it was something like, "Many WordPress and other blog-theme designers default to thinking of the blog is the product or main website itself, so their themes link the logo to the blog itself (and perhaps(?) there's no easy setting in the GUI to change this)?"
I see this default behavior on my own personal blog[1], which uses a fairly popular free theme, but in that case it's correct because my blog is really its own standalone thing.
[1]: Hehe, did you think I was going to try to plug my personal blog here?
In this particular case I'd say it's mostly because usually you'd send someone to "/" to go home, but this blog is a subdomain so that doesn't work. It's easier to miss that.
The biggest bank, Steward Bank, owned by the much adored Strive Masiyiwa and his Econet Wireless, is offline more than half of the time. They drop transactions and mobile money balances disappear without explanation or apology. Been happening for years now.
Awkward semantics for sure, but I think they regret having to learn from the experience as opposed to learning by considering the possibilities beforehand.
No big deal -- their backup was severely unprepared to handle live transactions. It's not like these guys leaked private information or otherwise lost people money.
>"At around 07:00 BST on Friday morning, our transaction database began to malfunction."
and
>"This server slowdown has never happened before"
Why does this transparency not include meaningful details of what actually went wrong? A "database malfunction" and a "server slowdown" are technical gibberish. This is not dissimilar to the kind of crap a traditional bank says when there are problems. It reminds of hearing "The system is down" from a customer service representative.
This apology is on the company's blog alongside a blog post titled "How to extract analytics data from your iOS application", complete with Swift code snippets[1]. So clearly they aren't worried about content being too technical.
We are talking about a bank and your money, why would it not be OK to err on the side of being too technical? Or at the very least provide a link for the more technically inclined.
> Why does this transparency not include meaningful details of what actually went wrong?
Because the people who can provide that are busy. This is customer communication, not post mortem from engineering. They had engineers spending whole day with extra hours on Friday to resolve the issue, then there was the weekend. I expect they'll take a day or more to dissect it internally, plan follow up, start working on it, etc. Publishing the plan can come later. (If they plan to do that)
What would you prefer them to do first: work on the follow up plan, or spend time writing a blog post about the follow up plan?
>"What would you prefer them to do first: work on the follow up plan, or spend time writing a blog post about the follow up plan?"
This blog post was published a whole day after the outage. Also I never questioned anything regarding the lack of follow up plan. I was questioning why technical detail wasn't more forthcoming.
>"This is customer communication, not post mortem from engineering ..."
The customer comms and service status were posted on their Twitter account on the 29th.
I had to look to find what this company does because the top logo doesn't take me to the page. If I weren't currently busy I might have missed it but I'm actually in the target market.
They don't - much like their competitors (e.g. Monzo) they are making losses for every customer they take on, with the hope of monetising it later, à la Twitter.
It's nice to see Revolut owning this issue but I've had nothing but bad experiences with them since starting to use them.
I was stranded in Australia with my card at the checkout of a supermarket and my card stopped working with no explanation. An hour later, their Twitter feed told us that there was issues.
Another incident where I got cash out of an ATM. The ATM didn't spit out my cash but still debited the money. Revolut said it would take 90 days to get my money back. This was £500. At Christmas.
Unfortunately as a result of this, they lost my custom for life.
67 comments
[ 3.8 ms ] story [ 119 ms ] threadI still appreciate the intention to strive for transparency, but you need to do it right. If I ever do it again, I'll need to make sure that I know for sure how many people it's affecting, and how much.
What kind of a company are you building?
What examples can you provide of "being transparent and sincere in trying to do better" back-firing? Did massive clickbait articles get reported, blowing up the issue?
Then I figure out that I can't fix everyone's problems by fixing bugs only, get emotional, write a weird and over-emotional post, and just leave lots of customers behind at the end of the process.
It's still a small-scale issue if you ask me, as it is mostly personal, but it's an issue for sure.
> If you were forced to use an alternative payment method and therefore incurred fees, we will fully reimburse you for this amount.
does make me waver, they're obviously trying to do the right thing.
I'm still confused about whether they provide the same guarantees other organizations do under the FSCS. I wouldn't feel comfortable without this. Responses by their team to questions about this suggest they protect your deposits under some other regulation... but since it's different I don't understand it yet and trust it less.
>No excuses — we let you down
>An apology from the Founder & CEO of Revolut
>The last 24 hours have been painful to say the least. Painful for our engineers who have been working around the clock to resolve the issue, but even more painful for our customers who have been unable use their accounts.
and
>You will get no excuses from me.
and
>We have let you down and I take full responsibility for what happened.
for what is essentially a very minor brief issue. He says:
>I would like to reassure you that all personal data and customer funds were completely secure at all times.
So I would like your help interpreting why he uses such completely inappropriate and over-the-top apology language for what amounts to brief [Edit: 1 day of] downtime.
What is your take? Why is it written like this?
Edit: I guess some Californians got upset and decided to downvote me? It’s not negative in any way, just a different style of expression. You certainly will never hear a Japanese or Romanian person exclaim “this is so amazing!” on a regular basis.
Also, I'm unsure of your exposure to Japanese, but superlatives are used frequently in daily conversation. "Sugoi"/"Suge" ("great/amazing!") or " kawaii" ("cute!") are a couple examples. There's some nuance and loss in translation, but the general usage and frequency is similar to American English from my experience with my Japanese in-laws, friends, and employees.
I do agree that Californians and our sun-fried brains seem more likely to take it to an extreme.
This reply contained all three of those. Kudos to the team for owning up to the problem and assuring their users that they take their money seriously.
However, the reverse is not true; an insufficiently-apologetic email would only serve to enrage customers who were already upset. These users would then be exponentially more likely to sever their relationships with the company.
Although I agree that this approach is a little off-putting, I think it may only read that way to someone not in that second group, which is really who the message is targeting.
That's my take, at least.
Revolut positions itself as a full service bank, including account number and sort code - they will have users that understandably use their product as their primary account. There will be regular users that will be unable to pay for lunch, and business users that will be unable to pay employees and suppliers (don't forget that it was last Friday of the month, a common payday in the UK). To let their customers go without access to their funds for an entire working day is not a 'brief downtime', it's completely catastrophic.
I'm also not surprised that user's did not hear back from support. I was formally a Revolut premium user that used the 'premium support', and they frequently had support response times >4 hours (one time over >6 hours). It's no surprise that when something serious happens their support infrastructure completely collapsed.
I want to give them the benefit of the doubt over their downtime, but I struggle to understand how it took them an entire working day to realise that an application server was playing up, and why a single server was able to take down up to 75% of their transaction processing?
Lastly, if there is anyone out there considering Revolut, please be aware that despite their advertising that they do not add FX fees they actually do add fees on the weekend: https://community.revolut.com/t/weekend-exchange-rate-surcha...
The title "there is NO excuse for 1 day of downtime" just seems so superlative to me.
It just seems to me that this is just an inconvenience. You yourself use the word "frequently" to describe the frustration you felt. a singe occurrence once doesn't seem like a catastrophe to me, though another respondent told me that it was.
For me, I don't consider it catastrophic if I can't access banking for 1 day. At any rate, thanks for the responses. It's clear some people consider this catastrophic.
The easiest way to run a database is to set it up as a SPoF.
You want to be serious and set everything up with redundancy, but then in case of problems it is just too easy to mess something up by doing the wrong thing when switching back or promoting a stale replica or something.
I don't even know if there are proper hosted replicated SQL-ACID databases-as-a-service available one would consider using for financial services and similar use cases.
There seems to be merit in paying DBAs proper money to handle these things.
ING uses Cassandra, which has no single point of failure, has been around for almost a decade, and would have made gitlab’s mistake a nonissue (and almost certainly would have made this one a nonissue as well - single instance failures don’t matter and linear scalability is a thing)
Here’s one: https://academy.datastax.com/resources/ing-groep-nv-exploiti...
And another https://m.youtube.com/watch?v=-sD3x8-tuDU
(And bad as it is, that's already better than British retail banks, though.)
I hope the best for them, but they have to start taking operations seriously.
Most people I've seen running companies are either clueless in general in IT matters (business or engineering people), or, "older" gents who grew up long before IT security and robust backups were "the norm" and stressed in every class and conference. If there's no direct money to be made, it doesn't matter.
Only in the last 3 years or so have I seen ANYONE of my clients care about potential hackers and breaches. And one of them only cared... because they were hacked by the Chinese.
Some of your points are almost dangerous. Not automatically failing over something handling large number of financial transactions may be a good decision, unless you can guarantee consistency between environments. Same goes for autoscaling databases (if you can even autoscale them in that environment - I'd expect them to be write-heavy and transactional, which is a good reason for scaling up, not out (or sharding or other things, not the point))
"At around 07:00 BST on Friday morning, our transaction database began to malfunction. Naturally, we followed procedure and switched to a backup server. Unfortunately, the backup server began to drastically slow down and was struggling to process live transactions."
That is very vague and sounds like a euphemism for "we don't know who changed the config file but it broke production and so we had somebody spin up a new instance on their laptop".
I any case I was in a foreign country when my card (and those of my travelling companions) stopped working without warning or explanation. I don't want him to be sorry, I want a rationale for why I shouldn't switch to an equivalent competitor like Monzo (on the assumption they have better reliability).
It's not a post from engineering. It's an announcement to customers. Most will neither understand nor care about the details of that infrastructure failing. What you want is an engineering post mortem which is a completely different kind of post. Don't get me wrong, I want them to publish one and would love to read the details. But their engineering team has (hopefully) much more important things to do at the moment.
https://onlinesbi.com
What behaviour do you expect the app to have and why?
For all the funding they have, I'd assume they could fix these, but no, they don't.
It's the first line of text after the header.
edit: After watching the video it seems it's an international online bank that allows you to have and exchange among a bunch of currencies and pay with card at the cheapest rate possible.
It doesn't interest me because I don't plan on going out of the EU, where I can pay with just the phone in the vast majority of stores.
I always guessed it was something like, "Many WordPress and other blog-theme designers default to thinking of the blog is the product or main website itself, so their themes link the logo to the blog itself (and perhaps(?) there's no easy setting in the GUI to change this)?"
I see this default behavior on my own personal blog[1], which uses a fairly popular free theme, but in that case it's correct because my blog is really its own standalone thing.
[1]: Hehe, did you think I was going to try to plug my personal blog here?
The biggest bank, Steward Bank, owned by the much adored Strive Masiyiwa and his Econet Wireless, is offline more than half of the time. They drop transactions and mobile money balances disappear without explanation or apology. Been happening for years now.
Well, there's a weird turn of phrase. In what way is learning from the experience regrettable?
and
>"This server slowdown has never happened before"
Why does this transparency not include meaningful details of what actually went wrong? A "database malfunction" and a "server slowdown" are technical gibberish. This is not dissimilar to the kind of crap a traditional bank says when there are problems. It reminds of hearing "The system is down" from a customer service representative.
This apology is on the company's blog alongside a blog post titled "How to extract analytics data from your iOS application", complete with Swift code snippets[1]. So clearly they aren't worried about content being too technical.
We are talking about a bank and your money, why would it not be OK to err on the side of being too technical? Or at the very least provide a link for the more technically inclined.
[1] https://blog.revolut.com/how-to-extract-analytics-data-from-...
Because the people who can provide that are busy. This is customer communication, not post mortem from engineering. They had engineers spending whole day with extra hours on Friday to resolve the issue, then there was the weekend. I expect they'll take a day or more to dissect it internally, plan follow up, start working on it, etc. Publishing the plan can come later. (If they plan to do that)
What would you prefer them to do first: work on the follow up plan, or spend time writing a blog post about the follow up plan?
This blog post was published a whole day after the outage. Also I never questioned anything regarding the lack of follow up plan. I was questioning why technical detail wasn't more forthcoming.
>"This is customer communication, not post mortem from engineering ..."
The customer comms and service status were posted on their Twitter account on the 29th.
How do they make money?
Especially the "magical" debit card.
I was stranded in Australia with my card at the checkout of a supermarket and my card stopped working with no explanation. An hour later, their Twitter feed told us that there was issues.
Another incident where I got cash out of an ATM. The ATM didn't spit out my cash but still debited the money. Revolut said it would take 90 days to get my money back. This was £500. At Christmas.
Unfortunately as a result of this, they lost my custom for life.