After listening to in-person talks from Atlassian in the past, I got the impression of them having a smugness from being a big fish in a small pond (Australian tech.)
Their original and early Jira product was great and put them out in front, but I couldn’t name any other product of theirs except Bitbucket which is very much a second place competitor in their market. Jira itself has faded and been caught up with too.
Reading through the post and seeing the level of support some people on Twitter have been tweeting about, I would take this as a great time to look at other offerings out there.
Having dealt with the pre and post sales support quite extensively over the years, I always got the impression they were a big fish in way over it's head, and with a tail that didn't work so good.
A lot of people look down on Jira. My experience was that Confluence was much worse. It turns writing a couple technical notes into a publishing journey with missteps and failures along the way.
For me Confluence dramatically improved in experience when I learned that it's possible to write files locally using $FAVORITE_EDITOR in confluencewiki format (or markdown if you don't care about macros) and post them through API.
Can you share anything more about your technique? As far as I know, Confluence hasn't supported a markup format natively for several years. I've found some tricks to allow me to author stuff in Markdown but it seems to come with a lot of limitations which ultimately mean that I have to start using the web editor as soon as I want to start collaborating with others.
In Confluence Editor, under the Macro "Plus" button, you should find an entry called "Markup" pretty far up the list, which accepts "Confluence Wiki" Markup (which is just Markdown + Macros). This works for both on-premise as well as cloud instances.
I wrote a tool that automates the posting part[1]. Unfortunately I could not solve the collaboration problem either. Setting the body of the page overwrites it.
I did see some examples of editing HTML through similar approach but never tried it myself.
I don't think you could attribute Trello to Atlassian. Maybe all the junk they've added to it.
The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about.
Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
Having used Asana in several organisations, I think it being a suitable replacement for JIRA says more about how poor a choice JIRA was for situation than anything about Asana.
I've tried it before, it's just as bloated as Jira and performance suffers as a reason. If I'm going to suffer slowness anyway I might as well stick with Jira.
I don't recommend anyone to swap from JIRA. Stability is probably more important for a PM tool, but for new projects trying out something new makes sense.
It's an important highlight that Atlassian is deprecating all of their on-premise offerings and forcing everyone to the cloud because it's "the future". And then promptly demonstrated how poor of stewards of the cloud they are.
it isn't nonsense you already can't buy new server products. Been like that since Feb 2021 and support ends entirely Feb 2024. https://www.atlassian.com/migration/assess/journey-to-cloud. Try not to defend companies that don't deserve it.
Not true at all. DC is still available for the next decade. It's just not at a price that smaller sized customers can afford to pay, so your point does apply to that group.
They tried to sell my company on their cloud offering but once we described our current deployments scale they backed away since it was larger then they had the capability to support.
Good on them for doing a relatively detailed, honest and public review like this. But goddamn that recovery time is absolutely unacceptable for a modern mission-critical product like this, especially since they've essentially forced everyone onto their cloud platform. Not having the ability to automatically restore backups in 2022, when you're a MAJOR SaaS provider that stores massive amounts of essential data is just mind boggling to me.
Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.
I think what they're implying is that they can restore _everyone's_ data quickly (i.e. 'usual' infra failure), and one Jira instance's data fairly quickly (but some manual steps), but the problem is they nuked a large subset of customers while everything else kept running. This means that they can't do a complete automatic restore, and instead have to deal with it customer at a time.
I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out.
The list of affected data that was wiped out by the compliance delete and now requires recovery also includes the metadata in their orchestration system, and what sounds like other multi-tenant data stores. It's not just restoring a single, entire PostgreSQL database from backups.
Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
I agree that the timeline is not really acceptable. I am not sure anything mission critical should be hinged off a ticketing system, though. At best, things should be reported to the ticketing system, but never triggered from it as some sort of source of truth.
The thing is that the data was wiped as part of a "Compliance Delete", albeit a wrongly executed one.
I am actually more surprised that they are able restore the data at all. If you can recover from a Compliance Delete, is it really a Compliance Delete?
If immutable backups can be nuked by a single compliance delete request why bother making backups at all? All it takes is a delete request per day to get rid of them.
By removing data and only keeping a 30 day backup any compliance deletes still simply take ~a month to complete.
It's clearly not mission critical, Atlassian has just demonstrated that.
"Move fast and break things" and "mission critical" are opposing forces that just do not mesh.
My team and I run emergency services communications infrastructure with 99.999% availability requirements and our company has to pay an abatement if we don't maintain those levels. That's actually mission critical. We don't run ANY script or SQL command in production that deletes data without having tested it in our dev/test environment, documented the steps of the change and the results of testing, having the change reviewed internally, and approved by the government.
We can get these changes approved and into production in around a week normally.
"Move fast and break things" is great, until the wrong thing breaks.
> Additionally, the majority of restored customers have had no data loss, while some have reported data loss for up to 5 minutes prior to the incident.
This phrasing is ambiguous on a major question.
If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.
> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs.
Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
I'd guess they mean in in the "blameless post mortum" way, in that it wasn't some Random User's fault, it was a total process breakdown that led to said hands-on-keyboard executing with the wrong information and safeguards in place
It helps to write post mortems like this. If you start writing them in a way that suggests blame, you miss out on process issues and everyone clams up the next time it's post mortem time.
I mean, if it’s got flags to represent what you’re asking it to do and you set the wrong flag that’s not a script issue. I can run rm -rf / and that’s not a fault of rm, that’s a behind the keyboard issue.
No. That is a fault of rm that it makes it so easy to do something that in 9.99999% of the cases isn’t what the user wants to do.
In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this.
Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
Adding a hardcoded extra check for / to rm feels like theater because it barely makes it more safe. It's nearly as damaging to most Linux systems to remove /bin or /usr or any number of other directories. If you wanted to actually solve this problem, it seems like you would need to add some kind of "dangerous-to-remove-this" metadata in the filesystem.
Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
It's a process issue that a script like that is ~commonly used on production data. Processes should be in place where that's essentially impossible to do wrong. "oops I passed the wrong flag" isn't user error, that's system error that it was set up like that.
On most modern systems you can't, not without going out of your way to specify --no-preserve-root.
It sounds like they have a "type system" problem, e.g. you shouldn't be able to give a script an ID for a "cloud site" when it was expecting an ID for an application. That is, this sort of outage should not have occurred because it should not have been representable to begin with.
rm literally has a specific handling of / that requires the `--no-preserve-root` flag to allow you to run that. Software should generally warn before being super destructive, and it's usually best if the process for doing something super destructive and day-to-day use are very different.
It was faulty in the sense of allowing it to be executed with incorrect human inputs, and in such a manner that there was no recovery process in place for what happened.
Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions.
I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete
script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable.
Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
Yes. Having the safe thing and very dangerous thing just be a boolean flag apart is bad. It increases the odds of the wrong flag being used at some point.
It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before.
While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
There’s some assumption here that the script directly updated/deleted customer data.
In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.
Okay, so the script was faulty and the data it was running on was faulty. But I see no mention of testing. In my entire I’ve never seen initiatives like this not have a staging environment with dry runs and validation. This is bad engineering practice.
“End-to-end, it takes between 4 and 5 elapsed days to hand a site back to a customer.” Cool marketing strategy. Maybe you’ll get your data in a business week.
> Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated.
This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female.
A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms.
I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.
All databases should offer an --i-am-a-dummy mode, and probably enable it by default in clients. It prevents you from running update or delete statements that don't have a where clause.
That's why some databases have a "where condition required on update" so you won't hit that wall (assuming there were no other where conditions involved).
Does this event make you re-consider using their software?
Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.
It was down only for the 400. But some of them were bigger than others. In general though, it's natural for the affected customers to he more vocal than the unaffected customers, and given a large portion of the target market use Reddit/Hacker News, it's to be expected that you'll see a lot of comments.
Interesting, looks like my original estimate that they were trying to restore each customer individually a backup wasn't even that far off.
Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.
My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers.
Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases)
Their backup strategy was just to restore RDS backups (recover whole database with many customers data together)
But they can't simply restore from RDS backups due to the co-mingling of customers.
As otherwise unaffected customers would now lose data.
I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
Interesting hypothesis, and if true, I have the utmost empathy for the engineering teams that have to figure this out a way to kludge all the right data together. The thought of extracting the lost data from the sharded backups, and then merging it into a possibly a shard with good, newer customer data makes my skin crawl.
This ID thing makes no sense. Did the team doing the deletion take these cloud site IDs and plug then into a script expecting app IDs? And it still found and deleted sites? Or is the script like a delete anything script? Give it an ID and it will go find it in any table and delete it? That seems to be what they are implying.
91 comments
[ 3.3 ms ] story [ 155 ms ] threadTheir original and early Jira product was great and put them out in front, but I couldn’t name any other product of theirs except Bitbucket which is very much a second place competitor in their market. Jira itself has faded and been caught up with too.
Reading through the post and seeing the level of support some people on Twitter have been tweeting about, I would take this as a great time to look at other offerings out there.
You lost me there. In no way was FogBugz a better product. It was at best mediocre by 2010.
Jira is much better. We also use SharePoint and Confluence is so much better.
Honestly I get the criticism of both JIRA and Confluence but till you've used the other big names in that space you don't realise how bad it can get.
I'm so sorry. Fogbugz has been in maintenance mode for years now, stuck in the middle of a half-finished redesign.
https://techcrunch.com/2010/09/29/atlassian-buys-mercurial-p...
I did see some examples of editing HTML through similar approach but never tried it myself.
[1]: https://github.com/VTimofeenko/confluence_poster
I haven’t used it in years, but that seems accurate. I could never find anything unless I knew exactly how it was worded in what I’m looking for.
That is, the pop-up results that appear as I typed where significantly better than the actual "search results" page.
The other two I've honestly never heard of but that says nothing about them as products except they don't get talked about.
Funny edit, Atlassian actually made a plugin for Jira called greenhopper that was very much like trello. But they discontinued it only to realise a few years later that they liked trello and paid a massive amount of money for it. If that doesn't exemplify incompetence of a companies strategy / directorship, then I don't know what does.
I have yet to find something as good, even with all of JIRAs pain points
Maybe I should check it out again
Tell this to all the big corps still running a slow, bureaucratic JIRA system demanding one to update multiple tickets multiple times.
https://www.atlassian.com/licensing/data-center#data-center-...
Hopefully this event has enough of an impact for them internally (and financially) that they seriously reevaluate things and maybe invest in some SRE capability, and better tooling.
I'm sure that they're going to make their customer at a time thing properly automatic now... Still a big failure, but a little more subtle than you make out.
(disclaimer - worked for Atlassian years ago)
Lesson learned is not not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong inputs.
I am actually more surprised that they are able restore the data at all. If you can recover from a Compliance Delete, is it really a Compliance Delete?
By removing data and only keeping a 30 day backup any compliance deletes still simply take ~a month to complete.
My team and I run emergency services communications infrastructure with 99.999% availability requirements and our company has to pay an abatement if we don't maintain those levels. That's actually mission critical. We don't run ANY script or SQL command in production that deletes data without having tested it in our dev/test environment, documented the steps of the change and the results of testing, having the change reviewed internally, and approved by the government.
We can get these changes approved and into production in around a week normally. "Move fast and break things" is great, until the wrong thing breaks.
This phrasing is ambiguous on a major question.
If the reality is that no customers had more than 5 minutes' data loss from prior to the incident, then it'd be good to say that.
Not to mention all the data that was lost, because you can't create it, because the servers are down.
Bad script! Bad! So faulty, running itself with the wrong list of IDs and running itself in the wrong mode.
In fact, recognizing that is precisely why you can’t even do that easily anymore. Setting aside that you probably need to sudo the function, rm prevents you from running the command unless you pass an additional long flag that basically say you really really want to do this.
Programs which expect perfect user input, and allow minor user mistakes to lead to major negative outcomes are faulty programs.
Nonetheless, even with rm you can override the check you mentioned with a flag. In principle, that seems to support the post you are referring to, not refute it (not in detail but in spirit).
It sounds like they have a "type system" problem, e.g. you shouldn't be able to give a script an ID for a "cloud site" when it was expecting an ID for an application. That is, this sort of outage should not have occurred because it should not have been representable to begin with.
Lesson learned is not have any semi-automated script dependent on human input/communication/decisions without also having a recovery plan for the worst-case scenario of said script being run with the wrong input/communication/decisions.
I doubt anyone can claim that for all such potentially dangerous scripts. But something that should be reconsidered for anything like a compliance delete script, even if the design intent is to specifically bypass the normal soft-delete process for something where it has been specifically decided that it should NOT be restoreable.
Fix in this case would be to make it a two-stage process: first an easily revertible soft-delete, evaluate the impact, and then a hard delete designed to only delete data that has already been soft-deleted. Never hard-delete active, live data directly.
It's also not clear from the post if someone had used the "permanently delete" flag accidentally before. It's very possible they had, but hadn't ever run it on a wrong id before.
While yes, someone ultimately did hit "enter" on a command with the wrong ids and the wrong mode, the existence of such a script enabled that mistake to have enormous consequences. Calling such a script "faulty" feels reasonable. In the same way having tooling which allows you to delete prod database disks without any additional safeguards would be called faulty.
When I do big deletions in an automated fashion, I always make it a two-step process.
The first step identifies exactly the items which will be deleted.
This gives you the user a chance to look at it and say, "Why are we deleting 100x as many entries as we expected?"
In many tools, this is as simple as running it twice, the first time with `--dry-run`.
In a microservices-centric architecture, it’s entirely possible that the script was used to orchestrate the change across a number of other systems, and an update on one of them led to to undefined behaviour and/or unforeseen consequences.
Something went wrong. We're moving mountains to get it sorted. View our status page and subscribe for service updates.
"Something went wrong. We're moving mountains to get it sorted."
Works as axpected.
Edit: After reviewing the snippets below looks like the script has gained sentience.
This reminds me of the time I was tasked with changing the gender of a user to female on the production database (since the feature was not available through an interface at the time) and so I opened SQL Explorer on our live database and ran a quick update. Except I forgot the UserID param and accidentally changed over a million registered users to female.
A colleague came up with the idea to use the person's title as a hint to repopulate the database, which worked for Mr, Mrs and Ms.
I apologise now to all the doctors that found they had undergone nonconsensual gender reassignment.
https://dev.mysql.com/doc/refman/8.0/en/mysql-tips.html#safe...
Does this event make you re-consider using their software?
Also, I'm not clear if the site was down for all customers for a few days and now only 400 (0.18%) have an issue or if it was only down for only 400 customers. After looking at many tweets, comments here and reddit it looks like it was down for everyone but I'm not sure.
Still odd that they didn't have any tools in place to help them with it. Only having a way to restore an entire backup is only useful for disaster recovery, it's nigh useless to correct mistakes.
My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers.
Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases)
Their backup strategy was just to restore RDS backups (recover whole database with many customers data together)
But they can't simply restore from RDS backups due to the co-mingling of customers. As otherwise unaffected customers would now lose data.
I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while.
Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.