I use Atlassian Jira regularly. The UI is actively hostile to coders, making things like code blocks hard to work with, and layering on a few gratuitous stupidities like rendering the common Python filename __init__.py as an italic-_init_ dot py, or O(n) as an O and a thumbsdown emoji.
It's a good thing people don't use Jira to write softw— oh.
So they are migrating to an internal Atlassian Cloud and the production deployment (after 1 year of development) is hitting some performance issue.
Seems to be expected. Things rarely deploy flawlessly.
Sadly BitBucket isn't what it used to be ever since they dropped Mercurial and Microsoft acquired GitHub (and introduced the super generous Actions / build servers). I see no reason nowadays to recommend them.
So, bitbucket is not going to self-host, they're moving to an AWS based Atlassian product.
For many companies, it makes sense to use AWS (or a product based on it). For a company like Bitbucket, that is almost like if AWS decided to stop self-hosting, and use Google or Microsoft's cloud products instead, with their "product" just a wrapper on top of that. What is the point of Bitbucket, again? It's now a wrapper around a wrapper (Atlassian) around a wrapper. Except, it doesn't work that well.
Yeah, running a datacenter is their core business. The UI around git hosting is not that important, despite what people think. (People overestimate UI in general; like medicine, government and sports, everyone has an opinion on how to make a computer UI done right, though strangely nobody can deliver.)
It seems like Bitbucket has always been the afterthought in Atlassian's stack. Are Bitbucket Cloud and Bitbucket Data Center still two separate code bases with different APIs?
Yes. And data center is a huge pain to work with in a clustered environment, it almost feels like they want you to not use it and use their cloud offering instead.
I'm obviously biased, but we're not an afterthought. I think the best way to explain things is that Bitbucket was a startup that Atlassian acquired; we had to independently solve many cloud-specific problems while the rest of the company was largely still focused on building server products; and as the company started investing in a platform, they prioritized onboarding Jira and Confluence first which are built on a completely different tech stack than Bitbucket Cloud.
The reality is that this migration is one of the clearest signals I can point to that the company is investing in Bitbucket. Our platform teams have been awesome and given us a ton of support, including features that didn't exist before (without going into too much detail... you can probably imagine that Jira and Confluence don't have nearly the same requirements around file system access that we do).
Yes, Bitbucket Cloud and Bitbucket DC are two different teams and code bases; but we work closely together and are talking a lot about both products' roadmaps and future vision. And we even have engineering teams working together on a shared project and may do more of that in the future.
Thanks for the reply. As unsolicited feedback, Atlassian should rip off the bandaid and pick one of the code bases and have that be the go-forward platform for both cloud and on-prem. Bitbucket's two biggest competitors have largely the same code base for cloud and on-prem. The market expects feature parity between cloud and on-prem products and to not have to re-write to a different API when moving from on-prem to cloud or vice versa.
Bitbucket is an acquisition, and I’ve never been especially impressed by Atlassian’s abilities with systems integration. You can always spot the seams.
Atlassian makes me long for Trac, which at first was an epithet, but I’ve come to see it more recently as a distilled product that doesn’t distract from the job at hand. There are a lot of things that get done in the tools because the tools have features, not because the tools are good at it, or because it’s a good way to accomplish a goal.
Mostly what I use are histories, and linking between docs, commits, and builds. If confluence had a way to start a task list in the docs and move it to Jira, that would be one thing, but it can’t even do that. It’s all feature factory work instead of workflow-centric, which is what most of the users actually need.
This right here is why we will not be moving our Self-Hosted BB to their 'cloud'. I can't afford these kinds of outages and being 'small' we can yell and scream all we want and they won't care. I'd much rather have the control in my hands when something goes wrong. Goodbye BB!
I don't view this a bad thing; just a sign of recognition for the teams that did the job. Using "we" may come off as if the management is taking credit for the job. Or at least, that's how I see it.
I wrote this post and that's definitely how I meant it! Our engineers are awesome. I think the word "we" appeared more in earlier drafts and I changed it in a few places because frankly I don't deserve any credit for the majority of incredible work they did.
I worked for Atlassian for 3 years. They are a VERY engineering-focused organisation. Both of the CEOs have engineering backgrounds and the whole company is organised around engineering teams. I think you're looking too deeply into some off-handed phrasing there.
> In contrast, we accepted that there would be increased latency for writes
It didn't sound like they were throwing the engineers under the bus at all to me. They seemed to be highlighting that engineering executed well on the items management wanted them to focus on.
Sometimes I just want to quickly merge a small change (maybe a small config change, 1/2 lines) and then pull on master, branch off and start working again.
I'm regularly having to wait several minutes for the merge and while I know that there are ways around this locally it just annoys me that something so simple is taking so long.
This feels less like an apology and more like Atlassian saying "Us changing a platform which has worked a certain way for years and that breaking your workflow is YOUR PROBLEM. It's you looking at this wrong, merges have been asynchronous all along." despite our many combined millennia of experience being entirely to the contrary.
I'm really sorry if it comes across that way. We definitely don't think you did anything wrong. I also get that when things work a certain way for a long time, it's totally reasonable to expect them to keep working that way.
If I could distill the message re: slower merges down to 2 essential points, they would be this: (1) we underestimated how impactful this would be for some customers and that's on us; (2) some, in fact I think many, users believed they need to wait for the merge to complete and we wanted to clear up that misunderstanding.
From your use case, where you merge a small change and then want to pull, create a new branch, and start working again right away, I understand this directly affects you. I am surprised merges would be super slow for you if you're just merging small changes, though; average merge times are still just a few seconds. Have you opened a support case?
For many other users, I do think the UX changes we're rolling out will make a difference. There are a lot of users who would click merge, maybe 5-10 seconds would pass, and they would assume something must be wrong so they'd refresh the page and then it would look like nothing happened. Today we pushed out an update so that if you refresh the page and the merge is still in progress, you'll actually be able to see that.
FWIW we do have some longer-term work in progress that will make merges (along with basically all file system I/O) a lot faster; but it's a ways off and represents yet another significant architectural project (though much less disruptive than this one!). I didn't mention it in the article because it will take a while.
I know we don't have to wait for the merge but having an operation that is syncronous and takes <1s on my comptuer but being asyncronous and taking 30 seconds to five minutes on Bitbucket is quite infuriating. Often I want to merge a PR, then switch to my term, pull master, and start branching from the new head. Now I cannot do this. Also I want to merge a bunch of PRs it gets even more confusing.
I suggest you up the capacity on the queue so it feels syncronous and snappy like GitHub, as this is now a pain point.
We know that CI build efficacy drops with duration. People wander off to do other things, and either come back to it later than anticipated (all estimates are off by 2x, including estimating when I will check the build again), or forget entirely. Merge-build is one action, and making merges async increases the perceived build time. Perception of sequential delays is a fundamental UX concept. They are seen as a single, longer delay.
That’s not the only reason merges are synchronous, especially in the Atlassian world.
At scale, we have problems with build triggers from commits. Sometimes you have to fire manually.
At scale, tracking deployment failure is painful. The PR you merge may be in a different module than the deployment plan. So I need to merge a PR, then watch the dominoes fall. But the bigger issue is that Atlassian never finished getting deployments up to feature parity with builds. I have several dashboards that report build health, but deployment health is hard to track. It involves more vigilance and frankly it’s exhausting. Exhausting things get dropped every time people feel tired. To work around this, first you do alerts to team chat, then there are too many alerts so they go to their own channel, then people forget to check the channel, and we have a smaller version of the same problem and no solution.
At scale, I can only control whether MY code has enough tests to detect regressions prior to deployment. Breaking preprod impresses no one, even if automation should have caught it. That means a workflow of merge-build-test prior to moving on to the next task.
Running a git server is a bit more challenging than running a traditional stateless web app because git is all filesystem centric. If you use NFS, pages that require many git operations can be very slow. Or if the storage servers have a low-level API some page loads might require many round trips and increased latency or long queues can make things bad.
Gitlab went thru a similar journey from NFS to a high level git api called gitaly:
Git is already good and synchronizing between peers, but it's not low latency, so does require an extra management layer to make sure everything is correct.
I was on a project for which I built a kind of git hosting (as a side-effect of, or to support, other features of the product—we weren't a git-host-as-a-service, exactly) and ran into two things:
1) Locking. It's a pain in the ass. You're probably going to need to take it over or otherwise work around it, some how. I never did[0], but likely should have. At scale and with unreliable HTTP operations and all kinds of crazy stuff triggering writes to repos, you're going to end up with locking problems at some point.
2) Caching. Cache the hell out of metadata. Cache entire repo-wide metadata read operation output. Cache commits. Cache archives. Cache, cache, cache. Cache early, cache often.
[0] in my defense, I built the whole thing solo and there are only so many hours in a day, and that was not the only thing I was working on.
[EDIT] this was, like, 2011 or 2012 or something, so there was a lot less info floating around about how to do this, too.
I'm not certain, but my first glance at this would be that git is not a good candidate to move to a cloud, at least if you want to take all of the advantages of cloud architecture, as it is promoted by the cloud services.
You would have to think of some way of federating git repos onto local storage (local to the git process), to avoid network latency inside loops, etc. But maybe that's what Atlassian did and it still didn't work out, or maybe they thought of that but something else (have to fit into the Atlassian cloud architecture?) outweighed it.
Point being, you really need to design the cloud architecture around the way your hosted programs were designed to work.
At this point, Atlassian shows active contempt for their users. The Bitbucket change has been a nightmare. It says a lot about how they think of their users that they did not plan a less impactful changeover (like maybe eat your own dogfood first and then do a slow cutover with some forewarned customers). Throw in the generally user-hostile UIs of all their apps and the overall slowness and I wonder how Atlassian keeps customers.
>> Merging pull requests takes longer than it used to, but it is important to realize that merges happen asynchronously in the background and so you do not need to wait for a pull request to be merged. If you navigate away, it will finish while you are doing other things. We're updating the UX to make this clear.
This is terrible, sounds like this is the new norm. If I have to merge in 10 pull requests I can't wait 5 minutes per PR to see if there are going to be any merge conflicts, etc.
Ever since this popped up I'm seriously considering migrating everything away from bitbucket because of how slow merging is.
yes, this is driving me nuts. I regularly have to merge 3-4 pull requests and it's taking over a minute for each. This used to be a few clicks and now it's a 5-10 minute job making sure everything has merged correctly. Then you go and merge dev into a feature branch and have to wait another minute.
I cannot believe this isn't a priority at Bitbucket and we're being told to get used to it and that the UI will hide it. I've been putting off migrating to Github because we use a lot of integrations, but Bitbucket has suddenly become a pain point where it was okay before.
36 comments
[ 4.5 ms ] story [ 96.5 ms ] threadIt's a good thing people don't use Jira to write softw— oh.
Seems to be expected. Things rarely deploy flawlessly.
Sadly BitBucket isn't what it used to be ever since they dropped Mercurial and Microsoft acquired GitHub (and introduced the super generous Actions / build servers). I see no reason nowadays to recommend them.
For many companies, it makes sense to use AWS (or a product based on it). For a company like Bitbucket, that is almost like if AWS decided to stop self-hosting, and use Google or Microsoft's cloud products instead, with their "product" just a wrapper on top of that. What is the point of Bitbucket, again? It's now a wrapper around a wrapper (Atlassian) around a wrapper. Except, it doesn't work that well.
The reality is that this migration is one of the clearest signals I can point to that the company is investing in Bitbucket. Our platform teams have been awesome and given us a ton of support, including features that didn't exist before (without going into too much detail... you can probably imagine that Jira and Confluence don't have nearly the same requirements around file system access that we do).
Yes, Bitbucket Cloud and Bitbucket DC are two different teams and code bases; but we work closely together and are talking a lot about both products' roadmaps and future vision. And we even have engineering teams working together on a shared project and may do more of that in the future.
https://web.archive.org/web/20160303204710/http://www.itwire...
Atlassian makes me long for Trac, which at first was an epithet, but I’ve come to see it more recently as a distilled product that doesn’t distract from the job at hand. There are a lot of things that get done in the tools because the tools have features, not because the tools are good at it, or because it’s a good way to accomplish a goal.
Mostly what I use are histories, and linking between docs, commits, and builds. If confluence had a way to start a task list in the docs and move it to Jira, that would be one thing, but it can’t even do that. It’s all feature factory work instead of workflow-centric, which is what most of the users actually need.
https://confluence.atlassian.com/bbkb/what-is-atlassian-s-bi...
> our engineering teams prioritized their efforts to ensure that Bitbucket
It's not a coincidence that company that refers to "our engineers" rather than "we" is having basic engineering problems.
> Our focus was primarily on read operations
> In contrast, we accepted that there would be increased latency for writes
It didn't sound like they were throwing the engineers under the bus at all to me. They seemed to be highlighting that engineering executed well on the items management wanted them to focus on.
Sometimes I just want to quickly merge a small change (maybe a small config change, 1/2 lines) and then pull on master, branch off and start working again.
I'm regularly having to wait several minutes for the merge and while I know that there are ways around this locally it just annoys me that something so simple is taking so long.
This feels less like an apology and more like Atlassian saying "Us changing a platform which has worked a certain way for years and that breaking your workflow is YOUR PROBLEM. It's you looking at this wrong, merges have been asynchronous all along." despite our many combined millennia of experience being entirely to the contrary.
If I could distill the message re: slower merges down to 2 essential points, they would be this: (1) we underestimated how impactful this would be for some customers and that's on us; (2) some, in fact I think many, users believed they need to wait for the merge to complete and we wanted to clear up that misunderstanding.
From your use case, where you merge a small change and then want to pull, create a new branch, and start working again right away, I understand this directly affects you. I am surprised merges would be super slow for you if you're just merging small changes, though; average merge times are still just a few seconds. Have you opened a support case?
For many other users, I do think the UX changes we're rolling out will make a difference. There are a lot of users who would click merge, maybe 5-10 seconds would pass, and they would assume something must be wrong so they'd refresh the page and then it would look like nothing happened. Today we pushed out an update so that if you refresh the page and the merge is still in progress, you'll actually be able to see that.
FWIW we do have some longer-term work in progress that will make merges (along with basically all file system I/O) a lot faster; but it's a ways off and represents yet another significant architectural project (though much less disruptive than this one!). I didn't mention it in the article because it will take a while.
I suggest you up the capacity on the queue so it feels syncronous and snappy like GitHub, as this is now a pain point.
That’s not the only reason merges are synchronous, especially in the Atlassian world.
At scale, we have problems with build triggers from commits. Sometimes you have to fire manually.
At scale, tracking deployment failure is painful. The PR you merge may be in a different module than the deployment plan. So I need to merge a PR, then watch the dominoes fall. But the bigger issue is that Atlassian never finished getting deployments up to feature parity with builds. I have several dashboards that report build health, but deployment health is hard to track. It involves more vigilance and frankly it’s exhausting. Exhausting things get dropped every time people feel tired. To work around this, first you do alerts to team chat, then there are too many alerts so they go to their own channel, then people forget to check the channel, and we have a smaller version of the same problem and no solution.
At scale, I can only control whether MY code has enough tests to detect regressions prior to deployment. Breaking preprod impresses no one, even if automation should have caught it. That means a workflow of merge-build-test prior to moving on to the next task.
Gitlab went thru a similar journey from NFS to a high level git api called gitaly:
https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-...
https://gitlab.com/gitlab-org/gitaly
There are some other projects like this one that seek to address the problem:
https://github.com/takezoe/gitmesh
Git is already good and synchronizing between peers, but it's not low latency, so does require an extra management layer to make sure everything is correct.
1) Locking. It's a pain in the ass. You're probably going to need to take it over or otherwise work around it, some how. I never did[0], but likely should have. At scale and with unreliable HTTP operations and all kinds of crazy stuff triggering writes to repos, you're going to end up with locking problems at some point.
2) Caching. Cache the hell out of metadata. Cache entire repo-wide metadata read operation output. Cache commits. Cache archives. Cache, cache, cache. Cache early, cache often.
[0] in my defense, I built the whole thing solo and there are only so many hours in a day, and that was not the only thing I was working on.
[EDIT] this was, like, 2011 or 2012 or something, so there was a lot less info floating around about how to do this, too.
You would have to think of some way of federating git repos onto local storage (local to the git process), to avoid network latency inside loops, etc. But maybe that's what Atlassian did and it still didn't work out, or maybe they thought of that but something else (have to fit into the Atlassian cloud architecture?) outweighed it.
Point being, you really need to design the cloud architecture around the way your hosted programs were designed to work.
This is terrible, sounds like this is the new norm. If I have to merge in 10 pull requests I can't wait 5 minutes per PR to see if there are going to be any merge conflicts, etc.
Ever since this popped up I'm seriously considering migrating everything away from bitbucket because of how slow merging is.
I cannot believe this isn't a priority at Bitbucket and we're being told to get used to it and that the UI will hide it. I've been putting off migrating to Github because we use a lot of integrations, but Bitbucket has suddenly become a pain point where it was okay before.