Like it or not, fortnightly and “once every two weeks” are your only unambiguous choices.
We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
It is usually their shared runners. I experience outages with them a couple times a month. It isn't a big deal since we maintain our own private pool, so I just move jobs to them. (They make you pay for CI minutes so we use them for certain jobs.)
I use GitLab for my side project (source control only; don't have any kind of CI set up just at the moment) and it's been years since there was any really noticeable slowness for me. I can't say I use it daily, but fairly frequently.
Yes. They (GitHub) have been doing a lot worse. Last time I checked, they where completely down 2 days ago. [0] Before that, GitHub Actions was down 14 days ago: [1].
At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear.
Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
> Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover...
Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI.
I suppose this is why people just end up self hosting gitlab instead.
If you want transparent failover you need to use your own domain for the repo URL. In case of a failure you (or some automatic job) would have to change the DNS record.
Gitlab has a repo mirroring feature [1]. But of course you'd also need to sync users public keys.
Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
You can instead set up a read-only mirror so people can at least still pull and browse the code.. Gitea [2] might be a better choice than Gitlab, since it's much more lightweight and easier to host.
> Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
That's essentially the same conclusion I keep coming too, occasionally it has hit me when I go to push something but rarely has it blocked me or anyone else from continuing to work.
> You can instead set up a read-only mirror so people can at least still pull and browse the code
Yeah, this I need to do eventually just for peace of mind as a more automated backup solution. At least I don't have to care about failover.
Maybe some kind of pre/post-push hook to automatically push to a secondary repo? Not entirely sure if this is possible, I’ve never written a Git hook before.
Gitlab is completely down until just now, similar happened not that long ago. Over the last three weeks continuously had issues with the Gitlab runners (even not shared ones) were it wouldn't jobs or only after a second or third retry attempt.
I think the flakiness of jobs succeeding is also related whether you use their package registry. I can't say I had similar issues with Github.
Not downtime, but some kind of issue, yes [1]. However github is not much better [2] and gives much less detail and fewer updates. Bitbucket is more solid recently [3] (but wasn't really better than the others in 2020)
I can't even get to the issue where they're tracking this any more. At least the status page is on different infra though, hopefully they'll keep it updated.
I’m not being facetious, or implying support for master/slave nomenclature. But if you ship a big change like this that impacts very core assumptions in 99% of the Git ecosystem, you’re perhaps in for a treat.
I love Gitlab because I love open source and competition but Gitlab either on-prem or gitlab.com has been having growing pains ever since I started using it around 2015.
And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing.
Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it.
I have nothing but praise for our enterprise Gitlab implementation. It hosts 2000+ projects and for many teams is mission critical. Our team that maintains it is wonderful and I've only seen it down without planning once for about an hour. Using Gitlab at work actually swayed me to move to it for my personal projects as well.
My company does on-prem gitlab as well, and it's pretty fantastic. We're in the beginning stages of migrating off of Azure DevOps but so far we're wowed by the features.
+1 for on-prem. I tried gitlab.com first but it was very slow (this was years ago, YMMV) but on-prem has been rock-solid and fast for us hosting ~100 projects. One thing I've been surprised by is how easy updates have been, I don't think we've ever had to even reconfigure anything, including runners. I know it seems like that is how it should work, but I've had a lot of opposite experiences with the open source tier of tools.
We don't use Gitlab CI, but only their hosted git. Just thought I'd throw in our sterling experience over the past three years. We've really not noticed whatever downtime they've had.
Any chance you could drop me some details about pushes hanging? I'm an engineer in the source code team doing a lot of performance work and I might be able to correlate it with known issues or investigate it further. I know it's a bit awkward to get data after-the-fact but just some links to the problem repos would be useful. I might not be very useful if it's related to network/hardware issues but I do work in that aspect of the software at least.
You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at rmay@gitlab.com if not :)
This status page is weird, it says that there is an active incident and that there is a databate latency issues, but everything is green:
https://i.imgur.com/oXSiyTG.png
Hi, GitLab team member here. Heads-up: GitLab.com appears to be stable again. Our engineers are monitoring and investigating the root cause of the incident.
Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
Gitlab is a great service and a great alternative to github and bitbucket. But it has been quite flaky over many years and it's quite surprising they haven't prioritised system stability over everything else. https://twitter.com/gitlabstatus
Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.
My guess is it has to do with their prioritization of only hiring people with "significant ruby/rails" experience. Kills a lot of the talent pool, which doesn't make sense as easy is it is is to pick up a new high level language and framework.
/r
took me a lot of effort to convince my CTO to move from Bitbucket to Gitlab.
Since we have started using it ( 3 + months ), we have experienced more than 4 instances of downtime. If some one from Gitlab is reading this message, please consider a code freeze and spend some time on stabilizing the system.
Can you elaborate on a) what kinds of laws on software backdoors? (Mandating them? Forbidding them? What sort?), and b) how that affects the GP's request?
Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian.
Still curious about what sort of backdoor laws they have.
I use GitHub and Bitbucket fairly extensively and I find Bitbucket painfully slow at times (eg. 7.38s to load a page just now with a whopping two open pull requests on it, whilst somehow transferring 8.8MB of crap; the equivalent page on GitHub is about 6 times better on both fronts) and the UI only ever seems to get more clunky. I fear the tight integration with JIRA is slowly infecting it with JIRA-esque problems.
Agreed. I use gitlab for administering all of my courses. I started when they had more features compared to github (free private repos, built in test runners), but over time github has regained some but not all of that ground. I will still be using gitlab in the future, but am going to convince our systems admin to set up a local instance.
For those curious: the root cause for the incident is, ultimately, degraded statistics on a table.
There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency).
That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic).
Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet.
Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.
Gotcha, that makes sense to me - refactoring queries which wanna flip would be my choice as well, just thinking that if the analyze fixes plans its (in my experience) "well we could have just used this plan anyway" - both of them are short term fixes for sure.
93 comments
[ 3.7 ms ] story [ 220 ms ] threadThough from the context it's 2.
We actually had a programming interview question using biweekly at a previous job. It was about thorough requirements definition and understanding/agreeing to measurements.
"A fortnight is a unit of time equal to 14 days (2 weeks). The word derives from the Old English term fēowertyne niht, meaning "fourteen nights".
https://en.wikipedia.org/wiki/Fortnight
At least with GitLab, you can set up a self-hosted solution for free as a backup, unlike GitHub (Unless you want to pay a lot for GH Enterprise) where some have 'gone all in on GitHub Actions' and then some couldn't push that critical change [2] before the start of the weekend. Oh dear.
Maybe its time to setup a self-hosted backup VCS and not depend entirely on GitHub/GitLab web.
[0] https://news.ycombinator.com/item?id=26439075
[1] https://news.ycombinator.com/item?id=26301659
[2] https://news.ycombinator.com/item?id=26439213
I've been wanting to do this for some time purely for git hosting (don't care about CI at all) but wasn't sure about how to make it failover...
Personally i'd be happy with a headless git server, but that's not fair on everyone else who wants the GUI to browse and organise stuff, so I want gitlab etc to deal with that. What would be nice is to have a headless backup server that allowed everyone to continue pulling/pushing from the CLI with their existing repos when gitlab is down. I can't see a smooth way of doing that since it would require messing with the git remotes, unless the solution is inverted and uses a single remote pointing at the "backup" server which then replicates to gitlab, but I don't think gitlab can be configured to change the default remote when people clone it from the UI.
I suppose this is why people just end up self hosting gitlab instead.
Gitlab has a repo mirroring feature [1]. But of course you'd also need to sync users public keys.
Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
You can instead set up a read-only mirror so people can at least still pull and browse the code.. Gitea [2] might be a better choice than Gitlab, since it's much more lightweight and easier to host.
[1] https://docs.gitlab.com/ce/user/project/repository/repositor...
[2] https://gitea.io
> Downtimes are so infrequent and relatively short that this isn't worth the effort for me.
That's essentially the same conclusion I keep coming too, occasionally it has hit me when I go to push something but rarely has it blocked me or anyone else from continuing to work.
> You can instead set up a read-only mirror so people can at least still pull and browse the code
Yeah, this I need to do eventually just for peace of mind as a more automated backup solution. At least I don't have to care about failover.
https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
I think it mostly comes down to "I didn't write it, so it's bad" :)
I think the flakiness of jobs succeeding is also related whether you use their package registry. I can't say I had similar issues with Github.
1: https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
2: https://www.githubstatus.com/history (don't get fooled by them collapsing the incident list after three per month)
2: https://bitbucket.status.atlassian.com/history (also collapses after three per month)
That is correct - we also will be posting updates here: https://twitter.com/gitlabstatus/.
but I guess I'm too late to the party!
And it seems to be related to scaling because it's always iffy problems like higher error rates or jobs not executing.
Either way, I love our on-prem Gitlab. It's free and it hosts over 200 projects, Gitops and the whole shebang. I'm just now beginning to use Kubernetes from it.
They will sort this out eventually.
Pushing commits often hangs completely, and we've had a number of smaller downtimes/degradations in recent weeks that have been super disruptive.
Having been Gitlab users since our inception, we're now seriously considering Github (I know - grass is always greener).
You can tag me on an issue on gitlab.com with @robotmay_gitlab if the details can be public, or email them to me at rmay@gitlab.com if not :)
https://twitter.com/gitlabstatus/status/1371443832865222658
https://about.gitlab.com/handbook/marketing/community-relati...
Please note that about.gitlab.com contains all our static content, including our 10,000 page handbook. This typo isn’t on our main about page but deep in the marketing handbook.
I merely wanted to verify the @ was authentic, and that page was the top "site:gitlab.com" result.
The page has formatting problems, I'll rework this in a separate MR.
Gitlab has been a fully remote proponent going back a few years and these persistent reliability issues add some doubt to that model - even if there might be altogether other issues in play.
My company is full remote and we don't have these issue.
time to add a new backup remote to all our repos.
Australia has very weird and wide reaching laws regarding software backdoors.
Edit: Nvm on (b); somehow I got GitLab and BitBucket mixed up as to which one belonged to Atlassian.
Still curious about what sort of backdoor laws they have.
EDIT: I've done some more reading up on this, and The Assistance and Access Act of 2018 explicitly states that government cannot use backdoors: https://www.homeaffairs.gov.au/about-us/our-portfolios/natio...
https://about.gitlab.com/devops-tools/bitbucket-vs-gitlab/ ( marketing page, but mostly valid points )
There are a set of high frequency (very frequently run queries) that are quite sensible to plan flipping (they normally run with a given execution plan; if the plan changes to a worse one, effects can be dramatic, given their frequency).
That leads to a lot of query timeouts (GitLab's database limit query execution time to prevent further damage), which are visualized as database errors. This in turn leads to partial or complete downtime (effectively, if the database is running few effective --non timed out-- traffic).
Some of these queries depend on a particular Postgres planner way of working that is less than ideal (in PG11, it is improved in PG12), and may lead to plan flipping. Which in turn, happens when table statistics degrade. In GitLab's case, they are normally close to the threshold, and slight statistics degradation caused a plan change, which in turn lead to many queries time out and excessive load on the fleet.
At the end of the day, the fix is quite simple, however: update the table statistics (running ANALYZE). For more information: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...
Would this be a place where a plan hint would help? It seems like having to re-analyze would mean that this is just a matter of time before it becomes problematic again.
Thanks :)
> Would this be a place where a plan hint would help?
It is one option to consider (I wrote about it last week: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3...), but it is not an ideal solution, for several reasons.
Re-analyze is not the solution either, but as a short-term measure, cron-ed ANALYZEs will be run. Longer term, apart from refactoring some queries which are quite prone to trigger this plan behavior, statistics gathering process is going to be fully reviewed.