Thanks for sharing! :) FYI, the Rails monolith architecture is currently discussed in this proposal issue [0] for a modular monolith approach. This is following the blueprint work in [1][2][3].
Just dropping by to say that you guys have really good docs! Some of the other commenters have expressed that they can get a bit overwhelming which I guess is true, but after years of using GitLab I haven't had an instance where the docs couldn't help me figure out how to do something!
(well, there was a case or two in regards to updates in a non-standard config years ago, but community discussions helped there)
Half of me wishes I ever worked at a place that had invested this much in documentation. The other half realizes I'd probably hate it if everything was already this laid out.
How much should they expect to get back by somehow impressing upon potential customers that they have a sane-ish tech stack (and maybe how much will that help to recruit?)
vs how much $ cost in salary they save because new employees can look things up in the documentation instead of asking colleagues or figuring it out themselves.
I think part of this also contributes to their brand (open source first/foremost) and to potential contributors so there's more to the equation than just impressing potential customers
Its part of the remote-only and open source oriented culture. Haven really good written communication - which documentation is, really helps when you are spread all over the globe.
As much as I appreciate all the components that Gitlab utilizes behind the scenes running an instance of this on a 4GB droplet consumes the entire memory of that unit until things come to a crawl.
Does Gitlab actually need to utilize all those components and when I disable certain things like Prometheus and etc. it seems to only marginally make things better.
I self-hosted a gitlab instance for a while on a local VM with 4GB of RAM and it (being the VM) would crash randomly because the OOM-killer killed too many important processes.
I no longer self-host or use GitLab for this precise reason.
I’m running a high traffic rails app with 15 years of source history; our instances have no trouble at 4gb for both unicorn (8 processes) and sidekiq (5 processes).
Except for image processing, which uses a ton of ram and readily OOMs.
I've worked at two very large rails shops (think >10 billion dollar companies) both of which have had stupidly high ram requirements for their code without any image processing. Gitlab which also happens to be another billion dollar rails shop, also requires a stupid amount of ram as per the parent comment. I wonder what the common thread is...
It definitely correlates. Ruby is not the culprit though. It uses and abuses memory as fairly as other similarly GC backed dynamic languages. The issue may be rails and the monolithic pattern, which favours rapid feature development and postpones optimization for when necessary, or one can no longer realise that snippet you wrote 2y ago fetching data from the db scales well for 5 objects, but not 500 objects, but you forgot about that already.
Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.
It's apparent that a bigger company has more money to throw at the performance problem if that can solve it. Smaller companies want better optimization of their spending with better coding standards.
It has been a few years since, but I wrote a standalone small Rails app for an internal service for a company that also runs a giant Rails app for their main business.
I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now.
That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.
This is something I've noticed a lot of open core projects suffer from. They make architecture decisions that make sense for their huge, multi-tenant production system but correspondingly make self-hosting on a single node feel like administering a rube goldberg machine.
My opinion is that they optimize for huge deployments. Tiny companies are supposed to use cloud. And self-hosting is supposed to be used by big companies with thousands of people. I guess that's where their money is, anyway.
That said, they're doing good job of documenting things required for smaller installations.
So true. I was in charge of setting up self hosted GitLab at tiny computer company. Was very proud of the cicd setup untill the boss asked for review apps to be launched for each commit. Gitlab had no solution for this but to use Kubernetes.
Bullshit, the inability to properly optimize doesn't mean it's optimized well for huge deployments. It will need much more resource on huge deployments and still quite some RAM on small deployments too.
At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server.
It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.
This is so true. Sentry's architecture[1] is insanely complex. It runs some ~30 containers on the instance and all you've to pray is that their tooling[2] for `docker-compose` _just works_.
Armin from Sentry here: It's definitely something we're battling with. At the scale we're operating, providing all features on an alternative architecture that does not involve as many independent pieces is pretty tricky. I'm somewhat hopeful that we see reduction in the total types of moving pieces over time at least.
At least for our small use case, the web interface is snappy enough that it's just about spending more money to host it but I'd definitely like it to reduce its memory usage as I'll have to double the server size just for Sentry.
Hey Armin. That's very nice to hear that reduction is something on your roadmap. Pre Snuba, the number of external services were fairly limited but I understand as Sentry got more mature and added a bunch of extra services than "just error/crash reports" is when the architecture also sorta started to become complex.
It's not even just that there are extra services, it's that the volume of traffic is just too high for rabbitmq. So we put a bunch of kafka in and with that, the complexity quickly went up as you end up writing more and more consumers. Most of the services still mostly "just do crash and error reports" but they are doing it in more scalable ways now.
Snuba is one of the services that largely really requires kafka to function though I think theoretically limited functionality is also available by directly talking HTTP to it.
They are easy to deploy and upgrade (the upgrade procedure is for me the most important because I want to do it easily or at the end you do not do it).
I remember back in the day we were evaluating Puppet vs CFengine. During our testing we noted how memory intensive Puppet was, which we assumed at the time was due to it being based on Ruby. Afaik Ruby can quite easily consume a bunch of memory, it doesn't surprise me that an app like GitLab, which is based on Ruby on Rails I think, consumes somewhat largesse amounts.
My experience with GitLab is that they accelerated very quickly in the beginning with a focus on new features. This made the whole ecosystem incredibly slow, and it's unbearable for me to run as a service on my own hardware / cloud accounts. I guess that's the problem with these open source services that also rely on paid platform income. I have tested out https://gogs.io/ and the difference in speed is just incredible.
My experience with Gogs is now a few years out of date, but after using it for a few months we ended up switching to GitLab. Gogs was amazingly fast and required few resources, but with moderately-sized repositories or even slightly large pull requests it slowed to a crawl. At some point it screwed up some on-disk repository state causing pull request merges to trigger server errors, which required manual cleanup. Meanwhile GitLab (omnibus), with a few users on a VPS with 4GB memory, remains consistently fast and entirely problem-free.
44 comments
[ 3.3 ms ] story [ 116 ms ] threadThanks for sharing! :) FYI, the Rails monolith architecture is currently discussed in this proposal issue [0] for a modular monolith approach. This is following the blueprint work in [1][2][3].
[0] https://gitlab.com/gitlab-org/gitlab/-/issues/365293
[1] https://docs.gitlab.com/ee/architecture/blueprints/composabl...
[2] https://gitlab.com/groups/gitlab-org/-/epics/5278
[3] https://gitlab.com/gitlab-org/gitlab/-/issues/31121
Nanoc (https://nanoc.app/) is the static site generator we use for our docs.
More info: https://docs.gitlab.com/ee/development/documentation/site_ar...
(well, there was a case or two in regards to updates in a non-standard config years ago, but community discussions helped there)
What was the $ cost in salary to put this out vs
How much should they expect to get back by somehow impressing upon potential customers that they have a sane-ish tech stack (and maybe how much will that help to recruit?)
Does Gitlab actually need to utilize all those components and when I disable certain things like Prometheus and etc. it seems to only marginally make things better.
What is consumes so much RAM and why?
I no longer self-host or use GitLab for this precise reason.
That's just a complex rails application for you, at $DAYJOB we have to provision 11gb for just a single container running a single instance of the app
Except for image processing, which uses a ton of ram and readily OOMs.
"Why does the app keep crashing?"
"Because the share price went up again!"
Ruby has a generational GC which already supports compaction, but not in a production ready continuous way, which means most usages have it turned off, and memory pages tend to overcoming and stay fragmented for long running processes. Python has the same issue and node AFAIK.
I ended up heavily tuning the ruby GC parameters, which reduced the final resource usage of the app especially over longer continued use nearly tenfold. There is a lot to gain if you know the actual object and memory limits your app needs for it's core usage rather than letting Ruby allocate more and more memory. Caveat is that was a long time ago with Ruby ~2.3 and things have certainly changed and probably improved by now.
That service has been as stable as one could hope. It just runs in a docker container on a dedicated machine that's also used for other development purposes and consumes a couple of hundred MB of ram. The number of times it was down in the past ~6 years can be counted on one hand.
It's the developer who just doesn't know how things work under the hood and code as they like and memory usage explodes.
[1]: https://docs.gitlab.com/omnibus/settings/memory_constrained_...
That said, they're doing good job of documenting things required for smaller installations.
At this point, the code base is too big for them to fix the performance and they want to use their time on new features and bug fixes as that's what customers are asking more while customers may solve the performance problem by spending more money on the server.
It's just that the initial technical decisions sucked and it's no longer possible to fix it at this point.
[1]: https://develop.sentry.dev/architecture/ [2]: https://github.com/getsentry/self-hosted
Snuba is one of the services that largely really requires kafka to function though I think theoretically limited functionality is also available by directly talking HTTP to it.
[0] provides documentation for the search based on Algolia DocSearch.
[0] https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/do...
In the beginning I am always looking for features from other platforms only to find them readily implemented.
It does not matter if it's Mattermost, a container registry, or direct oidc logins for kubernetes and friends, it's just there.
The magnitudes of features, make it an easy choice as a core of your startup.
All in all thank you for creating an amazing piece of software. Like a fridge, for me, it just works.