Well, when we say GitHub, we are talking about thousands of services. I don’t think we can pinpoint the overall operational failures to programming language choice. In my experience, these kind of issues are mostly caused by poor organizational architecture + poor project management. In GitHub’s case, I’m also confident to say the choice of infrastructure plays a much bigger role than Ruby.
Presumably they intended to entertain. There are absolutely companies running much more complex services with more users than Github but better uptime, so it's absolutely fair to call Github out on their failures. Github has at least an order of magnitude fewer users than something like Google or Facebook.
If your attitude is reflective of the attitude of anyone at Github, calling people entitled who dare complain about the site's downtime rather than taking responsibility and working to better it, it's no surprise that Github has so much downtime.
You’re being really mean. Do you work for GitHub? Or do you just have a habit of taking things too personally? “i bet you can’t do better!” is not how an adult responds to someone expressing their displeasure at the fact that the performance of a service that they / their org is almost certainly paying for is below their needs / expectations.
Damn right I couldn’t do better. I’m also not the massive highly paid and talented workforce of GitHub. Next time you whinge about your mechanic not fixing your car or your plumber not fixing your toilet, I’ll be right there in your ear calling you a spoilt brat for not just doing it your damn self.
100% we rely on others for goods and services, regardless of our ability to perform said service we are able to (and should be able to) judge it. Otherwise everyone would need to be an expert in everything.
In other words, I don't need to be a banana farmer to say a banana is rotten.
> You make it sound like you are better at running a system at this scale with so many users.
I don't care that the system scales to millions of people, my team is a dozen people, that's all I need something to do. And my downtime stats are far less that githubs, and they have never occurred during the working day.
Nobody denies scaling is hard, they simply deny scaling is necessary.
I took it totally differently (it was humerous, with a hint of irony I'd expect a Brit[0] to appreciate): that due to Github's lack of resilience, the poster has had to become resilient in their own daily practice.
I always think this should be an incident in itself - why did our status page not reflect the reality of a degraded service? It's so common that they don't, and something user-driven like DownDetector is often more reliable
I don't think an accurate and automated public status page is something any management would want. If it was accurate they wouldn't be able to lie to customers about the uptime. So I always suspect status pages are adjusted manually.
That's exactly what happens. How we need to respond though is by not linking to status pages hosted by that party, instead we should be linking to a StatusGator or DownDetector page as a 'source of truth'.
I recently set up a status page for the services I run on my pi. The idea was to get some insights and apply experience at work.
My experience now tells me what we really need first is a solid alerting system, the status page can't be trusted. It's a PR tool (a useful one), not a sysadmin tool.
What about something like https://heiioncall.com/status (disclosure: helped build it) which gives a real-time view into what our monitoring & alerting system sees, both from various HTTP endpoint checks, and cronjob checkins?
But I don’t think most orgs would want a public version of this (too much transparency), which is why we haven’t built that.
True, and SRE teams get smaller and smaller and/or tasked to do other non-related things. So when users start leaving for some competitor, the same SRE team suddenly gets more heat, pressure, more belt tightening, more meetings "for transparency and coordination", and directly and indirectly more blame. Ask me how I know... I bet its a story as old as time though.
There is little incentive for them to invest in stability. Customers will eat the pain because they are so invested in the platform. They will never move away.
Meanwhile all the Staff engineers trying to get promotions are building AI shit.
Since last few months, GitHub has become very unreliable. For example, last month, they reported six official incidents in just one week. I've also noticed many shorter, non-official incidents (i.e. the ones where there is no active incident on their status page, but many API calls return a 500 error). I learned about this the hard way (we offer affordable GitHub Runners at Ubicloud, which gives us the opportunity to observe all these failures firsthand)
I’d say years though it might depend on both your physical location and te services you use.
We’ve had to harden a bunch of automation to be more reliable in case of github going tits up, and while github is still currently the broker (as it’s where users get statuses / information from it made sense at the time to also have it handle broadcasts) we’re definitely planning on direct communication between services eventually.
Thank god (Linus) that git is a distributed "protocol" letting you pull and push to any location on your file system or anywhere you can access over ssh.
Git itself is only a piece of the puzzle, you still need synchronisation points for collaboration e.g. bug reports, code reviews, discussions, CI, synchronisation, etc…
Those are the services forges provide.
Also git’s support for point to point distribution is bad. I miss `hg serve`.
This comes up pretty much everytime Github is down. Github is not a replacement for git. Github is an issue tracker and CI platform just as much as it is a git hosting site, none of which git can provide by itself. It's complementary. The comment really doesn't make any sense.
It's like being snarky about the distributiveness of git when the Linux kernel mailing list goes down.
I saw a post on LinkedIn, from GitHub, boasting about how many changes they deploy in a day. I couldn’t believe it. GitHub actions have been broken so many times for me this past month, not able to deploy my code when I want to, so it felt like a slap in the face.
Maybe what is happening in this thread should give everyone cause for
concern and a clear heads-up that more serious problems are afoot.
Very emotional reactions here, whether fuming at the unreliability of
the service, or zealously defending and identifying with it, indicate
people have far too much invested in this single point of service.
People are saying - "That's me done for the day, have a good weekend!"
What is the economic cost of this? On a global scale? The pattern is
the same as we've seen recently with Office365 and other cloud
services.
People talk about setting up elaborate monitoring services, yet with a
similar effort one could set up one's own development repositories and
tools.
It's not like this crowd is typically "helpless", or unaware of what
is being done, yet people remain beholden; Github has diversified into
a social network and AI tool, far beyond its initial remit as a code
repository. People seem genuinely terrified at the prospect of it
going. And rightfully so.
In the wake of the xz backdoor attempt Microsoft will soon move to
offer "developer protection" and a system of validation, testament and
supply-chain protections. At that point Microsoft will have all of you
absolutely in the palm of its hand and at its mercy.
Now would be a good time to seriously invest in long term thinking and
strategic protection of your development future, indeed of your
fundamental ability and rights to develop software free from
disruption, pressure and encumbrance.
80 comments
[ 2.8 ms ] story [ 132 ms ] thread[0]: https://github.com/rails/rails/commit/b83965785db1eec019edf1...
Oh yeah. If someone says C++ has poor memory safety than Rust then no one is triggered. But say something about Ruby (the language on a secular decline - https://www.dice.com/binaries/content/gallery/dice/insights/...) and the pitchforks are out.
A better question would be "is there anyone who doesn't think Azure played a role here?"
If your attitude is reflective of the attitude of anyone at Github, calling people entitled who dare complain about the site's downtime rather than taking responsibility and working to better it, it's no surprise that Github has so much downtime.
Damn right I couldn’t do better. I’m also not the massive highly paid and talented workforce of GitHub. Next time you whinge about your mechanic not fixing your car or your plumber not fixing your toilet, I’ll be right there in your ear calling you a spoilt brat for not just doing it your damn self.
In other words, I don't need to be a banana farmer to say a banana is rotten.
Or a paying client of GitHub's, venting their frustration that a business critical service is frequently encountering service outages.
I don't care that the system scales to millions of people, my team is a dozen people, that's all I need something to do. And my downtime stats are far less that githubs, and they have never occurred during the working day.
Nobody denies scaling is hard, they simply deny scaling is necessary.
git hosting is easy to scale (a.k.a. github classic). Dozens of technologies integrate into a frankenmonster (a.k.a. github modern) is hard to scale.
I like github projects, github co-pilot, CI/CD, etc. but I like stable github better.
[0] > 'bloody annoying'
Apologies if this was misunderstood, I'd absolutely not do better than Github teams.
But it definitely still happens now (500s on refresh on PRs and GitHub actions)
Edit: still ongoing
Edit 2: still ongoing at Apr 05, 2024 - 08:56 UTC (keeping updated for the record since their status page cannot be trusted apparently)
Edit 3: I see they have switched to a different (ongoing) incident ID now: https://www.githubstatus.com/incidents/5ly0psff2s5d
There was a conflict with marketing, market movement, sla contracts, and our image.
My experience now tells me what we really need first is a solid alerting system, the status page can't be trusted. It's a PR tool (a useful one), not a sysadmin tool.
Were you expecting me to say something else?
Well, the status page is not the main feature of github/gitlab but I agree that if customers decide to rely on it then it's a problem.
> Were you expecting me to say something else?
Yeah, something like "their status page is not their core business" or "status pages and SLA are two distinct things".
But I don’t think most orgs would want a public version of this (too much transparency), which is why we haven’t built that.
Meanwhile all the Staff engineers trying to get promotions are building AI shit.
We’ve had to harden a bunch of automation to be more reliable in case of github going tits up, and while github is still currently the broker (as it’s where users get statuses / information from it made sense at the time to also have it handle broadcasts) we’re definitely planning on direct communication between services eventually.
Those are the services forges provide.
Also git’s support for point to point distribution is bad. I miss `hg serve`.
It's like being snarky about the distributiveness of git when the Linux kernel mailing list goes down.
Once: an upgrade went wrong and I had to roll-back, that made me spend 30 minutes instead of 10.
Gitlab instance stats:
Projects 256
Users 216
Groups 17
Disk usage: 877 GiB / 3.38 TiB
Very emotional reactions here, whether fuming at the unreliability of the service, or zealously defending and identifying with it, indicate people have far too much invested in this single point of service.
People are saying - "That's me done for the day, have a good weekend!" What is the economic cost of this? On a global scale? The pattern is the same as we've seen recently with Office365 and other cloud services.
People talk about setting up elaborate monitoring services, yet with a similar effort one could set up one's own development repositories and tools.
It's not like this crowd is typically "helpless", or unaware of what is being done, yet people remain beholden; Github has diversified into a social network and AI tool, far beyond its initial remit as a code repository. People seem genuinely terrified at the prospect of it going. And rightfully so.
In the wake of the xz backdoor attempt Microsoft will soon move to offer "developer protection" and a system of validation, testament and supply-chain protections. At that point Microsoft will have all of you absolutely in the palm of its hand and at its mercy.
Now would be a good time to seriously invest in long term thinking and strategic protection of your development future, indeed of your fundamental ability and rights to develop software free from disruption, pressure and encumbrance.