I sure hope not, GitHub is supposed to be matured infrastructure at this point, where most if not all changes going into production should be very well tested and nothing that multiple people haven't verified as being correct should end up being deployed and released.
Besides, Microsoft surely has 24/7 watch of their infrastructure, even on weekends, it's a huge company.
Why does everyone need to be in the room? I have a groomed backlog and can talk to people async as needed. We also record meetings if you missed them and depending on the context of the meeting or importance, we'll hold timezone friendly meetings for everyone as required.
What's your pager rotation like? I want to say you have follows-the-sun, and so your on-call shifts are 12-hours long and you swap with a team on the other side of the world from you so you can get said sleep, but I don't want to just assume that.
Less "on" hours then? Even Google has diurnal patterns when there's a lower amount of traffic simply due to the fact that humans are unevenly distributed across the Earth's surface. And Google does code freezes for the holidays where they don't deploy at all.
> Besides, Microsoft surely has 24/7 watch of their infrastructure, even on weekends
"watching" with a dedicated team vs "waking up everyone in engg because things are on fire" are two very different things.
Besides, size doesn't work that way. The larger the organization and the more complex the product is, the higher the chance some unexpected interaction will occur. There are processes and automation that can mitigate this, but one can never be completely certain.
A pull request is a process which can merge new code into existing code.
"Software Engineer John was tasked to add a new logo to the website, when he was done he submitted a pull request of his feature branch into his organization's github repository for the website so that his team members could approve the changes before automation (like Github Actions) deployed live as a new version of the website."
It's a request to the owner of some reference in a Git repository to pull in some changes from some reference in some (possibly other) Git repository. You can do this via email, but centralised Git hosts like Github have their own interface to this basic workflow.
I agree the naming is misleading - it's not actually a request to pull anything - it's a request to merge someone's branch into another. This is known as a merge request on several other platforms.
Isn't that the same "bug" that happened to github a few weeks ago when they updated their Git version too? It wasn't a bug per say but they still had to revert because the new hashes were causing massive build problems. Maybe it's a different root cause though.
My on prem git lab has been fine with it’s pipelines, and given it’s impossible to have an on prem uptime higher than the cloud what you say can’t be true.
Once again, just two days ago [0], the whole of GitHub went down, after the RSA key leakage and the certificate key expiry on its user facing site.
It is also apparent that GitHub Actions has chronically been struggling to operate normally for at least once a month for years.
There is no question that GitHub has been more unreliable than if you were to use a self-hosted GitLab or Gittea instance yourself as I said before [1].
At what point are organisations going to ask themselves wether this is intentional or not. Your velocity is disrupted by a likely competitor. I'd move out.
I don’t have an opinion on it is or not but layoffs do have a significant impact on that and it’s generally something that’s pretty impossible to study.
how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages.
> how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages.
I mean Github Actions was released 5 years ago[0]. I imagine the infrastructure for actions is more susceptible to outages than the fairly simple features Github offered previously. It makes sense that the number of outages would increase with the additional complexity in the infrastructure.
Nobody's stock goes down when actions go down. Nobody's stock goes up when actions are working. But everyone's stock goes up when you have mass layoffs. Working as designed.
not sure what the SLA is on Actions but outages are a regular occurrence with these kinds of systems and are incredibly expensive to move to the next 9 of availability.
it's certainly a risk you'll need to evaluate when planning your desired build process.
what is the "29.3" in the title supposed to represent? is that supposed to indicate a date of March 29th? I do not see a reference to this on the incident page itself.
55 comments
[ 3.3 ms ] story [ 62.5 ms ] threadBesides, Microsoft surely has 24/7 watch of their infrastructure, even on weekends, it's a huge company.
Just because they can have people come in on weekends to fix things doesn’t mean they like doing that.
I know many “mature” software platforms that do not deploy on Fridays or off hours at all
We have a different work philosophy. I do work with teams in India and England, and it’s painful to accomplish anything cross team
Some teams do follow the sun type rotations, but my team is all in Seattle.
"watching" with a dedicated team vs "waking up everyone in engg because things are on fire" are two very different things.
Besides, size doesn't work that way. The larger the organization and the more complex the product is, the higher the chance some unexpected interaction will occur. There are processes and automation that can mitigate this, but one can never be completely certain.
Not even the aviation industry has mastered that.
https://status.gitlab.com/
"Software Engineer John was tasked to add a new logo to the website, when he was done he submitted a pull request of his feature branch into his organization's github repository for the website so that his team members could approve the changes before automation (like Github Actions) deployed live as a new version of the website."
1: https://gitlab.com/gitlab-org/gitlab/-/issues/402616 2: https://github.com/microsoft/vcpkg/issues/30481
It is also apparent that GitHub Actions has chronically been struggling to operate normally for at least once a month for years.
There is no question that GitHub has been more unreliable than if you were to use a self-hosted GitLab or Gittea instance yourself as I said before [1].
[0] https://news.ycombinator.com/item?id=35325850
[1] https://news.ycombinator.com/item?id=22867803
https://blog.gitea.io/2023/03/gitea-1.19.0-is-released/#-git...
[touches nose]
https://www.githubstatus.com/history
github has had 55 outages in 3 months. thats nearly an outage every two days.
the last six months of 2022 had 74 outages. In many shops thats tangibly worse than what their local greybeard Linux admin maintains.
arguments against spinning up my own gitlab/gitea/jenkins/whatever in podman under systemd are starting to ring pretty hollow lately.
I mean Github Actions was released 5 years ago[0]. I imagine the infrastructure for actions is more susceptible to outages than the fairly simple features Github offered previously. It makes sense that the number of outages would increase with the additional complexity in the infrastructure.
[0]: https://resources.github.com/devops/tools/automation/actions...
it's certainly a risk you'll need to evaluate when planning your desired build process.
I'm just jumping up and down how awful this representation is. iso8601 (YYYY-MM-DD) is a settled debate at this point.