I'm seeing issues with repo pages entirely (getting Unicorn "We couldn't respond to your request in time" errors) but the status page hasn't updated to show that yet.
In a universe, where Obi-Wan Kenobi is a developer: "I felt a great disturbance in the workflow, as if millions of requests suddenly cried out in terror and were suddenly silenced. I fear something terrible has happened."
Seconded. Hosting FOSS on a platform that isn't FOSS itself and run by a for-profit company who tried to work against FOSS for so long (Microsoft) doesn't make much sense to me.
Instead, we should dogfood the FOSS ecosystem on a platform that is FOSS itself, and run as a non-profit. Codeberg, for better or worse, is the best platform for this today.
Personal projects are on a vps and accessed through ssh. I’ve self-hosted gitea, then forgejo, then found out that I don’t like the interface or the auth dance. My plan is to ise cgit if I want something to be public and any forge if the purpose is collaboration (sourcehut is nice)
"are you sure your self-hosted solution will have better uptime?"
Hilariously, after 15 years working in self hosted bitbucket systems, YES entirely.
An underfunded university with an incompetent but doing their best IT department? Zero downtime.
A mid sized company full of overly confident and "just build it" nerds building fragile shit? Zero downtime.
A large corporation with a completely outsourced IT department that can't give you access to something unless you do exactly the right undocumented thing in our internal ticketing software? Zero downtime.
That includes self hosted jenkins and literally homebuilt infrastructure with zero documentation and the guy that built it left a while ago.
I have never been able to blame our build and code infra for lack of productivity.
One thing I started doing a few years ago is that, any time I check out a git repo, whether it's on github or anywhere else, is clone it to my NAS and then clone from there to whatever I'm building it on. Aside from guarding against losing dependencies to outages (or from being yanked), it can be handy for guarding against my own fat fingers, and for making sure that the same version is running on multiple systems.
I honestly can't believe how unreliable Github is. Outages are commonplace. It boggles my mind how nothing has been done to address the reliability regressions that have been creeping in ever since MS took over.
There's a not-entirely-small part of me that hopes Issues breaks entirely for long enough that people at GitHub rethink their implementation.
GH issues have been so useful for the better part of a decade, that I have an empty repo called my_life just so I can make issues about things like home maintenance. In the last few months, the UI has become so slow and flaky that I don't use it for nearly as much as I used to. And when I have to for a project, it slows me down noticeably.
I thought I was alone in this, but asking around I found that this is a common frustration among developers I communicate regularly with.
That's actually a really interesting way to leverage that feature. Have you found this easier than other services built specifically for this use case?
I've tried other tools. But I'm on GitHub most days, so it's been a seamless way to keep track of some things that would otherwise disappear into a calendar, or some tool that I don't use as often.
I do the same - I've been running so many non-software things out of private GitHub Issues. I even have a simple system for creating a new issue every day to use as a personal TODO list, described here: https://til.simonwillison.net/github-actions/daily-planner
I've noticed everything feeling a whole lot less snappy and responsive over the past few days - ironically I think it's because they've done a major rewrite of the frontend presumably with the aim of making it more snappy and responsive!
You're not alone. There's been an effort to transparently update the UI to a React implementation over the past year or two, and while I understand the benefits to that approach, they have introduced some flakiness in moving away from a the server-rendered pjax/html-pipeline/simple web components approach that was so cohesive and battle tested over the decade before it.
I’m low-key certain it’s because they started using more React on the front end, especially for the new Issue stuff. The entire UI feels slow and janky like GitLab.
Or, GitHub used to favor stability back in the day, and there wasn't a lot of changes. People were complaining that GitHub didn't "improve" enough day-to-day, so after the Microsoft purchase, Microsoft started forcing GitHub to add more features, stability be damned.
Most outages are caused not by stuff just randomly breaking, but updates/upgrades going wrong. If you try to increase the output of new features/changes to a platform, you're bound to have more outages and downtime if you aren't more careful than before.
Microsoft, who never really excelled at engineering, to the surprise of absolutely everyone, choose adding features over stability and since years back, we're seeing the consequences of that choice.
I'm not totally certain you can attribute it to this. I know there's a bunch of work going on to migrate things into Azure after the MS acquisition but it feels like more of an industry trend that we cut engineering spend at the cost (often) of lower quality output and outages like these.
GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.
Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.
> My experience of using their product has been broadly unchanged for years.
Compared to how fast/slow they were moving 2012-2018, they're moving at blazing speed now. It feels like every time I open GitHub now, there are new features/changes and "Beta" available stuff. The platform is almost completely different today than it was in 2018, for better or worse.
I wouldn't say this applies to cloud providers, they have a very different business on their hands.
But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.
I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.
I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.
This continues to be a regular thing for the past 5 years of outages and I've lost count of the many outages that GitHub continues to have since calling it. [0]
Besides projects/companies/people stuck on Bitbucket, is anyone actually voluntary using Bitbucket? I remember using it back when I was poor and it was the only choice for hosted free private repositories, but the rest of the platform was actively worse in basically every way compared to the alternatives.
71 comments
[ 4.4 ms ] story [ 145 ms ] threadI don't know about others but for me github is hardly a service that I'm sweating moment to moment uptime. My apps are still running and so on.
[1] https://codeberg.org/
Instead, we should dogfood the FOSS ecosystem on a platform that is FOSS itself, and run as a non-profit. Codeberg, for better or worse, is the best platform for this today.
Then, the outages accelerated from there.
[0] https://news.ycombinator.com/item?id=22868406
Hilariously, after 15 years working in self hosted bitbucket systems, YES entirely.
An underfunded university with an incompetent but doing their best IT department? Zero downtime.
A mid sized company full of overly confident and "just build it" nerds building fragile shit? Zero downtime.
A large corporation with a completely outsourced IT department that can't give you access to something unless you do exactly the right undocumented thing in our internal ticketing software? Zero downtime.
That includes self hosted jenkins and literally homebuilt infrastructure with zero documentation and the guy that built it left a while ago.
I have never been able to blame our build and code infra for lack of productivity.
As usual that will be fixed within the 10 to 600 mins.
At one point all git operations were down: https://news.ycombinator.com/item?id=42691184
GH issues have been so useful for the better part of a decade, that I have an empty repo called my_life just so I can make issues about things like home maintenance. In the last few months, the UI has become so slow and flaky that I don't use it for nearly as much as I used to. And when I have to for a project, it slows me down noticeably.
I thought I was alone in this, but asking around I found that this is a common frustration among developers I communicate regularly with.
So perhaps it's the same problem, that's been getting more prevalent until it was noticeable by monitoring. One can hope!
I've noticed everything feeling a whole lot less snappy and responsive over the past few days - ironically I think it's because they've done a major rewrite of the frontend presumably with the aim of making it more snappy and responsive!
I know you can sync repos, I don't think you can sync issues tho
It's no surprise we're now feeling those effects but damn, GitHub and other services like Slack have been really bad lately.
Most outages are caused not by stuff just randomly breaking, but updates/upgrades going wrong. If you try to increase the output of new features/changes to a platform, you're bound to have more outages and downtime if you aren't more careful than before.
Microsoft, who never really excelled at engineering, to the surprise of absolutely everyone, choose adding features over stability and since years back, we're seeing the consequences of that choice.
GitHub laid off 10% of their staff in 2023 and like you say, won't have slowed down to account for that.
Ignoring all that though... other than Copilot, what big feature changes have you seen in GitHub? My experience of using their product has been broadly unchanged for years.
Compared to how fast/slow they were moving 2012-2018, they're moving at blazing speed now. It feels like every time I open GitHub now, there are new features/changes and "Beta" available stuff. The platform is almost completely different today than it was in 2018, for better or worse.
Is this true beyond github and slack? Both were acquired btw, that could explain their availability issues (just like twitter, now X).
What about e.g. AWS or GCP? Has their availability meaningfully reduced?
But for SaaS in general I think the trend is noticeable? Twitter led the way with massive layoffs in engineering often in the roles around reliability. The industry as a whole have aimed to cut costs however possible, and reliability/ops is usually seen as a cost-centre that gets hit hard.
I've watched this in my own space (start/scale-ups and larger companies, I work in incident response tooling) as people start talking very differently about reliability and engineering investment. You hear "do more with less" about five times every day and spend that was previously greenlit by default around reliability/redundancy is under much more scrutiny now.
I see this as a silent mirror of the reduction in open-source efforts from companies now there's been a refocus on business impact and bottom-line.
[0] https://news.ycombinator.com/item?id=22868406