568 comments

[ 0.47 ms ] story [ 75.1 ms ] thread
"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."

Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.

Github is ripe for disruption and I hope it is disrupted soon.

If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.

If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.

Speaking from experience, it cost mW about a week or two per year to maintain GitLab for the startup I worked at.

My personal GitLab on the other hand really does take only a day or two per year.

That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.

GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.

> We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.

And from the RCA [1]:

> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.

[1]: https://www.githubstatus.com/incidents/zkxwbgr0cnmx

> Github is ripe for disruption and I hope it is disrupted soon.

It's an expensive, low revenue generating site.

There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.

How is it "ripe" for disruption?

I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile paying the $4/mo sub.
What if it was $2?
that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.

for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail

EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option

Oracle has a free tier that you can self host a git server on, if that's what you're looking for.
I spent a month trying to set this thing up, to no avail
What was the sticking point?
Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)
LinkedIn gave up after four years of trying: https://www.cnbc.com/amp/2023/12/14/linkedin-shelved-plan-to...

Azure just has very poor performance and reliability characteristics. It’s a particularly bad migration target for a colo-based company that mainly runs on owned hardware (such as GitHub or LinkedIn). Requires much larger changes than (say) a company coming from AWS.

  "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "
Wow, that is some incredible growth in a really short time.
It really is. I know I've gone from tens a month to thousands a month. They have to be projecting >100B/month in the next year or two.
wow. they should really institute a maximum amount of individual pushes per-month per-user.
Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.
We're small enough that we've been hosting our git infra for about a year now, I wonder how many other companies figured out they could make the trade. I've had a Github since a couple years after they started and I think they are going to become a Stack Overflow, albeit slower with MS at the helm. If Github is going to be 99% slop it's going to be really hard to use as a fun tool to show what you can do, what you've worked on, side projects, etc. I took github off my resume and I'm probably not going to relaunch my weblog if I end up job hunting, too much low-effort crap and people basically copying what a lot of us had been doing manually for years to really feel like it's other than a negative signal.
Ironically a bunch of people already migrated to Codeberg and then Codeberg announced "heads up, we actually don't want your AI slop, we're for real projects only" and AI coders threw a shitfit on HN.
That will just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk.

Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue.

As someone on the sidelines, this is really interesting to watch unfold.

Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then:

If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.

If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.

Why would any company want coding data now? It's all garbage. I'd be surprised if anything past 2025 is even used for training.
Me personally, I would like to own the world's code.

The richest man in the world is interested in this dataset as well.

There's nothing special about code here - if you can own the world's $X for any X, you win capitalism.
Let me add one more reply here.

Even if I accepted that given the new repos: all the new code = "It's all garbage." - I would certainly love to be in a position to observe the new code + metadata = software trends, as software eats the remaining world.

I don't think anyone would love to own the world's AI slop. The capex and revenue expenditure would be too high for any ROI.
If you use GitHub as just a software forge then sure you can find an alternative. But I suspect more people use GitHub for its social aspects and they will stick around despite the regressions because building an alternative to an established social network is incredibly difficult if not impossible.

You can get work done on any software forge. But potential employers will still ask for your GitHub. People will judge your personal project by its GitHub stars and be less reluctant to download a binary from GitHub than elsewhere. Potential contributors will leave a PR on GitHub but probably not if they have to make an account on a new platform and learn how it works.

And of course, let's not assume any competitor can just absorb even a fraction of the traffic GitHub receives without suffering similar reliability issues.

Why? I'm curious of your reasoning here. Do you think that capping commits would decrease the downtime problems? They have virtually infinity resources to scale the tech so I can't see that making any improvement, people would just leave.
I still remember when they decided to limit the number of private repos you could have as a free user. Kind of silly to me at the time, and even more so now!
I remember when you had to pay for private repos in the first place.
Initially there were no private repos for free users at all, it was a selling point for bitbucket that they offered that.
I don't think it's silly for free users to be restricted. If you want something, pay for it.
so... when you reach your monthly limit in the middle of the month, what is the recourse?

Or, is the idea just to drive everyone away from your platform?

Uh, pay money for the service you're using constantly?
That's mostly due to AI slop
Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).
Are you using free AI and built in harnesses?

Try spending $20/mo on gpt sol and combine it with OpenClaw.

Agree - this reminds me very much of the old joke of two economists increasing GDP by taking turns giving the same 200usd back and forth for having each other eat shit.

Useful / impressive for whom is the question. Not for us!

We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.

In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.

They do. Talk to your account representative for the alternate domain name to hit.
Should use on premises.
alot of distributed systems in big companies grow at this rate, its not exponential, its mundane
Eh, I would be more empathic in this situation[0].

Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.

When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.

[0] even at current MS owned github

cloud providers and hyper growth tech deal with these growth rates all the time
Did you read the article? It is growing exponential.
this is common at AWS/cloud providers
Not really. It's AI commits. Not quality commits.
the infrastructure does not care about the quality of the commits, just that a commit happened.
A lot of human commits are low quality
Because human commits ar necessarily quality?
no but AI commits are necessarily not
(comment deleted)
Why is Github talking about number of commits here, and not pushes? Are there a lot of tools/people using github as an online editing platform?
Bigger numbers sound more impressive.

"Our billion-dollar infrastructure crumbles under a tremendous flood of 50 PRs per second" would just sound embarrassing.

LOL, this kind of things will happen when projects like Bun (https://github.com/oven-sh/bun) are running on auto. :)
3.3k open issues holy shit
Strange to think they are probably triaged by LLMs at this point
AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.
I mean, isn't that the dream?

I don't know if that's sarcasm or not. I know it doesn't work, but that's the future we've been promised, right?

If the same AIs don't hallucinate and create bugs in the first place.

I don't know how much my own time is wasted on Claude imagining API response formats that never existed.

Let's just say my dream does not involve babysitting some robot
> I mean, isn't that the dream?

Unironically: no.

Why it doesn't work? It is working for me. It is working for bun. It is working for others who actually embraces it and puts in the work to get it working.
sadly someone didn't have free unlimited Fable and Opus like Bun team does
I recognize your username from other comment threads and would classify you as a bun fanatic, but even so, I’d hold off on saying, “it’s working for bun,” until 1.4.0 has been out of canary for, like, more than 24 hours. Most real users haven’t onboarded to it yet.
so first it was rust rewrite bad. Then rewrite with ai can never work. Then it will be riddled with bugs. It'll crash. It'll take years to fix.

On the other hand, rewrite was mostly done in record time. New version added massive number of features. Also huge bug fixes. Being used by Claude code by millions of people. Successfully used by some others even in canary. After release, multiple companies immediately switched due to massive amounts of resource savings and performance gains (and publicly posted about it).

Can there still be problems? Yes, I'm sure there will be. But denying the feat Oven pulled off with Bun in last few months is nothing but phobia/fud.

Many people are already posted about testing new bun version and I have yet to see a single post where the issue is the latest versions of bun. In some cases people posted it doesn't work but that's due to node compatibility etc and it didn't work on previous version either.

One does not need to be bun fanatic to see and call things as they are.

PS: I like bun because I hate how js ecosystem requires 100s of packages to do anything and bun is aiming to include batteries. This is good.

That seems to be what's happening. Like 2-4 bots talking to each other, then Jarred Sumner just does the final merge with little comment.

https://github.com/oven-sh/bun/pull/39743 https://github.com/oven-sh/bun/pull/39735

github-actions: "If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code"

robobun: "The ordering is load-bearing: reclaiming before this block made is_dead_request true and hung a parked textStream read (caught by body.test.ts in CI). The comment pins that constraint."

Ah, well, if something is load-bearing, then I guess that settles it. Need a comment to pin that constraint, in case a read is parked. These are words that normal humans commonly use in these ways.

(Always striking how much Claude obsesses over the minutiae of method contracts and side effects, exhaustively documenting them in comments. It’s much happier figuring out how to reorder some method calls with nonobvious side effects so the code works than it is refactoring them not to do unexpected things!)

(comment deleted)
And 5K PRs. I'm crying.
> 5k PRs

Wow, is Bun the record holder for number of PRs?

I recall GitHub recommends to keep the number of PR to a certain level due things such as GitHub Actions slowing down.

You never could scale tulip buds so quickly - progress! ;-)
And...? It just shows how active and big the project is.
More than 50% of the recent commits are from robobun, jesus
It's AI all the way down.

An issue reported by a person account but post made by AI. https://github.com/oven-sh/bun/issues/39800

AI (robobun) responds and creates PR. https://github.com/oven-sh/bun/pull/37459

AI (coderabbit, claude, github actions) review the PR, AI (robobun) applies the fixes.

Some AI back and forth.

A human finally merges the PR.

Not gonna lie, it's kind of beautiful.

Some guy, profiling his linux distro, wonders why ssh connections are a few hundred milliseconds slower than expected, finds there PR introduced and RCE backdoor and p0wns the whole project.
Seriously, this is driving me insane. The XZ hack told us what we had to do to secure our supply chain, and instead we went ahead and implemented a worldwide standard for NLP-to-action and figured we'd worry about the guardrails later. Mad.
As it should be if not more. What's wrong with it?
Here's the most recent nontrivial one as of this comment: https://github.com/oven-sh/bun/commit/d4de65e9a43224a14591ad...

The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't you think there would be a better way to ensure an object gets collected, like setting the variable to null?

The comments in the code don't make a lot of sense either. Something so obscure and brittle has to be explained extremely clearly.

While the issue might be real, this commit is so far away from the locus of normal that it's sending red alert. Plus a hallucination is very likely with such a long investigation - once an LLM agent starts investigating it just assumes there is a problem. And this is the 1 out of 1 robobun commit that I looked at.

Edit: here's the next one: https://github.com/oven-sh/bun/commit/72ec6e2594892455df0090...

Make sure the fs module keeps working if someone freezes or seals its exports table. I was wondering who was going around freezing random tables from other modules, so I checked the linked issue - robobun reported the issue, too. Why? I'm skeptical of whatever robobun was doing when it decided that it was necessary for code outside of a module to freeze their export tables. It needs a very good justification.

The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nested scopes or wrappers in Java to deal with this in tests.

Don't know anything about the second one.

If this is needed in tests it needs to be known how it works, it needs to work consistently, and it needs to be documented how it works. It can't be an ad-hoc deep investigation and random fix each time. The comment should then be just // ensure foo is no longer a GC root, see gc_roots.md
It needs a comment for sure. I always write a little helper utility to manage that when I need it.
This will result in pretty low quality software.

I've had opus 5 along with its AI code reviewer agree to do some pretty stupid shit.

I'm sure Bun is more higher quality than anything you've put out in your WHOLE life.
Opus 5 is a trash model
Opus 5 is only usable when Sol is reviewing its output.
Wait till you see what some people agree to.
Ah the good old "AI isn't bad because people are bad too" argument
Wow...

I am currently using bun, but may have to switch. I can't see how this can possibly turn out well in the long run...

I mean if I set up a for loop spamming my own SAAS it would be incredible growth too
Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that
If you grow 2x a month, you'll run out of resources very soon. Like the whole Solar System.
Hey, um. Little bit of math. Exponential growth.
Truly incredible in every sense of the word.
For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months.

https://github.blog/news-insights/product-news/github-copilo...

Makes sense given the ubiquity of agentic coding. I made a joke to my coworker today that all we do is make sure AI agents can communicate with other AI agents.
It’s funny how the quality of the average commit message has gone up, now that we rarely read or write them directly anymore.
Has it? Most of the LLM-generated commit messages and PR descriptions I read are needlessly verbose and miss the point: elucidating the why of a change rather than the what. If I actually pose a question based on them the author often tells me “yea that was some AI generated nonsense, I rewrote it now”.
You may be confusing articulate and confident with useful and accurate.
They really aren't. AI commit messages are much more thorough and correct than some of the stuff I've seen humans do over the years e.g. Fix, Fixup, Fixed some Stuff, Should work now, Definitely should work now etc etc etc.
If you can’t tell how this is a non sequitur then it’s very likely you, too, confuse articulate and confident with useful and accurate.
I definitely prefer a "fixup" to a three paragraph meaningless word soup which doesn't cover why the commit is there either
Nah, most often they are just long-winded beatings around the bush.
Not really. In my experience, the average human written commit messages were almost always useless 1-3 word "Should work non" kind of messages. The agents had a low bar to clear.
It's almost double. If you have scaling prepared, it should be linear, but I doubt they've prepared for this.
they are/were in the middle of a migration
I would give the growth hacker on their team a big raise, these KPIs are incredible
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
To some extent we are getting more things done as well. In my company (mid-sized startup), they're making us push features every other day now as opposed to maybe 1-2 features per person per sprint. Back when I joined, things were a lot slower. Today, they expect freshers to push new features on day one.

But ofc, slop has increased a lot more as well.

That's very fast. Feels like you cannot really review this code so it's all just "working" AI code with few railguards and few human supervision?
Jumping on the great asymmetry of output between humans and llms, yet still being responsible for the product is pure kafka.
And how many of that features are actually used?

How long do they last until the get replaced by the next feature?

I have some big issues with this technology and the companies behind it, but I know of quite a few people personally who were not previously coders but have now been able to use LLMs to make their own custom software, solving real problems they had.
Do you believe this is a significant part of the increased GutHub traffic, people’s personal software?
Are non-coders these days aware that they should use version control (and push their code to GitHub)? Or does their agent helpfully suggest setting up a GitHub repo?
One of the things I've done is create an entire fully functional GitHub alternative that does more of what I want and hosts all of my other projects, so yes I at least am getting considerably more done.
You could have just installed Forgejo.
I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.
Same. Helping family out when I generally was too tired to do it earlier. They're all very happy too
Are they happy about the circumstances of AI too?

Higher resource consumption and the set back in CO2 reduction?

you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc
> raised heat is a kind of pollution

> water is not unlimited

> you don't want to release a bunch of acid into a river

> I know next to nothing about this

> I know basically nothing

> corn is one of the thirstiest major crops grown in the US

What is this rant supposed to inform? Whats wrong with OP being concerned about the costs of operating a DC?

Nobody is concerned about the costs and environmental impact of data centers.

When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative.

The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproductive.

Wrong.

I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets.

So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs.

Since the rise of AI the environmental impact doesn’t seem to matter anymore.

I guess because it’s the shiny new toy of the hackernews audience.

Privacy also lost importance given the fact that the same people who refused to give information like their phone number to companies like Google and Meta now upload their whole life to their AIs to asks what should the eat, hyperbolically speaking

You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI.

The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.

The environmental impact is anything but moderate and the benefits are not obviously tremendous at all. I'm happy using Claude Code as much as the next guy, but saying that the impact has been "tremendous" is vastly overstating the actual results.
I disagree, with the projected doubling by 2030 we're looking at 3% of global electricity consumption or 3.4 EJ, less than 1% of final energy consumption.

That is moderate. Energy-intensive industry is around 130 EJ, and global final energy consumption > 450 EJ.

Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035.

Now that was about operational energy consumption. Someone might bring up manufacturing and construction.

From what I could find the climate impact of those are estimated somewhere between 10-35% of the total climate impact of data centers, so relatively small compared to the operational energy consumption.

It is very hard to justify more than moderate environmental impact here, in my opinion.

For the benefits of AI, my personal results have been great, so I am quite optimistic. And objectively, I find it hard to ignore recent results in mathematics and security research.

"Global data centers consumed around 415 terawatt-hours (TWh) of electricity—about 1.5% of the world's total electricity—with AI acting as a primary accelerator for new power demand."

https://www.iea.org/reports/energy-and-ai/energy-demand-from...

That is today where we already consume too much. If by 2030 AI's consumption doubles it gets worse. While training large models draws major initial power, everyday AI usage (inference) now drives roughly 80% to 90% of cumulative AI energy

"Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035."

Seems like AI helps slowing down the rise of energy consumption.

We are at a point where we want less CO2 not moderataly more. In the end more is more.

If your doctor tells you to lose weight or you get sick it's not a success to gain weigth slower

Well, at least we could establish that the environmental impact is moderate rather than extreme.

And if the potential of >13 EJ/year is actually realized, it would seem like the net impact of the data centers is not just "moderately more CO2" but possibly "moderately less".

Please avoid low quality analogies on HN.

Please avoid acting as the judge for analogy quality.

Anything but a reduction is bad and AI is a setback for that.

Potential benefits are as long useless as they aren’t realized.

I'm pretty sure that most of people private data contains data about third parties too. It's more as their own privacy they compromise
What make you think I‘m talking about water consumption?

The construction of data centers needs resources and also creates more the CO2.

The energy for these data centers is often created through fossil fuels which also creates additional CO2

What do you think why Google and MS scrapped their CO2 reduction targets

A link to a YouTube video without context e.g. summary of findings, primary author/creators, and primary citations, methodology, and so on is a next to useless for making a point. For all we know you’re linking to a crackpot or an industry sock puppet and I don’t care to “watch” any of what can potentially be a dubious video or worse a malicious video. It is your job as the linker to convince me the video is worth even one iota of my time.

On the other hand recent papers highlight the validity of concern/suspicion:

> This systematic review demonstrates that the environmental footprint of artificial intelligence is a structural and increasingly consequential challenge, shaped by interdependent decisions across algorithms, software pipelines, hardware infrastructures, and deployment contexts. The synthesized evidence shows that energy consumption and carbon emissions associated with AI systems are highly variable, context-dependent, and often underestimated — Beyond Efficiency: A Systematic Review of Energy Consumption and Carbon Footprint Across the AI Lifecycle (published in “Sustainability” an international, peer-reviewed, open-access journal) https://www.mdpi.com/2071-1050/18/3/1359

"I know nothing about the subject, but I would _guess_ that the power consumption is bigger concern than the reported and studied water usage."

That's the whole 21 minute 59 second video in a nutshell.

A loud and hectic quick cut rambling video essay. Millions of views, naturally.

I’m curious, what was the application of AI that helped the family?
Most of my family are small business owners, so I've done a handful of Excel automations that they needed that generally take me a while to figure out. Landing pages and a few 3D designed prototypes that I used the Fusion 360 MCP server for
Other projects get unmaintained with maintainers burnt out by a torrent of vulberability reports
This I don't understand.

If it's not your job, then just ignore the reports.

If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out.

Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.

> If it's not your job, then just ignore the reports.

If you have a highly conscientious personality, this is easier said than done.

It's also a great opportunity for character development. Use it for that, and not for trying to overbook yourself to 300%.

You don't owe the world anything at all. If you're conscientious, then give a little -- here and there. Don't turn it into an unpaid job.

" highly conscientious "

Just doing what others wish is not conscientous in itself! It _may_ be depdending on situation but it can be just pathological towards the self.

When it's psyhocologically hard to do things you imagine will dissapoint someone that's probably not concientousness. It's more like low self-esteem or codependency.

It's very hard for someone to tell these apart themselves. Hence when this topic pops out it's good idea to remind that being super-accomodating may in fact be a personality flaw - that can be healed if acknowledged.

There is very large spectrum between "trying not to dissapoint anyone" and doing what you know is the right thing.

Not everyone acts rationally even when knowing that they act irrationally.
Sounds like their problem, not something a SaaS product should dance around.

Yet they kind of did. I've limited participation in my libraries with GitHub's setting that nobody who made an account in the last 6 months can do anything in my repos (after some misguided hustler thought they're an easy target and posted an ad).lp

Time's marching forward though. Wonder what will happen after a few more months. We'll have bot spam accounts that are no longer as fresh.

https://xkcd.com/2347/

A great majority of business applications do run on open source projects, and in turn, are affected by them if things go awry. It’s a prisoner’s dilemma in this case.

usual crying from the usual people. AI slop, blahblahb, ...

(I don't mean you. just these so called open source developers.)

I'm not sure if it's going to persuade you but here is an example:

https://github.com/uclouvain/openjpeg

Basically the only library for reading jp2k data (complicated specs, ask your AI to one shot an implementation, mine said "it's 3000 lines of fiddly spec, too complicated"). Issues full of buffer-overflows. Recently unmaintained.

Used in tons of projects, now all possibly vulnerable.

and? don't use it. switch. abandon.

did we lose anything of value? probably not.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination.

If the accepted claims are of "100x productivity" (increasing every month), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in the status quo of software?

If one claims such extraordinary figures of 100x increased productivity, a step forward never seen in the history of humanity in such short timespans, they must present extraordinary proof or be branded as a complete lunatic.

I've heard people use the same word, I was dubious but they did produce some stuff, yet I think that it ends up as an itch-project. You're satisfied you saw the thing emerge into existence but that's about it. No more drive after that. Maybe because LLM don't require you to have a real long term intense need for that thing.
This is exactly my experience. A month of excitement building something that I wouldn’t have had time to do myself, then something broke in the setup and my motivation didn’t extend to fixing it.

I’m only back at it four months later and I don’t really know what happened before, or why it’s working now, I’m just happy that I can scratch that itch again.

> I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.

The ratios are factual though. Just look at the "Insights" tab of any LLM written project. https://github.com/oven-sh/bun/pulse

This kind of velocity is impossible to achieve manually.

velocity - maybe. usefulness? Maybe even easier. As they say data is not information. Commits are not (necessarily) anything useful.
That’s a measure of lines of code, I suspect the parent is talking about what results those LOC create.

AI built me a 1.5k+ LOC react component which is probably a 15x increase on the file size I would have created, with negative impact on the project for those extra LOC.

> where are the 100 year improvements in the status quo of software?

Most software work is just churn / doing the same thing over and over. More productivity can just mean more output, not better output.

The analogy here would be : a kid who wants a toy but does not have money. So he keeps dreaming how he some day would get that toy and how he would play with it and how it would make him happy, but times goes by and he still does not have money to get that toy. Then suddenly along comes LLM and you have infinite money to buy you all the toys that you wanted, you get them, but you now don't have time to play with them. Because time is money and just like before you didn't had time to "buy" the toys, now that you have them you still have no time to play with them.
I think using AI often feels faster than it is because you put less thought and effort into the problem yourself.

Another possibility is that the people who experience these 100x productivity increases are honest, correct, and simply had abysmal productivity which has now been increased to near-average junior levels thanks to AI.

The variance is extreme though. Thanks to Claude Code and Codex I've been able to make several non-trivial internal tools and libraries without writing much in terms of code, just some reviews here and there.

I spent a couple of days on those, and its would have taken me months to write manually I am sure, so in that regards it's close to 50x.

I've also had Claude track down some logic issue in a module I was unfamiliar with which had very large and complicated flows. Would have taken me many days, since I did not have a reproducible case, so had to go by logs and customer description alone. I spent 5 minutes writing a prompt and when I checked back, Claude had identified the issue. The fix I had to implement myself, but was fairly easy. So there Claude definitely was a 100x increase in productivity.

Then there are cases where they're much more modest, or where they might even be negative, when they think they're fixing stuff but actually are introducing more bugs.

Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources and $$$. Now I can iterate over the ideas I didn't have the capacity before, both intellectual and time-wise.
> I never have been this productive before

So, what are you doing/ have done with all your productivity?

writing code by hand of course /s
Enjoying free time that otherwise I wouldn't have. Building stuff that otherwise I wouldn't have time or resources for.
You can accept what you like, but it's true. Our team and our business is incredibly more productive. The number of new valuable customer facing features, and the number of PRs (which represent REAL work, not taking a PR and splitting it into 200 PRs game) have all increased dramatically. We've shipped something like ~10x more PRs so far this year than all last year. And we've done that without increasing the number of bugs and outages.
Great, number go up. Have the features led to actual customer growth, or just increased productivity?
Yes lots and lots of growth.
Best code is the one that you have not written :) Because the goal is not the code , it is the things that code does, and if it can be done without code, its the best code. Also if you produce lots of code that does not do anything in reality, then its worst code.

And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.

But what about my pet project? The one that I just do for the fun o seeing AI go brrrr, do you mean now I should care about the Earth?

Note: It's sarcasm.

More done + more leisure.
That's how it's sold, but have you heard about any employer that sends their worker home when they've achieved what they used to achieve pre-AI?
(comment deleted)
This is anecdotal, but I know that a lot of my coworkers and coasting and putting up one AI generated PR per day which they've hardly even self reviewed.
In a good company that will come back to bite them next performance review.

So, if it doesn't then you learned something about your workplace (and it's not good).

> In a good company that will come back to bite them next performance review.

In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.

Sure. I meant if they are stubborn.
Middle management was all cut so all managers have 30 direct reports and "Don't have time" to do more than a couple 1:1s per year.
These individuals are demonstrating that they can be replaced with AI. I worry for them.
The only way that you can translate productivity gains into more free time in a salaried position is by working remotely. Seems obvious, no?
Is it used afterwards or done and forgotten?
I built a travel companion app including planning full itinerary in two hours that would have taken a few weeks of full time work and then added many more features in less time than those features would take in just meetings in a real office let alone build/test them.

I'm sure I'm not alone. While AI luddites are spreading FUD, real builders are building heads down.

Lol you don't need an app for this. My (barista) gf did the same thing by just telling Claude to create pages in notion.
And you don't need excel. Pen and paper gets the job done. Good logic.
Has anyone noticed an increase in the quality, performance or capabilities of the software they use?
Only in software that is used to make other software. So we're all just patting ourselves on the back within our bubble.

What I have noticed an increase though is in demands and pressure to deliver.

I have. Mostly for the software I use that I maintain myself though :P
I have. In fact, some of my hand rolled stuff is actually _provably_ faster and more secure then even heavily battletested and widely used "industry standard" solutions. This is mainly due to them passing an exhaustive barrage of millions of lines of code of tests (literally, in fact I just did a pass over all my tests/ dirs and it's sitting at 6.3 million as of today) ranging from adversarial CVE probing attacks to fuzz tests. Ironically, my same testing suite has caught _numerous_ bugs in production stacks (openSSL/libuv) particularly, literal hard SIGSEGVs and the like.

On CVE probing, and I haven't really seen anyone describe/use it (or I may be oblivious), but the way you do it is you curate a list of CVEs for the class of software you're writing, say a web server. Then you take this list in chunks and hand them off to your agents to devise and implement adversarial technically analogous attacks against your codebase. If it's red, report and patch. Ironically (even with Fable 5) it's never complained/refused to do it.

My unsupervised agentic setup has built something that I wanted for decades but never built, so I guess that’s an increase in capabilities.
I play modded Starfield. There's been a clear increase in mods lately. Some of them are from self-proclaimed non-programmers who are using the LLM's to reverse engineer the game or other abandoned mods, and they've started to create new cool mods or they've fixed various engine limitations. Can confirm that these actually work and I can finally have my 1000+ modlist.

There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.

I'm writing much more secure software and infrastructure (e.g. IAM now gets configured well).
Well, many of the saas I use continue to have problem meeting their SLA. And I’m on support just as often as always trying to get through to a human to file a bug that will never get fixed. “It’s in our roadmap”.. no it isn’t. Even with LLMs, product will focus on making new features to push their AI mission.
At my company, we are using AI to clean up huge amounts of technical debt that would have just hung around otherwise, so yes. I'm not sure that your average user would notice, but we do without a doubt have a much better product now.
That’s like asking whether fuel consumption was productive or leisure as the number of cars on the road increased. It’s both! I don’t think you can separate one from the other in any reasonable way.
Having the full monthly commits data would be interesting. It shows the AI coding adoption rate. According to Google AI:

2020-2024: 10Ks per month baseline

2025: 1B per year

April 2026: 1.4B per month

August 2026: 2.9B per month

Its forecast is 24-27B commits for 2026, which means 2500% yearly growth.

Oh the productivity increase.

I feel it in my fingers. I feel it in my veins.

It's like cancer growth though, not the 'good' kind of growth ;) E.g. I doubt that the number of Github users has doubled in that month too. Github should probably introduce daily commit- and merge-limits that slow down excessive clanker activity, but are high enough that a human doesn't notice. Alternative put users with excessive resource usage on their own 'sub-infrastructure' so that when this is overloaded, the regular users are not affected by the outage.
this is impressive too

"We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity"

Those 3 million CPU cores are even incredible. I can't quite believe that number.
> Errors in those services triggered a client-side retry loop that increased traffic during recovery.

The worst outages I've been part of always have some version of this :(

Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
I wish I had say in our git forge decisions at work, but I don't and I can either tolerate this or quit my job. So I will continue to disparage one of the world's biggest tech companies not being able to manage GitHub properly.
[dead]
thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.
It's funny, I bet you could take any software dev and blind AB test a page written by a corporate manager and a page written by an engineer.

Here's something an engineer writes, loaded with facts:

"I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called my IT managers and learned we had hard resetting in progress for the past 20 minutes with minimal impact on recovery."

Totally missing from the article.

i am confident that if "sorry" appeared, someone would make a comment about "hollow apologies" or similar.
They should rewrite their Ruby code to a performant language.
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.

Github down, no hard drives available, no memory available, thanks AI!

Seems like we are headed for Tech Gridlock.

I fear to ask, how archive.org keeps up to catch all those events for archiving...
AWS CloudWatch has an option to show the trend and what it will be like after x-period.

Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem

(comment deleted)
The problem is there are a class of problems that only appear after you go over the tip of what your system can handle, which are very difficult to predict or model.
(comment deleted)
Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.

What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.

Notice there's nothing about paid customers, I'll add in what they are missing:

Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.

This. I own a small company with 5 people. I pay Github $250/m. I'm sorry but the narrative of, "look at this burden we have, it's hard to take care of all of this code!" is pretty insulting when I'm paying $50 per person per month to host code and run CI pipelines. If they do not want my money, I'll find a company who does.
Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.

I don't like paying for free stuff but gh certainly worth it.

You are always paying one way or another, I prefer to pay in dollars and not frustration, attention, or privacy.
Not an option. Companies are happy to sell you products at premium prices, take your money, and then still collect, mine, insecurely store, and sell all your private data.
Microsoft are doing many things to help the AI coding surge, I'm not sure if this'd be a cost effective thing towards it.
I am surprised they didn't rollback their change from 2019 when they included up to 5 private repos in free plan. Before that you had to pay to keep your code from being publicly visible.
Has to be one of the most vague outage summaries of the year
Yeah... it has the vagueness and awkward staccato of heavily edited Claudish.

I find it impossible to try to hammer raw Claudish into tolerable prose. I usually have to re-write it entirely by hand if I care how it sounds.

Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.

> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations

How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.

> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.

Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.

The comments just shows how entitled people have become. Most people use GitHub and features for free and have the audacity to complain.

The outage is due to massive load increase. In 4 months the number of commits doubled to 2.9 Billions. Anyone worked with high load systems knows that’s it’s not a normal growth and how difficult even to keep on horizontally scaling in a short time period such a complex system.

GitHub should charge at least maybe 5$ monthly fee and most of the entitled freeloaders would leave the platform and it would free up resources

> the entitled freeloaders

Now remind me again, who trained a coding-assistant without consent on those "freeloaders" code and sold it for profit?

Entitlement? Please. It's not like Github is a charity that operates on kindness and goodwill.

It's a service that is owned and operated by Microsoft Corporation, and we're the product of it.

"Most people use GitHub and features for free"

Do you have a source for that factoid? (I suspect the vast majority of Github resource usage is paid. And we are upset.)

That sounds unusual for a free platform (not a limited trial but an actual free tier). Isn't it usually the case that only some small percentage can be convinced to pay?
They're arguing that most usage of resources would be by companies, who are presumably paying.
I don't think my employer pays for our use of Github
Meh, just needs better QoS. Let the free tier shoulder the outages.
I mean, a lot of us have paid GitHub a lot of money for CI on private repos. And when GitHub themselves encourages the insane behavior of vibe coders and agents instead of just charging or rate limiting access of bots, it's hard to give them sympathy.
It seems that paid users are equally impacted as free users.

It sounds a little bit unfair to me.

Using a corporation's free offerings isn't freeloading. Microsoft wants people to put their code on GitHub. They want GitHub to be the place where source code is hosted, it is incredibly valuable. Saying "GitHub is sucking and I might leave" is information that Microsoft wants to know if they want to preserve GitHub's dominance.

Of course some people take it too far. Of course there are reasons that the outages are occurring. But Microsoft wants GitHub to be a core, reliable pillar of the software world. Nobody's making them do that, they do it because it's good for them.

LOL. If it dies, it dies.

People take the weirdest rhetorical hostages.

Agreed - up-time problems have been going on with Github for a while. They need to sort out their problems.
True but paid users are also impacted.

We're still staying on Github at work, but have had backup self hosted git repos as a break glass option when Github is completely broken and leveraged this several times now.

Whos fault is the massive load increase, or their inability to navigate it?

Did I iject copilot into every part of github? In fact not only did I not insert it, I have never even used it.

Did I move their infrastructure to Azure?

Did I sell them to MS?

Did I set all the directives and priorities that MS has set on them like telling everyone they must use openai for everything, and then telling them they must stop doing that and user their own ai instead?

The outage is not due to a natural disaster that no one could anticipate and no one had any input on creating the conditions. They keep the free tier because THEY want what THEY get from the free tier. They could easily have a $1 tier and various totally sensible throttle limits on various services and apis that would have avoided all this, but that would not get them the 100% user coverage that they want. So THEY choose to provide free, swiss cheese service.

It's not some unreasobable burden they labor under that anyone else should be understanding and forgiving about.

This is exactly the entitled attitude I'm talking about. Nobody forced you to use Github for free. Don't like it, use something else. Beggars cannot be choosers
I don't have to use a thing, for free or otherwise, to observe it's nature, and nowhere in that observation is there any claim to any entitlement.

It's curious why anyone outside of GH or MS leadership would even care to try to make such a claim.

   Central US data center failed to scale with it
I'm in Europe and I experienced token failures as well.
There are no EU instances of GitHub, it's all in the US only.
> What we have done and what comes next

"You've seen what we've done. The August 21st outage comes next. See you then!"

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.

I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

(comment deleted)
> I suspect that Microsoft would even prefer to have Github operate at a loss,

I assumed it does, do you know that it doesn't?

I presume there’s a lot of companies out there paying GitHub very large sums to host all their private repos.
(comment deleted)
operating at a loss and non-operational because of outages are very different. If they can't maintain service levels nobody - AI super user or quant, old-fashioned human - will be happy.
Microsoft has a financial incentive to push LLMs and also an existential reason since LLM-generated code is incompatible with the GPL.
The GPL is irrelevant nowadays anyway since everything is MIT (corporations won the license propaganda war). I think my computer has, like, five GPL programs? Ten? It's just that two of them are coreutils and Linux, and that basically holds up the GPL as a concept.
Microsoft, who would also have an incentive to extinguish the largest hub for open source development, having already embraced and extended it.

15 years from now, they'll say it was obvious.

They could do this in a way that lessens external pull requests.

For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins.

It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.

> bolting co-pilot on to github

They already did this, no?

I meant as a prerequisite to unlimited checkins.

Right now anyone can publish to public repos in an unlimited manner. They could choose to limit that and elevate unlimited to a paid co-pilot of codex bundled plan.

This is like letting someone stay in your house for free while charging them to burn it down.
If you can sell them fuel, it might be a lucrative business.
Thinking more, the object of the "house" is to extract value, whatever it is by rent or letting them burn it down, as long as the correct dues are paid, the mission is accomplished.
The long view is that MS will sell those developers the tools that slow down or put band-aids on the damage they inflict on their own codebases by using AI.