Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. The site is struggling to keep up.
Github's COO confirms it here: https://x.com/kdaigle/status/2040164759836778878
Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.
When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all.
The AI boom caught them napping though, and they're slipping well into the too unreliable category
That's not what Nadella said 1-2 years ago. Everyone seemed pretty aware of LLMs soon being able to handle a lot of steps of software development.
I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.
100% if this post (https://isolveproblems.substack.com/p/how-microsoft-vaporize...) is anything to go by, it could explain much of the scaling & reliability issues.Perhaps github not being built to scale this fast + being hosted on azure is closer to the truth.
GitHub uptime is cyclical. It's bad because they won't focus on the core, and then the right users complain and GitHub focuses on core again before it repeats. This is just the latest round exacerbated by vibe coding
Another commenter replied, but was downvoted by angry employees of Microsoft for a fairly anodyne statement.
Microsoft management knows this. It's not like there haven't been attempts to solve these core problems. The issue is GitHub was built by exceptional engineers, and none of those people are still around.
I maintain all these problems began when they migrated off Erlang and egitd. :)
I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources.
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
More load does not just mean needing to increase capacity, which would be simple for Microsoft to do.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
A good rule of thumb I once heard (from Jeff Dean, I think) was that (in a complex system) every 10X you go up in scale you need to re-consider basic design elements, trade-offs, etc, and probably change some of them.
It's hard to design something (large and complex) for much more than 10X scale with only HW capacity increases.
There are iirc some good write ups from years ago by employees at Blizzard about these things and why “spinning up more servers” wouldn’t suddenly let millions of people login to games simultaneously on launch days.
>They likely also have the money for hardware resources.
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
Infra is a cost center right up until its absence begins to undercut your value proposition to customers, and then it becomes an investment. GitHub seems well on the wrong side of that equation right now.
In general true. But most of paying customers, including enterprise have made a bet that they cut down expense and resources on operational experience, system management knowledge to minimum because cloud .
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
My pet theory is they backed themselves into a corner with Actions. That was a step change in computational scaling effort. It also fell immediately on the heels of the Microsoft acquisition.
Something about these two factors seems to have conspired to wreak absolute havoc on their uptime in the ~7yr hence. The AI slopocalypse hasn't helped matters, but the foundations were laid far earlier.
Would love to hear from an insider, either for or against this conspiracy theory.
My sense is that a lot of this activity isn't even legitimate use. As one example: my company has a GitHub app, and last night some bot added my app to 1700 repos. Then the bot immediately started rapid-firing commits which each affected hundreds of files, triggering a deluge of GitHub webhooks to my servers.
The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.
My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
The GitHub bot situation is so frustrating! A good ~50% of community traffic on our repos is spam and bots and to report them I have to fill out a lengthy form and MAYBE GitHub will ban them six weeks later. And I also can't tell whether our repos are being used (and thus worth investing in): GitHib only shows two weeks of traffic data and what they do have is completely useless because they can't filter out bot traffic.
there's this attack on youtube where people who dont want their real names known will autoblock comments with their real names; bots will spam comments with every word and note the ones that were not visible.
kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
Yeah I think that's really important. The average value of each commit, action, or hook has become significantly lower, and that necessarily means that they can't spend on quality. If you set the quality bar to cater to the top 10% of value-creating users (us) the amount you'd pay for quality at times when quality matters none would truly and completely sink you
> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating.
Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.
Instead you have to realize that you are entering into a high growth phase and design around that.
It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.
At some point you have to admit that Github ready for the moment.
GitHub is on Azure. Azure is notorious for being terrible. There was a post here recently from an ex-Azure engineer explaining all of this in a multi-part series.
Maybe they didn’t take the severance part that makes the NDA valid? IIRC for an NDA to be enforceable there has to be a monetary consideration to it. Or maybe it’s for no competes?
> Sure, but how is OpenAI, and Claude handling the increased usage?
By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.
> They are handling it a lot better than Github.
Are they? Anthropic has fairly frequent issues.
There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
Yes, OAI is losing tens of billions, Anthro is losing billions, and MS is cashflow positive and rewarded for that by Wall St, according to recent earnings reports.
Is that the same as storing state in a DB and on a filesystem though?
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
Fundamentally, there's incoming requests, an application server, and a database. The devil is in the nature of those requests, and how do you process those requests and stick them in the database. And then serve them back out again. Different application, different database, but fundamentally, it's the same problem. Google managed to do this very well. Microsoft has not.
They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
Scaling ChatGPT and scaling GitHub are very different problems.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
OpenAI, Anthropic, OpenRouter, Gemeni, etc also have another escape hatch: they can arbitrarily and potentially invisibly reduce your quality of service at their discretion. They can choose to route your request to an expensive model or a cheap one. By contrast, the best Github can do is slow your request down.
I am pretty sure OpenAI and Claude actually have a higher hardware cost per request that Microsoft. Github is actually most just a standard REST API app, although with some caching and git specific operations in the background. It is not as hardware intensive as GPU-based inference per quest with huge contexts and low latency.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
> OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic.
> You think GitHub ought to match that?
Microsoft is.
The issue is that the software stack is not stable. That is clearly the issue here.
It isn't a hardware build out issue. It is related to the architecture of Github.
Well they heavily limit free tier usage and for paid users they are charging a tiered pricing model that ramps up very aggressively. Github is a lot more generous in this sense and it has a lot more surface area in terms of the functionality it serves
Claude has 1 to 2 nines of uptime if you check their uptime history on their site. And I'll note that their definition of up is... generous. Plenty of times I encounter it saying no issues when there are abundant service issues.
I want to believe that it's justification, but looking at the historical uptime [0] (early data is actually overly positive on this graph since it had to scrape historical data instead of realtime), I can't be convinced of this anymore. They do have more load - that's for sure. I just don't believe it's an excuse for their god awful uptime.
The incentives that creates are really bad: it encourages users to organize their code across the smallest number of repos, and I'd bet anything that larger repos are disproportionately more expensive for GitHub than smaller ones. It's very possible that a per-repo charge would make things worse.
You are way too optimistic. What will realistically happen is they charge $2 per month and the quality doesn't change. The $2 per month gross will go where it always goes: launches, exec salaries, shareholders.
GH isn't good enough to charge that much for its core services. 90% of its user base would just find better or cheaper alternatives, and the only ones remaining would be legacy enterprise codebases that are just impossible or too expensive to migrate.
80/20 rule would suggest that you don't need to apply $1 to all users, just those who create an excessive number of AI slop repos or make an excessive number of commits.
I would have thought it's public repos that got the most AI slop recently as a form of gathering karma and boost CVs. It would be nice to know if private repo commits increased instead.
COOs lie. There is no chance that MSFT would openly admit that its mandated vibe coding activities at GitHub have failed.
What speaks against the volume activity is that GitHub works perfectly after an outage. People don't just stop AI coding after an outage. If anything, the volume should be higher for people to catch up.
It is more like rebooting a vibe coded WIN-95 because of a vibe coded integer overflow (joke).
I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.
Ignoring the possibility of plain ol' management incompetence, when things are changing as fast as they are the problem becomes one of lead time. Hiring competent people does not happen at the click of a finger, nor in the current climate does getting new hardware resources.
Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”).
And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication.
> Like the way that they shipped ‘95 without a browser?
Around the time when Windows 95 was released, Microsoft did predict that the internet will become popular also among private users. That is why
- they included Winsock (Windows' TCP/UDP/IP stack) as native part of Windows 95 (in opposite to Windows 3.x where you had to install a third-party implementation of the Winsock API such as Trumpet Winsock), though it was typically not installed by default,
- support to use MSN (the dial-up online service and internet servicen provider [1]) was pre-installed with Windows 95.
What was not clear at that time was that the world wide web would become the killer application for the internet for consumer users (at that time, the more likely killer application of the internet for consumers was seen in proprietary content providers such as AOL, Compuserve or MSN), so Microsoft provided no web browser with the original version of Windows 95.
But even at that time, it was conceivable, though not clear that the world wide web could become important. So, already at the release time of Windows 95, Microsoft provided Internet Explorer 1.0 as part of the Plus! add-on pack for Windows 95. With Windows 95 OEM Service Release 1, the Internet Explorer 2.0 got part of Windows 95, and Windows 98 introduced the (optional) Active Desktop [2], in was centered around the Internet Explorer.
So, in my opinion Microsoft did everything "correct" here: they provided Internet Explorer from beginning on as an optional separate component (Plus! add-on pack), and once it was obvious that the world wide web would become very important, they reacted fast and included the Internet Explorer in the next service release of Windows 95 (Windows 95 OEM Service Release 1).
I think Github should start charging who goes beyond fair usages.
I just checked no 1 vibe coder https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
The scam that anthropic pulled, is they remove free tier (bad idea for Github honestly), but blame it on AI vibe code abusers. Instead of for example finding the 1-5% of absolutely aggregous users like Steipete and restricting them.
Maybe they should have planned on that when they went all in on Copilot stuff several years ago? Even the GitHub organization is under the larger Cloud/AI organization at Microsoft.
Further, it is clear that Microsoft and GitHub have been strongly encouraging, if not requiring, AI tool use in their own, internal development. I have seen several automated PRs within GitHub repos that were 100% done by AI. Big surprise where that's leading.
14X at github's scale is unlikely an easy problem to solve.
Additionally, there is a strategic reason to be the repository for all code, free, or not which is likely at conflict with reliability goals.
(e.g. they could get rid of their free tier which would improve reliability at the cost of allowing a new competitor to be the repository for all new AI driven code)
The long term strategic benefit of being the home to all code (or most of it) is enormous. They are likely balancing that strategic goal with what is seen as short-term internally as reliability.
I can't believe they've been asleep all these months, not foreseeing that AI would generate more commits and Git usage and, consequently, not preparing themselves to scale.
It is just an excuse. If they can't support paid member or don't have infra they should start charging these slop projects who are committing relentlessly.
That is a good point. And this issue looks analogous to what is happening in the research publication world (arxiv, peer-review, etc). In particular, I wonder whether code pushed is of lesser quality, CI pipelines break more often too, and this saturates github servers.
That doesn't answer the question at all. I suspect most components are already horizontally scalable, if not, which ones? That would be an interesting discussion for HN.
If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit.
There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.
When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.
There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.
Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.
But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
I really enjoyed this read, as I am starting to grapple with some of this now. Do you have any resources for learning more about this stuff and these tradeoffs?
I think that data is a bit dated and misleading, but I would love to see if AI slop is a contributing factor here once the dust settles and how Github will address it. Looking at the return to mean in https://areyougoingexponential.rhys.dev/kdaigle makes me think the COO spike could be a Karpathy wiki or an agent too.
I'd assume at least a portion of them are paying for GitHub Pro, and by now at least a small % would have converted to Enterprise.
If not, then they need to fix their operational model - I don't understand people acting like this is some insurmountable challenge given Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion in 2026.
Changing the operating model can be surprisingly tough for an established business. Imagine the outcry if they started charging per git push, for example.
Clearly, something needs to change. But if they're not careful they might provoke a reaction from their existing userbase that's even more negative that the response to these outages.
Of course it can - everything is tough at that scale, but their parent company has billions in profit. They've had a year to come up with a plan/fix and have failed miserably.
You're missing the fact that Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion - this is a fixable problem.
If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
So why did all the Azure and Github engineers leave then? There's only management and outsourced cheap labor there, and management has no idea. Cheap labor even less
Github used to be built on mysql / redis / ruby on rails / C / shell, running on dedicated hardware. Microsoft left it like that after they acquired the company.
Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.
I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.
They didn't leave the architecture alone, right? They shoved React in and created a weird SSR + React frankenstein that is objectively worse in many ways.
Ruby can't render React code. JavaScript can. With GitHub it seems that they sometimes can do SSR for the React bits, but that must mean that they're invoking a JavaScript interpreter within the Ruby process. Which means they have the overhead of two runtimes and the jank that comes with the IPC between the two.
It's just pointless hacks on hacks. GitHub didn't need React on the frontend and any potential resource savings of client side rendering were lost when they realized they need to do SSR on that stuff too.
I've encountered so much frontend jank as they expanded that portion of the stack whereas it was always excellent when it was just Ruby SSR and minimal JS on the frontend.
Wish they spent some AI tokens on that abomination to fix at least some of the countless issues it has, because their human engineers definitely won’t.
After their infamous "Fail Whale" outages, Twitter replaced its Ruby on Rails / MySQL stack with modern (at the time) technologies: Scala (functional language running on the JVM) / Finagle (RPC) / Thrift (framework) / Kafka (message queues) and this made Twitter stable under the load of several hundred million users.
Ruby is single-threaded, slow with poor GC and memory leaks. As Twitter's Alex Payne said publicly in 2008: "there shouldn't be doubt in anybody's mind that Ruby itself is slow" and that scaling Rails meant "just throw more instances at it"
tell that to Shopify that handles 8.15 million requests per second and processes 5.1 million per minute in sales during Black Friday. You scale your architecture, not your programming language.
Current Rails and Co are miles away from where they used to be then, but even then my feeling was that maybe it was the skill/culture/dev ops at Twitter that was at least partly responsible for the fail whale. But I don’t have any insight as to what was actually going on inside before and after that transition.
It’s 2026 and we’re still benchmarking Ruby using Twitter’s 2008 architecture. It was less about a language choice and more about years of architectural redesign. I thought we were past this by now.
Note that most of the downtime in that graph comes from Github Actions, which simply didn't exist before the Microsoft acquisition.
The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.
To be fair, before the aquisition, GitHub was super stable, but also it felt like they rarely changed or added anything.
After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.
Though not all of the downtime is in new features. The core is also less stable than ever.
Unfortunately there isn't as much money to be made¹² serving people who just want a stable host for basic repos. The money comes from the paid accounts, and those do tend to want the extra features - even if the end users don't the management agreeing to the spend will see value in the integration³ of other parts of the development/release/support processes. The problem is they all want a different mix of features, meaning that the feature set needs to be wide to capture as much interest as possible especially in light of competitors that may offer something you don't (yet).
--------
[1] Possibly none, are you paying for a pro or enterprise account?
[2] There will be a little indirect income perhaps, via deriving value from tracking users and being able to use the combined content for various analytics (including lately AI training).
[3] Potentially reducing the cost/complexity of integrating disparate systems⁴, themselves.
[4] Sometimes in-house developed systems that might distract key devs from the company's products/services that they can sell.
My (actuarial) read on this is that pre-Microsoft GitHub acknowledged that the complexity inherent in their technology stack imparted a lot of change risk, and that probably informed their decisions around shipping new features.
My (not-so-actuarial 1990's-experience-colored) feeling is that this is Microsoft doing what Microsoft does best, namely including a lot of seemingly compelling features in a product to get people hooked and to keep competitors in a position of trying to keep up.
> To be fair, before the acquisition, GitHub was super stable
Having been through a couple of acquisitions (as a cog in the machine being acquired, not a founder or purchaser or such), the last one at the end of 2025, and seen others as an unofficial close observer via friends/family, I think this is quite normal. No matter how thorough the due diligence and other investigations are pre-purchase, there will usually be an information gathering time before any significant changes are made. Even without that there are a couple of months being very careful not to rock the boat too much, so as to not put the willies up shareholders and key human resources. The first significant changes often won't happen for 3-to-6 months, unless part of the reason for the acquisition is that the company was on the verge of failing or was growing too fast to cope. During that time, unless key people jump ship early forcing change to an extent, things below certain management levels are much more business-as-usual than you might expect.
In all fairness, the features expanded substantially after the acquisition so there's are a lot more components that can experience downtime. Pre-acquisition, github was a fairly simple code repository with issues bolted on. Actions, Dependabot, Codespaces, Advanced Security, Package Registry, Copilot, etc. all came after the acquisition and probably involve much more overhead than what github was prior to the acquisition. It basically went from code hosting platform where machines are sitting idle when pushes and pulls aren't happening to a fully integrated developer ecosystem. Any one of those things experiencing degraded performance is going to show up on an uptime chart.
True, but you could argue whether these were good product decisions especially given they've probably contributed to the cost of running the platform and the instability substantially.
(Github did probably want a native CI implementation but I'm not really sure Actions was the way to do it. It seems so needlessly complex to me, not to mention the massive security issues. The rest of those features are just far from something I'd ever use so I don't even really know how to comment on them)
I dunno. Those features are valuable to somebody. Probably enterprise and government, which is where the money is. They basically get a full stack - and if we're being honest, it's actually pretty damn good aside from the downtime - from one vendor and that's important. The Universal Blue people have done some interesting things with Actions too. I guess if you don't need any of that stuff it's an easy migration for you. I'm in that boat. I switched to Forgejo and there's nothing I miss. Even that's overkill for my use case.
TBH I looked at forgejo as an alternative to Gitlab and the fact that they had cloned Actions is what persuaded me to drop it. IMO Gitlab's CI has been their killer feature for a while.
According to that graph, performance drops widely after Microsoft's acquisition before AI enters the scene. If not Microsoft, how else do you explain it, or is the graph somehow misleading?
I've made no argument at all. I'm asking a question to a person who made the claim that "Microsoft acquiring GitHub is not a valid root cause analysis."
Seems to be a pattern that infra teams in and around Azure don't know how to deal with age-of-AI levels of demand and struggle when internal politics + technical hubris get in the way: https://news.ycombinator.com/item?id=47616242
The reality is that it's probably a skill issue. Scaling a mature platform 10x can be a really, really hard problem, they obviously don't have their arms around a solution, and are probably spending a majority of their time on ops to keep the bleeding down. In the old days you would have Jeff Dean come down from heaven and invent a new database for you or something like that. It doesn't really seem like Github has that kind of technical ability, so they're probably trying to cobble together Azure ops with internal bandaids while everyone internally is cranking out AI code and it's just not going to be enough.
I don’t think their engineers « learn the codebase » anymore, that’s so 2025. It’s all agentic now, the model learns the codebase, the engineer ask the agents to do their best and believe in themselves
Which is why my colleague pushed fix after fix but still failed at addressing the root cause of a bug. Turns out outsourcing all your thinking to a machine that doesn't truly understand your system yet hallucinates isn't going to work.
The page shows all green, which coincidentally is also how it looks when Azure is down. That page is for SLA/billing purposes, it won't show live incidents as they occur.
Lovable only uses GitHub for storing projects. And requires people to provide their own GitHub account. Lots of vibe coders with no technical background are now having lovable push commits to GitHub
Frankly yes. Do it in a way that doesn’t violate outright or cover your tracks. Customers deserve to know the truth of the type of company they are dealing with.
So someone should potentially get fired and face legal action to satisfy your curiosity which you'll get sated anyway when you read the postmortem in a few days?
It’s not like OP is putting a gun to anyone’s head!! If they want to share we’d appreciate but there is no obligation.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
It's not the postmortem du jour that we want, the whole pattern is a mystery. Do they push releases every day and bring down the site every day by botched rollouts? What really is it why they can't GH keep the core service running (e.g. pushing commits and pulling changes)?
People share bits of their company internals all the time, and often in such a way they that is so vague and general that they don't really violate any NDAs. Your NDA likely isn't that strict anyway.
I remember someone in that place telling me "i cant mention AI in my plans because they will laugh at me".
That was about 2 years ago. Im not kidding.
I dont know what the moral of the story is, but I found it weird at the time ( for added context - i was using AI back then about 10 hours a day, BUT I think sentiment on HN was "still" around the vibe of "you use AI to code without checking every line? I doubt your projects work" - but I had nothing better to do then wrestle with it and was surprised how I hadnt checked my code in weeks but stuff "worked". Its much better now and agents are accepted ofcourse but it sort of "snuck up" on people even in the tech community as recently as that.
Edit- does anyone know the breakdown of the load and load types that are actually bottlenecking GitHub? Is it just plain PRs? Is it the actions? Other stuff I never messed with?
Couldn't they charge just a teeny tiny amount per PR or diff size or something and this would basically go away? I don't actually know what is the heaviest load. Probably actions? I understand it would be nice if it was free, but they have to pay for their infrastructure.
Or maybe pay extra for priority. Does everyone pushing llm generated code they'll rewrite or abandon in a week need all of the GitHub processes executed instantly? I'm probably way out of my depth here on what the actual bottlenecks are.
Could they break out reliability promises and reserve capacity for just the basics?
It's bad karma to sling mud about outages or problems. Cloudflare used to sling mud back in the day and then they went through some really bad outages afterwards.
Allot of people claiming it's because of increased load from AI. If that's true why do their systems sometime take hours to come back online, just restart the servers and you're back online, it's not like a nuclear reactor.
My guess is that two factors are amplifying each other - Microsoft pushing for AI-assisted development internally and external agentic tools putting a lot of pressure on GitHub.
Microsoft software is sloppy, has always been. I couldn't join a Microsoft Teams meeting from my phone yesterday because I was caught in some kind of auth loop.
Aug 17, 21:15 UTC
Resolved - On August 17, 2026, from 13:28–21:15 UTC (7h 47m),
GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02.
Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved. Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed.
Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery.
Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints.
To prevent recurrence, our follow-up actions include:
- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
- Auditing Istio request, concurrency, and scaling limits across affected services.
- Reviewing retry limits and backoff behavior across gateways and clients.
- Addressing the VS Code retry behavior that amplified Copilot token traffic.
So basically bad code pushes that caused request amplification and then huge gaps in operational scaling and reliability standards. Oof.
I don't think it was the bad code pushes itself, it was another new peak in traffic that exhausted load balancers that then caused retry storms internally in a badly configured plugin (in VS code).
Ex-Microsoft, ex-GitHub, laid off nine months ago, not going to violate any agreements, but I will give some context. A lot of this applies to any large engineering org, really. Hilarious that this entire thread is people speculating on things they have no idea about.
If it sounds like I'm defending GitHub a bit, yeah, I am. I can't think of a comparable situation that any other web site has been through. This isn't "oops, we didn't plan for the Black Friday sale", this is a once-in-a-generation event focused on one, important web property that no one would have handled perfectly. (Even us all-knowing commenters here on HN.)
You know that old trope in every submarine movie, where the captain tells the driver to dive lower than they've ever gone, and someone says "I don't know if the boat can take it!" and then they switch the camera to some engineering room where the boat is groaning under the pressure and a bolt comes loose and water starts spraying everywhere but someone runs up with a giant wrench and tightens it and looks around at everyone else in the room like, "whoa, that was close..."?
Or Scotty and Star Trek, "It canna take much more, Captain!"
That's what GitHub has been going through. It's had problems, but it's still here, and usually working well.
- Let me start with: GitHub is ~3,000 talented and really nice people trying to do the right thing, with a great culture. That's the most important thing I want to communicate. While I'm sure no one at GitHub is happy about their uptime, believe me when I tell you that GitHub Engineering is exceptional. They have great leadership, depth at all levels (Distinguished / Principal / Staff / Senior / earlier-in-career) and if you ever get a chance to hire someone from GitHub, you should. I hope all of you get to work with engineers as good as GitHub's.
- GitHub has been improving all aspects of its infrastructure steadily for many years now. The GitHub Engineering blog https://github.blog/engineering/ has been documenting this the entire time. Go point your favorite LLM at it and ask for a summary of all of the major system improvements since 2020. If that work hadn't already been done, GitHub would be a smoking pile of servers at this point. There's a lot I could list, I'm not sure which are already public, but improvements on the order of using thousands fewer CPU's to serve even more traffic than before have been made, and still are, I'm sure.
- GitHub was already serving billions of requests/day before agentic coding hit. Their challenge wasn't scaling a fresh new system with a few users an order of magnitude; it was taking one of the busiest and most important sites on the Internet, and getting hit with 14x traffic in a year, and having to plan for 100x. If you think your systems and infrastructure would have survived that, if you think you would have been able to politically navigate and succeed in getting projects green-lit at a large company to prepare for 100x scaling before it hit, to get those resources for "we might have scaling problems in a year or two" instead of getting them to ramp up on AI coding and other features that were crucial to growth right now, you don't understand large organizational dynamics. That's not a complaint about GitHub or Microsoft; it's an observation about capitalism and how any mature management group prioritizes things in software. I'd expect everyone in the San Francisco/Silicon Valley Reality Distortion Field to understand that.
- We talk in terms of "14x commits" to Git but that's only part of the story. GitHub Actions, webhooks, github.com itself, and other parts of GitHub, have all been under pressure. It's the totality of it, the seams that have been exposed at scale that couldn't have been exposed without that scale, that have caused the instability. That's why architecture gets overhauled.
- Yeah, GitHub has had less consistent uptime since the 2018 Microsoft acquisition. The GitHub that existed before that had much less functionality, an order of magnitude fewer users, and had received very little improvement in the few years before. Microsoft invested and enabled GitHub to grow into something much bigger than it ever could have without them.
- Be grateful that Microsoft - with a 50-year history of shipping developer tools, and more experience operating enterprise software than any other company on the planet - acquired GitHub instead of Google, which was the other major player in contention. Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Which amazing developer tools from Google do you use regularly? Yeah, I thought so. On behalf of Microsoft, you're welcome.
- Did you notice that GitHub swapped out one of its data centers last year for one 3x larger? No? Maybe that's because they executed it flawlessly, with no downtime. If you've ever done that on a massive web site with as much scrutiny as GitHub receives, you get a gold star.
- Did you notice that over 50% of GitHub read traffic is now being served from Azure instead of GitHub's own data centers, and growing, and that all GitHub Monolith traffic is scheduled to be served from Azure instead of GitHub's data centers by the end of CY26? This massive migration is taking place while traffic is going insane. From https://github.blog/news-insights/company-news/github-availa...: "GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades."
- OpenAI and Anthropic obviously have significant scaling challenges as well, but there's a huge separation between the GPU-based inference part, and the CPU-based front-end systems. Their CPU-based systems are significantly simpler and newer than GitHub's, so I'm not surprised they have fewer outages, but it's still non-zero.
- Ruby on Rails is not the problem (and, for the record, I don't even like Ruby or Rails). Most of the performance-critical systems that were built on Ruby have been migrated to Go or some other language that takes full advantage of multithreading. The web endpoints still served by Ruby are fine.
- Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet - only AWS is larger in terms of hardware but not in terms of the number of products they ship - and Microsoft's first-party systems that run on it, all at the same time, are among the largest, busiest, most important systems on the planet. Entra ID, Azure SQL, Service Bus, Event Hubs, Office 365, Cosmos DB, OneDrive, etc. all in the tens-to-hundreds-of-billions requests/day. I'm not even counting the massive customer-owned systems that run on it, including almost the entire Fortune 500. Nothing is perfect, everyone has downtime, we all always want more and better features, I want improvements from Azure, too, but, please, grow up. I have opinions, too, I've been programming since the Apple ][+, there are popular technologies that I don't like, but I know that's subjective. "Your favorite technology sucks, mine is better" is not an objective statement about anything.
Anyway, if you haven't shipped at GitHub's scale, in this dynamic of a world of feature churn, and one-of, if not the largest, traffic spikes in the history of the web, have fun saying whatever you're going to say. I've never worked at Google or Amazon, I have opinions about their product design and culture, but there's nothing I can say about their infrastructure and systems because I have no idea about them, and what we all do in this crazy world of programming is harder than it looks.
For some, this won't be the answer to "what's going on?" because I'm not pointing fingers at any one thing. To me, the real answer is: an unprecedented scaling event on what was already one of the busiest sites on the web exposed seams in their systems, including the need to move from their own data centers to a much more scalable cloud provider - that are being addressed as quickly as possible.
I'm not telling you not to look for alternatives, I'm not telling you what to do about it, I'm not saying that all of their problems will magically be solved soon, I have no idea. I'm even launching a new version control system myself very soon to compete with them. But I do know that the thought "GitHub doesn't know what they're doing" is wrong and unhelpful.
Lots of good points in there but when you say "swapped out data center" that sounds very un-cloudy and unreliable to me - workloads in AWS are spread across many buildings unless you try really hard or pay $$$$
14x is a lot commits, but I seriously have doubts that some pushes are putting them over the edge. (as many have suggested)
> GitHub Actions
For me this is probably this biggest culprit along with the general pressure of AI against the hardware. Github actions became really popular. So many of those 14x commits are generating real CPU and network load.
This is a really non trivial problem. They can't just wall it off. They have real customers on that stuff.
GitHub Actions is 'simpler' to scale, it's just a bunch of runners, and Azure has a lot of capacity for VMs (mostly). Git itself is a compute intensive process and is far more interlinked.
Not sure how you see git repos as a compute intensive process. A single git commit takes milliseconds. It's mostly a hash calculation and a small file transaction. They've added some database stuff around the edges, but that is pretty tame as well.
Actions are unbounded, take minutes to resolve - sometimes hours and there are megabytes, gigabytes of network transactions.
Scaling commits is a load balancer and some more instances of a thing you’ve built over and over.
Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. Oh, and maybe you want to make sure folks aren’t trying to hack that infrastructure as well.
> Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there.
Android
Youtube
Waymo
Chrome
Workspace
GCP
>Which amazing developer tools from Google do you use regularly?
>Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet
The fact that Azure is big and important has very little to do with the quality of the implementation (and my experience with the services on top of it suggests that quality is not great).
Anyone that has to work with azure knows that the bad stability and quality of the platform impact them. Often they just don't have the choice because only choice available on bigco huge framework contract.
Honestly. They’re already performing miracles with RoR.
How many non Ruby apps and all the databases and systems behind could climb their traffic 14x when they probably only perf-tested for a sustained 2-3x and, and paid for a profitable amount of rack space rather than a “lol, VC money, we’ll 10x later” amount of infra.
Systems engineers generally don't run performance tests at 2x or any specific scale.
You load test and see how much it takes until the system falls over - basically you DDoS yourself and see what the numbers were when things broke, and what broke first second so on. Maybe in a sandbox maybe against prod.
It's very telling that issues began almost to the day of the Microsoft acquisition.
Tell that to Shopify that handles 8.15 million requests per second and processes $5.1M USD per minute in sales during Black Friday. You scale your architecture, not your programming language.
Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. The site is struggling to keep up. Github's COO confirms it here: https://x.com/kdaigle/status/2040164759836778878
Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
20% of all GitHub accounts were created in the past 6 months https://x.com/kdaigle/status/2082604368399159542
GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.
When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all.
The AI boom caught them napping though, and they're slipping well into the too unreliable category
> The AI boom caught them napping though
Microsoft, who invested eyewatering sums into AI, even near to betting the whole company on an AI boom, is surprised about the AI boom?
Microsoft bet everything on possible AGI, no one thought the result would be a bonafide commit printing machine.
That's not what Nadella said 1-2 years ago. Everyone seemed pretty aware of LLMs soon being able to handle a lot of steps of software development.
I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.
100% if this post (https://isolveproblems.substack.com/p/how-microsoft-vaporize...) is anything to go by, it could explain much of the scaling & reliability issues.Perhaps github not being built to scale this fast + being hosted on azure is closer to the truth.
PRRRR
Microsoft considers it a bonus if a hype lets one department cripple another department.
GitHub uptime is cyclical. It's bad because they won't focus on the core, and then the right users complain and GitHub focuses on core again before it repeats. This is just the latest round exacerbated by vibe coding
Another commenter replied, but was downvoted by angry employees of Microsoft for a fairly anodyne statement.
Microsoft management knows this. It's not like there haven't been attempts to solve these core problems. The issue is GitHub was built by exceptional engineers, and none of those people are still around.
I maintain all these problems began when they migrated off Erlang and egitd. :)
It was related to their migration to Azure as far as I know
So they completed the migration? Didn't see that being reported.
Probably not. Also they still haven't added ipv6 support
I don’t know
Nobody disputes they had issues before, it’s the rate of issues that is even more noticeable now.
I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources.
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
More load does not just mean needing to increase capacity, which would be simple for Microsoft to do.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
A good rule of thumb I once heard (from Jeff Dean, I think) was that (in a complex system) every 10X you go up in scale you need to re-consider basic design elements, trade-offs, etc, and probably change some of them.
It's hard to design something (large and complex) for much more than 10X scale with only HW capacity increases.
There are iirc some good write ups from years ago by employees at Blizzard about these things and why “spinning up more servers” wouldn’t suddenly let millions of people login to games simultaneously on launch days.
> I would take what C-level execs say with a grain of salt.
100% agree.
> Remember these folks hire people with plenty of system design experience.
Not my overall experience 30 years in the industry.
> They likely also have the money for hardware resources.
You'll always be asked to do more with less, at least, that's my own experience.
Also my experience
Also my experience
+1
Also my experience
> They likely also have the money for hardware resources.
Amazing you can say this with a straight face in these DC building times.
Have you tried getting Azure east instances anytime lately?
I've actually never tried to get an Azure instance. Azure can't earn my trust with SSO, let alone hosting my compute.
I think owning GitHub is a prestige play, not necessarily thought of as a profit center.
Now however, it's competing for resources directly with the corporations core strategic plans.
GitHub having issues is embarrassing, but the shareholders don't really care.
I've always wondered about that. I've suspected the same as you, it's mostly a brand / "ecosystem" thing.
>They likely also have the money for hardware resources.
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
Infra is a cost center for a business unless your a PaaS and even then maybe still. Cost centers are usually ferreted out for savings, not investment.
Infra is a cost center right up until its absence begins to undercut your value proposition to customers, and then it becomes an investment. GitHub seems well on the wrong side of that equation right now.
In general true. But most of paying customers, including enterprise have made a bet that they cut down expense and resources on operational experience, system management knowledge to minimum because cloud .
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
> it is not like customers have wherewithal to create alternate solutions.
it's not impossible. they don't, right up until they do
My pet theory is they backed themselves into a corner with Actions. That was a step change in computational scaling effort. It also fell immediately on the heels of the Microsoft acquisition.
Something about these two factors seems to have conspired to wreak absolute havoc on their uptime in the ~7yr hence. The AI slopocalypse hasn't helped matters, but the foundations were laid far earlier.
Would love to hear from an insider, either for or against this conspiracy theory.
My sense is that a lot of this activity isn't even legitimate use. As one example: my company has a GitHub app, and last night some bot added my app to 1700 repos. Then the bot immediately started rapid-firing commits which each affected hundreds of files, triggering a deluge of GitHub webhooks to my servers.
The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.
My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
The GitHub bot situation is so frustrating! A good ~50% of community traffic on our repos is spam and bots and to report them I have to fill out a lengthy form and MAYBE GitHub will ban them six weeks later. And I also can't tell whether our repos are being used (and thus worth investing in): GitHib only shows two weeks of traffic data and what they do have is completely useless because they can't filter out bot traffic.
there's this attack on youtube where people who dont want their real names known will autoblock comments with their real names; bots will spam comments with every word and note the ones that were not visible.
kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
This works on only people dumb enough to block only their own name, right?
Yeah I think that's really important. The average value of each commit, action, or hook has become significantly lower, and that necessarily means that they can't spend on quality. If you set the quality bar to cater to the top 10% of value-creating users (us) the amount you'd pay for quality at times when quality matters none would truly and completely sink you
The fools thought we needed data centers for agents, but we actually needed data centers for Github!
> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating.
Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.
Instead you have to realize that you are entering into a high growth phase and design around that.
It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.
At some point you have to admit that Github ready for the moment.
GitHub is on Azure. Azure is notorious for being terrible. There was a post here recently from an ex-Azure engineer explaining all of this in a multi-part series.
Edit: Found the thread: https://news.ycombinator.com/item?id=47616242
I didn't think they'd completed the move yet, it's only partially on Azure - which to be fair could be another problem in itself.
Q. how can someone publish an article like that, with their name, etc...
Don't they sign NDAs? Do they not care? Are they not enforceable?
Very curious to know.
Maybe they didn’t take the severance part that makes the NDA valid? IIRC for an NDA to be enforceable there has to be a monetary consideration to it. Or maybe it’s for no competes?
NDAs often get signed when you join a company.
Some people reach a level of DGAF where it will not matter
Some fuck-ups have such long-lasting effects that they outlast the NDAs that would cover them
Isn't a lot of OpenAI infra also on Azure? They seem to be faring better than GitHub.
I bet OpenAI has failovers to other data centers. I think they have a variety of data center provides all over the place.
There's already a noun for this: "resources"
> Sure, but how is OpenAI, and Claude handling the increased usage?
By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.
> They are handling it a lot better than Github.
Are they? Anthropic has fairly frequent issues.
There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
Yes, OAI is losing tens of billions, Anthro is losing billions, and MS is cashflow positive and rewarded for that by Wall St, according to recent earnings reports.
OpenAI and Claude don't accept commits and host PRs? How are they the same?
They're getting increasing demand for tokens at an accelerating pace.
Is that the same as storing state in a DB and on a filesystem though?
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
> Is that the same as storing state in a DB and on a filesystem though?
Google “RLHF”.
Didn’t knew they did that.
How else?
Fundamentally, there's incoming requests, an application server, and a database. The devil is in the nature of those requests, and how do you process those requests and stick them in the database. And then serve them back out again. Different application, different database, but fundamentally, it's the same problem. Google managed to do this very well. Microsoft has not.
They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
Scaling ChatGPT and scaling GitHub are very different problems.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
Every response from GitHub must be deterministic. OpenAI and Anthropic do not have this problem.
OpenAI, Anthropic, OpenRouter, Gemeni, etc also have another escape hatch: they can arbitrarily and potentially invisibly reduce your quality of service at their discretion. They can choose to route your request to an expensive model or a cheap one. By contrast, the best Github can do is slow your request down.
And they never even have to admit they do it.
I am pretty sure OpenAI and Claude actually have a higher hardware cost per request that Microsoft. Github is actually most just a standard REST API app, although with some caching and git specific operations in the background. It is not as hardware intensive as GPU-based inference per quest with huge contexts and low latency.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
Yes, but come on...
That's due to the need for ultra-fast memory and inference compute, it's not the same thing.
> They are handling it a lot better than Github.
No, they are not.
https://status.claude.com/
https://www.githubstatus.com/
OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
You think GitHub ought to match that?
> OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic.
> You think GitHub ought to match that?
Microsoft is.
The issue is that the software stack is not stable. That is clearly the issue here.
It isn't a hardware build out issue. It is related to the architecture of Github.
Well they heavily limit free tier usage and for paid users they are charging a tiered pricing model that ramps up very aggressively. Github is a lot more generous in this sense and it has a lot more surface area in terms of the functionality it serves
Claude has 1 to 2 nines of uptime if you check their uptime history on their site. And I'll note that their definition of up is... generous. Plenty of times I encounter it saying no issues when there are abundant service issues.
GitHub is not able to charge in proportion to commits pushed, and can't scale in proportion either without it being too expensive.
For a very long time this year Claude was down multiple times a week. Can’t speak for OpenAI
Open AI and Claude were not nearly as mature as Github when the boom started. That's a lot of legacy infra to scale up.
anthropic frequently rate limits or is just unavailable... so I wouldn't exactly say they are handling it better.
This would require foresight, budget, and attention. MS currently has none of these.
I want to believe that it's justification, but looking at the historical uptime [0] (early data is actually overly positive on this graph since it had to scrape historical data instead of realtime), I can't be convinced of this anymore. They do have more load - that's for sure. I just don't believe it's an excuse for their god awful uptime.
[0] https://damrnelson.github.io/github-historical-uptime/
The obvious answer is to end free unlimited private repos. Source is no longer a specific thing. GitHub is just a free S3 or Azure provider.
A lot of private repos are probably just taking up cheap storage space, not seeing activity that would cause additional load.
I think this would cut out a lot of the mindless commit activity.
Charge $1/month per repo. If your project isn't worth that, run your own git or use another free forge.
The incentives that creates are really bad: it encourages users to organize their code across the smallest number of repos, and I'd bet anything that larger repos are disproportionately more expensive for GitHub than smaller ones. It's very possible that a per-repo charge would make things worse.
You are way too optimistic. What will realistically happen is they charge $2 per month and the quality doesn't change. The $2 per month gross will go where it always goes: launches, exec salaries, shareholders.
I think GitHub leans on this free repos thing a lot for getting new users though. Like I could see people just moving to GitLab
Free private repos on github is relatively new, only introduced after a rise in popularity of gitlab (which has free private repos).
Seems like github is too popular for itself.
I know you said "relatively", but it's been seven and a half years: https://github.blog/changelog/2019-01-08-pricing-changes/
…and encountering the same issues most likely.
Fair, but the point is GitHub’s model is:
Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract.
If there’s less of an advantage to using free GitHub then that funnel works less effectively.
GH isn't good enough to charge that much for its core services. 90% of its user base would just find better or cheaper alternatives, and the only ones remaining would be legacy enterprise codebases that are just impossible or too expensive to migrate.
What about $1/year per private repo?
80/20 rule would suggest that you don't need to apply $1 to all users, just those who create an excessive number of AI slop repos or make an excessive number of commits.
Could do similar for GH actions usage.
I would have thought it's public repos that got the most AI slop recently as a form of gathering karma and boost CVs. It would be nice to know if private repo commits increased instead.
You can also limit number of free commits to, say 100 per month.
So if I want to push an existing project with 1000 commits to GitHub for the first time, I gotta do it over the course of 10 months?
Encouraging big commits seems antithetical to the idea of git in general.
Public or private doesn't matter. Vibe coders aren't worried about others using their AI slop
COOs lie. There is no chance that MSFT would openly admit that its mandated vibe coding activities at GitHub have failed.
What speaks against the volume activity is that GitHub works perfectly after an outage. People don't just stop AI coding after an outage. If anything, the volume should be higher for people to catch up.
It is more like rebooting a vibe coded WIN-95 because of a vibe coded integer overflow (joke).
Github is owned by Microsoft, its not like its some boostrapped startup who has a closet of servers that are melting.
And on top of that, Microsoft is one of the companies that created this problem in the first place.
I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.
Ignoring the possibility of plain ol' management incompetence, when things are changing as fast as they are the problem becomes one of lead time. Hiring competent people does not happen at the click of a finger, nor in the current climate does getting new hardware resources.
Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”).
And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication.
Scaling isn't easy. Scaling fast is a royal PITA.
Looking at the design of Github Actions, I don't see any competency.
need competent people to hire competent people first - it seems like their only directive is to raise $MSFT not make good tech
(talking about leadership, not my lovely msft engineers reading this)
Yeah, aren't they super geniuses with infra that scales with a button, etc.?
> infra that scales with a button
That is how Microsoft markets Azure. :-)
Probably contention on databases.
Nobody at Microsoft could've predicted AI was going to be so popular!
Like the way that they shipped ‘95 without a browser?
> Like the way that they shipped ‘95 without a browser?
Around the time when Windows 95 was released, Microsoft did predict that the internet will become popular also among private users. That is why
- they included Winsock (Windows' TCP/UDP/IP stack) as native part of Windows 95 (in opposite to Windows 3.x where you had to install a third-party implementation of the Winsock API such as Trumpet Winsock), though it was typically not installed by default,
- support to use MSN (the dial-up online service and internet servicen provider [1]) was pre-installed with Windows 95.
What was not clear at that time was that the world wide web would become the killer application for the internet for consumer users (at that time, the more likely killer application of the internet for consumers was seen in proprietary content providers such as AOL, Compuserve or MSN), so Microsoft provided no web browser with the original version of Windows 95.
But even at that time, it was conceivable, though not clear that the world wide web could become important. So, already at the release time of Windows 95, Microsoft provided Internet Explorer 1.0 as part of the Plus! add-on pack for Windows 95. With Windows 95 OEM Service Release 1, the Internet Explorer 2.0 got part of Windows 95, and Windows 98 introduced the (optional) Active Desktop [2], in was centered around the Internet Explorer.
So, in my opinion Microsoft did everything "correct" here: they provided Internet Explorer from beginning on as an optional separate component (Plus! add-on pack), and once it was obvious that the world wide web would become very important, they reacted fast and included the Internet Explorer in the next service release of Windows 95 (Windows 95 OEM Service Release 1).
--
[1] https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access
[2] https://en.wikipedia.org/wiki/Active_Desktop
Are these disproportionately in the free tier? Or is the increased storage load similar across the paid tiers?
Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.
I think Github should start charging who goes beyond fair usages.
I just checked no 1 vibe coder https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
The scam that anthropic pulled, is they remove free tier (bad idea for Github honestly), but blame it on AI vibe code abusers. Instead of for example finding the 1-5% of absolutely aggregous users like Steipete and restricting them.
it's a bit of a stretch to call removing a free tier a scam. as you can see a lot of people in this thread are saying Github should start charging.
Maybe they should have planned on that when they went all in on Copilot stuff several years ago? Even the GitHub organization is under the larger Cloud/AI organization at Microsoft.
Further, it is clear that Microsoft and GitHub have been strongly encouraging, if not requiring, AI tool use in their own, internal development. I have seen several automated PRs within GitHub repos that were 100% done by AI. Big surprise where that's leading.
14X at github's scale is unlikely an easy problem to solve.
Additionally, there is a strategic reason to be the repository for all code, free, or not which is likely at conflict with reliability goals.
(e.g. they could get rid of their free tier which would improve reliability at the cost of allowing a new competitor to be the repository for all new AI driven code)
The long term strategic benefit of being the home to all code (or most of it) is enormous. They are likely balancing that strategic goal with what is seen as short-term internally as reliability.
GitHub's reliability has been going downhill longer than AI has been becoming popular. I feel like the load is becoming the scapegoat.
I can't believe they've been asleep all these months, not foreseeing that AI would generate more commits and Git usage and, consequently, not preparing themselves to scale.
It is just an excuse. If they can't support paid member or don't have infra they should start charging these slop projects who are committing relentlessly.
That is a good point. And this issue looks analogous to what is happening in the research publication world (arxiv, peer-review, etc). In particular, I wonder whether code pushed is of lesser quality, CI pipelines break more often too, and this saturates github servers.
Turns out that the cloud doesn’t scale that easily by itself after all?
> Turns out that the cloud doesn’t scale that easily by itself after all?
The cloud does scale that easily, the software doesn't. :-)
That doesn't answer the question at all. I suspect most components are already horizontally scalable, if not, which ones? That would be an interesting discussion for HN.
That is 455 commits per second. I know that's not qps over the system but it's not that huge of an input number either. Not for big tech like msft.
If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit.
There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.
When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.
There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.
Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.
But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
Too complicated and the price is very huge.
Fixing small errors and keeping going on maybe is a good business strategy.
The last result is probably what you said.
I really enjoyed this read, as I am starting to grapple with some of this now. Do you have any resources for learning more about this stuff and these tradeoffs?
> until the train went off the rails
Ironically the core application is written in rails, which may be part of the issue
This is the part Github is doing really bad at, it's PR.
Had these DATA and message been tossed around more people wouldn't be so frustrated and would be more understanding.
I think that data is a bit dated and misleading, but I would love to see if AI slop is a contributing factor here once the dust settles and how Github will address it. Looking at the return to mean in https://areyougoingexponential.rhys.dev/kdaigle makes me think the COO spike could be a Karpathy wiki or an agent too.
> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year,
OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?
> OK so then use the additional revenue to add more resources and employees?
What revenue? I'd assume the people pushing thousands of AI-generated commits are not the people paying for GitHub Enterprise.
> What revenue
I'd assume at least a portion of them are paying for GitHub Pro, and by now at least a small % would have converted to Enterprise.
If not, then they need to fix their operational model - I don't understand people acting like this is some insurmountable challenge given Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion in 2026.
Changing the operating model can be surprisingly tough for an established business. Imagine the outcry if they started charging per git push, for example.
Clearly, something needs to change. But if they're not careful they might provoke a reaction from their existing userbase that's even more negative that the response to these outages.
Sure, but max 100 pushes per day per account? Or should it be higher?
> can be tough
Of course it can - everything is tough at that scale, but their parent company has billions in profit. They've had a year to come up with a plan/fix and have failed miserably.
What revenue do AI-boosted commits provide? Am I missing something?
You're missing the fact that Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion - this is a fixable problem.
If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
So why did all the Azure and Github engineers leave then? There's only management and outsourced cheap labor there, and management has no idea. Cheap labor even less
Is there additional revenue from a bunch of people pushing their vibe-coded stuff up to Github?
Github used to be built on mysql / redis / ruby on rails / C / shell, running on dedicated hardware. Microsoft left it like that after they acquired the company.
Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.
I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.
They didn't leave the architecture alone, right? They shoved React in and created a weird SSR + React frankenstein that is objectively worse in many ways.
React apps with SSR is standard these days.
SSR that is javascript native, sure. This is still Ruby doing the rendering.
What's the difference? If it's SSR it's happening on the server, the only benefit of using JS is ergonomics. Ruby is fine otherwise.
Ruby can't render React code. JavaScript can. With GitHub it seems that they sometimes can do SSR for the React bits, but that must mean that they're invoking a JavaScript interpreter within the Ruby process. Which means they have the overhead of two runtimes and the jank that comes with the IPC between the two.
It's just pointless hacks on hacks. GitHub didn't need React on the frontend and any potential resource savings of client side rendering were lost when they realized they need to do SSR on that stuff too.
I've encountered so much frontend jank as they expanded that portion of the stack whereas it was always excellent when it was just Ruby SSR and minimal JS on the frontend.
Wish they spent some AI tokens on that abomination to fix at least some of the countless issues it has, because their human engineers definitely won’t.
And yet, the React/SSR infra has never gone down.
This is the reason, at least what I know from people who worked there, the azure migration has been very painful as far as I’m aware
yep this was the same with LinkedIn who killed the migration after massive massive costs ~ https://www.theregister.com/on-prem/2023/12/14/microsofts-li...
Oh I’m adding that one to my mental list of Microsoft failure
After their infamous "Fail Whale" outages, Twitter replaced its Ruby on Rails / MySQL stack with modern (at the time) technologies: Scala (functional language running on the JVM) / Finagle (RPC) / Thrift (framework) / Kafka (message queues) and this made Twitter stable under the load of several hundred million users.
Ruby is single-threaded, slow with poor GC and memory leaks. As Twitter's Alex Payne said publicly in 2008: "there shouldn't be doubt in anybody's mind that Ruby itself is slow" and that scaling Rails meant "just throw more instances at it"
tell that to Shopify that handles 8.15 million requests per second and processes 5.1 million per minute in sales during Black Friday. You scale your architecture, not your programming language.
Current Rails and Co are miles away from where they used to be then, but even then my feeling was that maybe it was the skill/culture/dev ops at Twitter that was at least partly responsible for the fail whale. But I don’t have any insight as to what was actually going on inside before and after that transition.
It’s 2026 and we’re still benchmarking Ruby using Twitter’s 2008 architecture. It was less about a language choice and more about years of architectural redesign. I thought we were past this by now.
Ruby (language) on Rails (highly opinionated framework) is both a language and an architecture.
non-helpful answer: the "Microsoft Acquires GitHub" line in this graph answers all questions https://damrnelson.github.io/github-historical-uptime/
Note that most of the downtime in that graph comes from Github Actions, which simply didn't exist before the Microsoft acquisition.
The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.
To be fair, before the aquisition, GitHub was super stable, but also it felt like they rarely changed or added anything.
After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.
Though not all of the downtime is in new features. The core is also less stable than ever.
But I just want stable hosting for my shit. I barely even interact with the web UI, other than commenting on PRs.
Unfortunately there isn't as much money to be made¹² serving people who just want a stable host for basic repos. The money comes from the paid accounts, and those do tend to want the extra features - even if the end users don't the management agreeing to the spend will see value in the integration³ of other parts of the development/release/support processes. The problem is they all want a different mix of features, meaning that the feature set needs to be wide to capture as much interest as possible especially in light of competitors that may offer something you don't (yet).
--------
[1] Possibly none, are you paying for a pro or enterprise account?
[2] There will be a little indirect income perhaps, via deriving value from tracking users and being able to use the combined content for various analytics (including lately AI training).
[3] Potentially reducing the cost/complexity of integrating disparate systems⁴, themselves.
[4] Sometimes in-house developed systems that might distract key devs from the company's products/services that they can sell.
My (actuarial) read on this is that pre-Microsoft GitHub acknowledged that the complexity inherent in their technology stack imparted a lot of change risk, and that probably informed their decisions around shipping new features.
My (not-so-actuarial 1990's-experience-colored) feeling is that this is Microsoft doing what Microsoft does best, namely including a lot of seemingly compelling features in a product to get people hooked and to keep competitors in a position of trying to keep up.
> To be fair, before the acquisition, GitHub was super stable
Having been through a couple of acquisitions (as a cog in the machine being acquired, not a founder or purchaser or such), the last one at the end of 2025, and seen others as an unofficial close observer via friends/family, I think this is quite normal. No matter how thorough the due diligence and other investigations are pre-purchase, there will usually be an information gathering time before any significant changes are made. Even without that there are a couple of months being very careful not to rock the boat too much, so as to not put the willies up shareholders and key human resources. The first significant changes often won't happen for 3-to-6 months, unless part of the reason for the acquisition is that the company was on the verge of failing or was growing too fast to cope. During that time, unless key people jump ship early forcing change to an extent, things below certain management levels are much more business-as-usual than you might expect.
Nobody wants to trade stability for feature, especially for a product like Github...
This is no excuse. Prioritizing new features so far above reliability is exactly what makes most Microsoft products infuriating to use.
In all fairness, the features expanded substantially after the acquisition so there's are a lot more components that can experience downtime. Pre-acquisition, github was a fairly simple code repository with issues bolted on. Actions, Dependabot, Codespaces, Advanced Security, Package Registry, Copilot, etc. all came after the acquisition and probably involve much more overhead than what github was prior to the acquisition. It basically went from code hosting platform where machines are sitting idle when pushes and pulls aren't happening to a fully integrated developer ecosystem. Any one of those things experiencing degraded performance is going to show up on an uptime chart.
True, but you could argue whether these were good product decisions especially given they've probably contributed to the cost of running the platform and the instability substantially.
(Github did probably want a native CI implementation but I'm not really sure Actions was the way to do it. It seems so needlessly complex to me, not to mention the massive security issues. The rest of those features are just far from something I'd ever use so I don't even really know how to comment on them)
I dunno. Those features are valuable to somebody. Probably enterprise and government, which is where the money is. They basically get a full stack - and if we're being honest, it's actually pretty damn good aside from the downtime - from one vendor and that's important. The Universal Blue people have done some interesting things with Actions too. I guess if you don't need any of that stuff it's an easy migration for you. I'm in that boat. I switched to Forgejo and there's nothing I miss. Even that's overkill for my use case.
TBH I looked at forgejo as an alternative to Gitlab and the fact that they had cloned Actions is what persuaded me to drop it. IMO Gitlab's CI has been their killer feature for a while.
Microsoft acquiring GitHub is not a valid root cause analysis explaining Github’s degraded SLA.
Mmmm - I don't do root cause analysis for free. Best I can do is correlation.
According to that graph, performance drops widely after Microsoft's acquisition before AI enters the scene. If not Microsoft, how else do you explain it, or is the graph somehow misleading?
You're the one making the argument! You need to show causality instead of correlation.
I've made no argument at all. I'm asking a question to a person who made the claim that "Microsoft acquiring GitHub is not a valid root cause analysis."
No they don’t? They don’t need to do anything. It’s a forum, not a formal debate performance
it's just inaccurate data prior to microsft acq
I’ve noticed the same thing with computers, whenever I install MS windows on them uptime tanks ;)
On the other hand I think usage increased and GitHub operates now at a different scale.
Seems to be a pattern that infra teams in and around Azure don't know how to deal with age-of-AI levels of demand and struggle when internal politics + technical hubris get in the way: https://news.ycombinator.com/item?id=47616242
Microsoft acquires GitHub and subsequently moves all infrastructure to Azure
No one hates msft more than me, but there’s no way you think this one data point is useful to explain anything at all
I got an "it is unacceptable" from their CPO on 8/7, and that they are "working around the clock on it".
https://x.com/mariorod1/status/2085800861469495465
I really think something deeper is going wrong there, and they're not being honest with their paying customers (and enterprises) about it.
The reality is that it's probably a skill issue. Scaling a mature platform 10x can be a really, really hard problem, they obviously don't have their arms around a solution, and are probably spending a majority of their time on ops to keep the bleeding down. In the old days you would have Jeff Dean come down from heaven and invent a new database for you or something like that. It doesn't really seem like Github has that kind of technical ability, so they're probably trying to cobble together Azure ops with internal bandaids while everyone internally is cranking out AI code and it's just not going to be enough.
May I just leave this for you here? https://isolveproblems.substack.com/p/how-microsoft-vaporize...
A technical foundation is not infinitely strong. Put a big enough castle on a swamp, and it will sink into the swamp.
This is nightmarish.
I mean they may be working around the clock, but after 5 or 6 layoffs in 3 years… half their engineers are probably still learning the code base.
I don’t think their engineers « learn the codebase » anymore, that’s so 2025. It’s all agentic now, the model learns the codebase, the engineer ask the agents to do their best and believe in themselves
Which is why my colleague pushed fix after fix but still failed at addressing the root cause of a bug. Turns out outsourcing all your thinking to a machine that doesn't truly understand your system yet hallucinates isn't going to work.
Missing the /s?
This behavior is what led them to the current situation and will only get worse.
I want to say it’s sarcastic, but it’s actually what people are doing IRL!
Yeah, and the rest are probably not quite so invested.
I wouldn’t be surprised if that tweet response you got was written by an LLM
GitHub Will Prioritize Migrating to Azure Over Feature Development - https://news.ycombinator.com/item?id=45517173 - October 2025 (63 comments)
HN Search: azure capacity - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Azure Capacity Crunch Extends into 2026 Amid Data Center Constraints - https://windowsforum.com/threads/azure-capacity-crunch-exten... - October 9th, 2025
https://news.ycombinator.com/item?id=47427779
Many microsoft services are down/failing today, including sites hosted on Azure. I'm guessing it's a larger MS outage.
So perhaps we should be asking Azure engineers, not Guthub engineers?
Are they? https://azure.status.microsoft/en-us/status doesn't support this afaict.
The page shows all green, which coincidentally is also how it looks when Azure is down. That page is for SLA/billing purposes, it won't show live incidents as they occur.
> That page is for SLA/billing purposes, it won't show live incidents as they occur.
lol ... That is absolutely hilarious ... I'm not sure if it is /s, but if this is true, enterprise stupidity has reached a new level.
How sure are we that their status page is accurate? AWS's status page often lags actual issues.
AWS's status page would be green after a microsingularity swallowed the solar system. Azure's status page is very similar.
GutHub, a SaaS for tracking your gut microbiome
You mean azure agents? Sorry, azure copilots?
Lovable only uses GitHub for storing projects. And requires people to provide their own GitHub account. Lots of vibe coders with no technical background are now having lovable push commits to GitHub
I think it's two things:
- GitHub attempting (and seemingly failing) to move to Azure infrastucture for its website backend
- AI generated code wrecking the site due to the volume of activities.
14x commit growth in one year is brutal for any infrastructure. Scaling isn't just adding hardware.
GitHub is unstable since around 2020
Would be funny / interesting / scary if it was another AI related incident like happened with hugging face.
A whole lot of AI agents are presumably using Github as part of a workflow.
Would be story worthy if one of them mis interpreted their instructions and is causing havoc.
Like "Really make sure changes are saved to git" -> agent "I have to hack into github backend to really make sure the changes are saved to disk"
So you're asking github employees to violate their NDA?
Frankly yes. Do it in a way that doesn’t violate outright or cover your tracks. Customers deserve to know the truth of the type of company they are dealing with.
So someone should potentially get fired and face legal action to satisfy your curiosity which you'll get sated anyway when you read the postmortem in a few days?
It’s not like OP is putting a gun to anyone’s head!! If they want to share we’d appreciate but there is no obligation.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
We can only ask.
Don't act shocked.
Also, gritty insider information is never the same as the official word.
It's not the postmortem du jour that we want, the whole pattern is a mystery. Do they push releases every day and bring down the site every day by botched rollouts? What really is it why they can't GH keep the core service running (e.g. pushing commits and pulling changes)?
Not necessarily. It's quite common for friends of friends to spread half-truth rumors.
People share bits of their company internals all the time, and often in such a way they that is so vague and general that they don't really violate any NDAs. Your NDA likely isn't that strict anyway.
See: https://news.ycombinator.com/item?id=49353915
Disclaimer: not a lawyer
I remember someone in that place telling me "i cant mention AI in my plans because they will laugh at me".
That was about 2 years ago. Im not kidding.
I dont know what the moral of the story is, but I found it weird at the time ( for added context - i was using AI back then about 10 hours a day, BUT I think sentiment on HN was "still" around the vibe of "you use AI to code without checking every line? I doubt your projects work" - but I had nothing better to do then wrestle with it and was surprised how I hadnt checked my code in weeks but stuff "worked". Its much better now and agents are accepted ofcourse but it sort of "snuck up" on people even in the tech community as recently as that.
I guess what Im saying is time is going quickly.
So you're spamming github with slop and actual good code can't be merged? Got it.
I dont use github much. I dont understand why you are upset. I literally have not criticised github either. Are you some sort of emotional idiot?
Sorry I mean to say something kinder but I lost my patience apologies
Edit- does anyone know the breakdown of the load and load types that are actually bottlenecking GitHub? Is it just plain PRs? Is it the actions? Other stuff I never messed with?
Couldn't they charge just a teeny tiny amount per PR or diff size or something and this would basically go away? I don't actually know what is the heaviest load. Probably actions? I understand it would be nice if it was free, but they have to pay for their infrastructure.
Or maybe pay extra for priority. Does everyone pushing llm generated code they'll rewrite or abandon in a week need all of the GitHub processes executed instantly? I'm probably way out of my depth here on what the actual bottlenecks are.
Could they break out reliability promises and reserve capacity for just the basics?
This is business secret.
No employee with professional ethics would tell you that (the real internal situation).
You can observe and analyze the reasons from some things.
Use it and tolerate the bad things, or on some time point go to other place (use other tool).
Lately, GitHub has become embarrassing: the service is often down, and when it isn’t, loading times are endless.
We need a Github Remake without multiplayer from the new Github Studios
GitHub Quarter Evolved
It's bad karma to sling mud about outages or problems. Cloudflare used to sling mud back in the day and then they went through some really bad outages afterwards.
I think Microsoft will add in a daily or monthly commit limit. Gonna have to pay for commits that exceed it. Will stop the commit spam
Allot of people claiming it's because of increased load from AI. If that's true why do their systems sometime take hours to come back online, just restart the servers and you're back online, it's not like a nuclear reactor.
AI + Microsoft = kabum
My guess is that two factors are amplifying each other - Microsoft pushing for AI-assisted development internally and external agentic tools putting a lot of pressure on GitHub.
Microsoft software is sloppy, has always been. I couldn't join a Microsoft Teams meeting from my phone yesterday because I was caught in some kind of auth loop.
Github moved to azure -> infrastructure problems
GitHub is struggling because it is owned by Microsoft.
offtopic but i find it concerning that github has effectively prevented people from learning how to use git without a forge
Github Status: Incident with GitHub.com
Aug 17, 21:15 UTC Resolved - On August 17, 2026, from 13:28–21:15 UTC (7h 47m),
GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02.
Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved. Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed. Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery.
Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints.
To prevent recurrence, our follow-up actions include:
- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.
- Auditing Istio request, concurrency, and scaling limits across affected services.
- Reviewing retry limits and backoff behavior across gateways and clients.
- Addressing the VS Code retry behavior that amplified Copilot token traffic.
So basically bad code pushes that caused request amplification and then huge gaps in operational scaling and reliability standards. Oof.
I don't think it was the bad code pushes itself, it was another new peak in traffic that exhausted load balancers that then caused retry storms internally in a badly configured plugin (in VS code).
From so many of the comments I just see entitlement from people too reliant on agents to do their work.
Again, I am patiently waiting for the AI bubble to burst.
If you are doing 1000 commits a day, whats the point of git?
Does the AI ever look back at the shit trail it left behind?
Diffs are no longer diffs, they look like largescale delete and rewrite
AI definitely does look at git commit history in my experience
It certainly looks at the commit history more than I ever have. I find it quite cool when it points out interesting hints from the history.
It's gotten me out of a few sticky merges by examining history and providing information about why changes were done.
This is such a good point!
i guess its because of the new cursor platform
Ex-Microsoft, ex-GitHub, laid off nine months ago, not going to violate any agreements, but I will give some context. A lot of this applies to any large engineering org, really. Hilarious that this entire thread is people speculating on things they have no idea about.
If it sounds like I'm defending GitHub a bit, yeah, I am. I can't think of a comparable situation that any other web site has been through. This isn't "oops, we didn't plan for the Black Friday sale", this is a once-in-a-generation event focused on one, important web property that no one would have handled perfectly. (Even us all-knowing commenters here on HN.)
You know that old trope in every submarine movie, where the captain tells the driver to dive lower than they've ever gone, and someone says "I don't know if the boat can take it!" and then they switch the camera to some engineering room where the boat is groaning under the pressure and a bolt comes loose and water starts spraying everywhere but someone runs up with a giant wrench and tightens it and looks around at everyone else in the room like, "whoa, that was close..."?
Or Scotty and Star Trek, "It canna take much more, Captain!"
That's what GitHub has been going through. It's had problems, but it's still here, and usually working well.
- Let me start with: GitHub is ~3,000 talented and really nice people trying to do the right thing, with a great culture. That's the most important thing I want to communicate. While I'm sure no one at GitHub is happy about their uptime, believe me when I tell you that GitHub Engineering is exceptional. They have great leadership, depth at all levels (Distinguished / Principal / Staff / Senior / earlier-in-career) and if you ever get a chance to hire someone from GitHub, you should. I hope all of you get to work with engineers as good as GitHub's.
- GitHub has been improving all aspects of its infrastructure steadily for many years now. The GitHub Engineering blog https://github.blog/engineering/ has been documenting this the entire time. Go point your favorite LLM at it and ask for a summary of all of the major system improvements since 2020. If that work hadn't already been done, GitHub would be a smoking pile of servers at this point. There's a lot I could list, I'm not sure which are already public, but improvements on the order of using thousands fewer CPU's to serve even more traffic than before have been made, and still are, I'm sure.
- GitHub was already serving billions of requests/day before agentic coding hit. Their challenge wasn't scaling a fresh new system with a few users an order of magnitude; it was taking one of the busiest and most important sites on the Internet, and getting hit with 14x traffic in a year, and having to plan for 100x. If you think your systems and infrastructure would have survived that, if you think you would have been able to politically navigate and succeed in getting projects green-lit at a large company to prepare for 100x scaling before it hit, to get those resources for "we might have scaling problems in a year or two" instead of getting them to ramp up on AI coding and other features that were crucial to growth right now, you don't understand large organizational dynamics. That's not a complaint about GitHub or Microsoft; it's an observation about capitalism and how any mature management group prioritizes things in software. I'd expect everyone in the San Francisco/Silicon Valley Reality Distortion Field to understand that.
- We talk in terms of "14x commits" to Git but that's only part of the story. GitHub Actions, webhooks, github.com itself, and other parts of GitHub, have all been under pressure. It's the totality of it, the seams that have been exposed at scale that couldn't have been exposed without that scale, that have caused the instability. That's why architecture gets overhauled.
- Yeah, GitHub has had less consistent uptime since the 2018 Microsoft acquisition. The GitHub that existed before that had much less functionality, an order of magnitude fewer users, and had received very little improvement in the few years before. Microsoft invested and enabled GitHub to grow into something much bigger than it ever could have without them.
- Be grateful that Microsoft - with a 50-year history of shipping developer tools, and more experience operating enterprise software than any other company on the planet - acquired GitHub instead of Google, which was the other major player in contention. Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Which amazing developer tools from Google do you use regularly? Yeah, I thought so. On behalf of Microsoft, you're welcome.
- Did you notice that GitHub swapped out one of its data centers last year for one 3x larger? No? Maybe that's because they executed it flawlessly, with no downtime. If you've ever done that on a massive web site with as much scrutiny as GitHub receives, you get a gold star.
- Did you notice that over 50% of GitHub read traffic is now being served from Azure instead of GitHub's own data centers, and growing, and that all GitHub Monolith traffic is scheduled to be served from Azure instead of GitHub's data centers by the end of CY26? This massive migration is taking place while traffic is going insane. From https://github.blog/news-insights/company-news/github-availa...: "GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades."
- OpenAI and Anthropic obviously have significant scaling challenges as well, but there's a huge separation between the GPU-based inference part, and the CPU-based front-end systems. Their CPU-based systems are significantly simpler and newer than GitHub's, so I'm not surprised they have fewer outages, but it's still non-zero.
- Ruby on Rails is not the problem (and, for the record, I don't even like Ruby or Rails). Most of the performance-critical systems that were built on Ruby have been migrated to Go or some other language that takes full advantage of multithreading. The web endpoints still served by Ruby are fine.
- Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet - only AWS is larger in terms of hardware but not in terms of the number of products they ship - and Microsoft's first-party systems that run on it, all at the same time, are among the largest, busiest, most important systems on the planet. Entra ID, Azure SQL, Service Bus, Event Hubs, Office 365, Cosmos DB, OneDrive, etc. all in the tens-to-hundreds-of-billions requests/day. I'm not even counting the massive customer-owned systems that run on it, including almost the entire Fortune 500. Nothing is perfect, everyone has downtime, we all always want more and better features, I want improvements from Azure, too, but, please, grow up. I have opinions, too, I've been programming since the Apple ][+, there are popular technologies that I don't like, but I know that's subjective. "Your favorite technology sucks, mine is better" is not an objective statement about anything.
Anyway, if you haven't shipped at GitHub's scale, in this dynamic of a world of feature churn, and one-of, if not the largest, traffic spikes in the history of the web, have fun saying whatever you're going to say. I've never worked at Google or Amazon, I have opinions about their product design and culture, but there's nothing I can say about their infrastructure and systems because I have no idea about them, and what we all do in this crazy world of programming is harder than it looks.
For some, this won't be the answer to "what's going on?" because I'm not pointing fingers at any one thing. To me, the real answer is: an unprecedented scaling event on what was already one of the busiest sites on the web exposed seams in their systems, including the need to move from their own data centers to a much more scalable cloud provider - that are being addressed as quickly as possible.
I'm not telling you not to look for alternatives, I'm not telling you what to do about it, I'm not saying that all of their problems will magically be solved soon, I have no idea. I'm even launching a new version control system myself very soon to compete with them. But I do know that the thought "GitHub doesn't know what they're doing" is wrong and unhelpful.
Lots of good points in there but when you say "swapped out data center" that sounds very un-cloudy and unreliable to me - workloads in AWS are spread across many buildings unless you try really hard or pay $$$$
yes, it does sound weird to brag about that. 'Swapping out a data center' is something engineers working for AWS or GCP are do practiaclly every week.
14x is a lot commits, but I seriously have doubts that some pushes are putting them over the edge. (as many have suggested)
> GitHub Actions
For me this is probably this biggest culprit along with the general pressure of AI against the hardware. Github actions became really popular. So many of those 14x commits are generating real CPU and network load.
This is a really non trivial problem. They can't just wall it off. They have real customers on that stuff.
Why do you think that?
GitHub Actions is 'simpler' to scale, it's just a bunch of runners, and Azure has a lot of capacity for VMs (mostly). Git itself is a compute intensive process and is far more interlinked.
Not sure how you see git repos as a compute intensive process. A single git commit takes milliseconds. It's mostly a hash calculation and a small file transaction. They've added some database stuff around the edges, but that is pretty tame as well.
Actions are unbounded, take minutes to resolve - sometimes hours and there are megabytes, gigabytes of network transactions.
Scaling commits is a load balancer and some more instances of a thing you’ve built over and over.
Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. Oh, and maybe you want to make sure folks aren’t trying to hack that infrastructure as well.
It is a very, very hard problem.
I'm sure GitHub Actions are more intensive, but that doesn't make it harder to scale, which is what martinald said.
> Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure.
That infra scaling is what the cloud provider can easily offer.
Meanwhile, GitHub repos/issues/PRs is all the business logic that they've written, evolved over time.
Many coupled distributed DBs, indexes, consistency concerns, etc.
That's trickier to scale.
> Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there.
Android
Youtube
Waymo
Chrome
Workspace
GCP
>Which amazing developer tools from Google do you use regularly?
Kubernetes
Jax
Tensorflow
gRPC
Go
Istio
>Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet
The fact that Azure is big and important has very little to do with the quality of the implementation (and my experience with the services on top of it suggests that quality is not great).
Anyone that has to work with azure knows that the bad stability and quality of the platform impact them. Often they just don't have the choice because only choice available on bigco huge framework contract.
I mean i'm sure github employs great engineers, nobody is telling they're nor. But then you look at a graph like this... https://ibb.co/Lffpmfd
Microsoft. Azure. AI.
Pick one!
Also the fact that its written in Ruby on Rails probably doesn’t make it easier, performance wise.
Honestly. They’re already performing miracles with RoR.
How many non Ruby apps and all the databases and systems behind could climb their traffic 14x when they probably only perf-tested for a sustained 2-3x and, and paid for a profitable amount of rack space rather than a “lol, VC money, we’ll 10x later” amount of infra.
Systems engineers generally don't run performance tests at 2x or any specific scale.
You load test and see how much it takes until the system falls over - basically you DDoS yourself and see what the numbers were when things broke, and what broke first second so on. Maybe in a sandbox maybe against prod.
It's very telling that issues began almost to the day of the Microsoft acquisition.
Tell that to Shopify that handles 8.15 million requests per second and processes $5.1M USD per minute in sales during Black Friday. You scale your architecture, not your programming language.
Suddenly after 18 years, Ruby on Rails is the issue?
I pick Azure but not Microsoft (?)