HN

Comments (696)

• Menu

  afc
  > Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
• Menu

  prennert
  Why does Github not segregate the free offerings from the enterprise or even better, all paid offerings?It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.
• krupan
  Why are we even committing AI generated code and uploading it to GitHub? I've heard we don't need to read the code anymore. Why preserve a detailed version history if AI has it all handled? Why even share code if we can all just have AI write whatever software we need for ourselves? GitHub feels like a dead end of AI is really headed to where we believe it's headed
• Menu

  blakesterz
  "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.
• Menu

  madrox
  I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free.I've been saying this for a while: https://news.ycombinator.com/item?id=47534499
• Menu

  aesthetics1
  > Since April, monthly commits have grown from 1.4 billion to 2.9 billionBonkers.You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.
• Menu

  arn3n
  Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
• Menu

  jdm2212
  > Errors in those services triggered a client-side retry loop that increased traffic during recovery.The worst outages I've been part of always have some version of this :(
• Menu

  cube00
  > Errors in those services triggered a client-side retry loop that increased traffic during recoverySymptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
• Menu

  altcognito
  Distributing across different services wouldn't be a bad idea....I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.
• Menu

  Quarrelsome
  Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
• iot_devs
  > Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.I am sure there is a good answer to why this is impractical, but it would be nice to know
• Menu

  hnburnsy
  >We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.Crazy.
• Menu

  StilesCrisis
  "... these incidents make clear that we must accelerate this work."It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
• Menu

  swedishuser
  I wonder how much of the traffic increase is enterprise vs. hobbyists? A 7 hour outage for enterprise customers is really, really bad and it's sad if caused by a mass of non-paying vibe coders. It's becoming absolutely obvious that the unlimited free tier needs to go.
• Miyamura80
  Afaik the other main issue with github downtime is the choice they made a while back coming back to bite them
• _fzslm
  I appreciate the unprecedented load GitHub is currently experiencing, but it's not just the (admittedly extreme) load of commits/pushes that is to blame.Their Copilot cloud agent offering is suffering with a case of some of the worst corporate ADHD I've seen. We built a cloud agentic development pipeline on it, and it seems like almost every other week they silently change something with zero public announcement that creates real disruption for our team.Note: that's not bugs in the Copilot platform like the article discusses. That's real, breaking changes to the platform that clearly aren't being tested/reviewed before being pushed to prod, with zero public announcement or documentation.Support is useless – we're paying customers in the 4-5 figures and our tickets go unanswered.I love(d) GitHub, but I do think they've lost enough public trust at this point that their time is ticking. With talk of new VCSes designed specifically for agents, I do believe it is just a matter of time. Which pains me somewhat to say.
• Menu

  cs1996
  "The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?
• mous_tik
  Is Azure the right choice?
• Menu

  iSloth
  Has to be one of the most vague outage summaries of the year
• Menu

  frumiousirc
  Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?
• bob1029
  I wonder what the ratio of repositories to physical machines is these days. I'd also be curious to see this as change over time.I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.
• silver92bullet
  This article seems to say that there is "no excuse" for these issues but look at all these things we changed and are changing. It doesn't really feel transparent and it feels like they aren't really taking true ownership on what has happened.
• _hzw
  I recently received a PR fully automated by Claude for an 8 years old repo. The bug is legit and the scope it affects is larger than what that PR addressed, but I no longer care too much about that legacy code anyway, so I also let Claude run free for the first time in my life, from handling that PR to fixing all related bugs.I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.No wonder why GitHub is down all the time.
• Menu

  bearjaws
  Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."
• ethin
  Is it me or is all of this essentially "we don't want to show the user anything at all when something breaks?"And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.
• verzali
  The trend doesn't seem sustainable.
• burstlimit
  After touting 1 billion commits over 2025 at universe last year… they are now handling 3 billion per month jeez. I’ll give them a little more grace after all…
• firtoz
  A lot of these projects and commits would benefit a ton from proper decentralisation. What functionality of GitHub are you *actually* using?
• pooploop64
  I don't know where else to ask this but it's killing me. Does anyone know what the hell that GitHub physical CD thing was about? Did anyone in the world get theirs?
• rrvsh
  Haven't they been migrating to Azure for a few years? How is it still only 58% done... Microslop needs to lay off the focus on AI features and get it done
• Menu

  dpweb
  Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.I don't like paying for free stuff but gh certainly worth it.
• NameError
  The 'growth in completed actions runs' graph is interesting. I assume the periodic drops are weekends, so intuitively the floor of those drops corresponds more with hobby/personal projects than people at work. It looks like there's a sharp uptick specifically in that floor since July ish.
• alex7o
  Cant they just put ai agents at optimizing their slow internal paths.
• fukaiall
  Would it be okay to suspect the recent upsurge in AI agent usage as a possible main cause of this issue?
• Menu

  jzer0cool
  Interview question. How would you handle the growing traffic needs and traffic spikes. I'm curious whether any existing architectural diagram of theirs would highlight a potential failure post-mordem.
• dowonseo
  Not again but..the increase in traffic over the last few years is way bigger than I thought
• Menu

  throwaway96230
  5X as much code in <2 years. What is all that software?
• steve1977
  Maybe Github (and especially things like Actions) just need to become more expensive?
• teiferer
  > Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.With that mindset I'm not surprised this happened and it will surely happen again.Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.
• kvemkon
  I fear to ask, how archive.org keeps up to catch all those events for archiving...
• haul_up
  The distinction between running out of capacity and collapsing when you run out of capacity is exactly right. Every distributed system hits limits, the question is what happens next.
• Menu

  yipinwong
  AWS CloudWatch has an option to show the trend and what it will be like after x-period.Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
• afgrant
  “Required several coordinated actions” is the key moment for reflection.
• lonertecher
  Is git still the best VCS today? I ask because it seems so much effort in the industry has been invested in making git scale, like Cursor's Origin, or the stories in the past with Facebook's monorepo, but they all seem like bandaids to its intended design.
• rarisma
  Github you can only post you are doing stuff about outages if its actually effective.The vibes are off.
• Yhippa
  Centralized decentralized code repos. It feels like an oxymoron.
• luciana1u
  the retry loop that made the outage worse is just the internet being extra helpful. every client decided the best way to help was to ask again, louder.
• 47635274172635
  What would happen if github was down for like a week?
• smgpie
  I have setup a gitea instance on my gitea server which I think is good for me and GitHub both. For one, I dont have to worry about GitHub service outages, and GitHub gets to be free from my toy (and mostly AI slop) projects that no one else will ever read/use/participate in :-)
• Menu

  rcleveng
  Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.Notice there's nothing about paid customers, I'll add in what they are missing:Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
• drcongo
  > We have made progress, but these incidents make clear that we must accelerate this workPretty sure this line appears in every one of these.
• anon
  undefined
• Menu

  sergiotapia
  Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts?Why let your entire platform suffer?
• Menu

  jryan49
  With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?
• Menu

  CodeCompost
  Central US data center failed to scale with it I'm in Europe and I experienced token failures as well.
• Preston67
  That is amazing
• Menu

  ivraatiems
  "We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.Github is ripe for disruption and I hope it is disrupted soon.
• greatgib
  > Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery.Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.In addition they say that they will continue their migration to azure and that azure is supporting their actions run. But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.
• anon
  undefined
• gigatexal
  With all the outages at GitHub there has to be someone willing to unseat them as the social git repo… how bad does it have to get before folks go elsewhere? Bitbucket and gitlab exist but are pawns compared to a king no?
• 0xbadcafebee
  As I mentioned before (https://news.ycombinator.com/item?id=49333107), they can mitigate these issues with limits, even for failure cascades. There should've been an all-hands-on-deck feature freeze 6 months ago to implement the limits needed. That clearly didn't happen.I think it's because their leadership actually doesn't care that it goes down. A weekly outage is now an accepted cost of continuing to allow unlimited free access with infrastructure that cannot possibly handle the load. As a result, everyone is looking at their GitHub Enterprise bills and cost of stopped work, calculating how much they'd save by self-hosting.
• kypro
  This is a really good post.I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
• sleepybrett
  github controls the productivity of a large number of very large tech companies. When there is an outage like this they are basically shutting down a significant number of factories for hours at a time. This would be as if during the hayday of detroit they just turned off the power grid at a time randomly at least once a week for hours. It's unacceptable. The amount of productivity lost is staggering.We should be building tools that help us all move off of github as soon as possible. The amount of action code that will need to be rewritten is daunting.
• hbcdbff
  1.5 billion commits of worthless slop
• ashu0x
  someone needs to build a open source aws
• mark89h
  nice
• ChrisArchitect
  Related recently:GitHub has alternatives, but no replacementhttps://news.ycombinator.com/item?id=49135365Why developers are ditching GitHub for Codeberg and self-hosting alternativeshttps://news.ycombinator.com/item?id=48842611and new entry:Cursor Origin Code Hostinghttps://news.ycombinator.com/item?id=49334209
• Menu

  lenerdenator
  We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
• jbrooks84
  Use less AI slop coding
• globular-toast
  This seems really bad, to be honest. I think we may be fucked. There's just no way all these lines of code are doing anything useful. We're now just burning stuff in desperation and confusion.
• amazingamazing
  Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
• promptsphere
  [flagged]
• rvz
  And another outage. [0] Looking forward to the subsequent post-mortem on that one.You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.6 years ahead [1] on not going all in an centralizing everything on GitHub.[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp[1] https://news.ycombinator.com/item?id=22867803
• Menu

  nycpig
  Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post."If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.I’m done.
• djha-skin
  [dead]
• throwaway613746
  [dead]
• vladsiu
  [dead]
• codegeek
  [dead]
• kburman
  Just add a queue. Now the outage is eventually consistent. /s
• mnmnmn
  [dead]
• rawland
  [flagged]
• Menu

  sajithdilshan
  [flagged]
• bibimsz
  well written
• pkilgore
  Ctrl+F "Sorry"No results.Cool
• Menu

  annoyingnoob
  Github down, no hard drives available, no memory available, thanks AI!Seems like we are headed for Tech Gridlock.
• danieltk76
  i wanna vibecode a replacement for git and call it jit
• monlockandkey
  They should rewrite their Ruby code to a performant language.
• delduca
  No sorry we messed up your work?
• addaon
  > What we have done and what comes next"You've seen what we've done. The August 21st outage comes next. See you then!"
• Menu

  jjordan
  I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-ghNo side projects? Nothing? Just seems odd.
• Menu

  kjuulh
  Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operationsHow do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
• Menu

  ryanisnan
  Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post:https://imgur.com/a/zIbT0GiIt shows zero contributions in the past year, on this account. This is a huge, huge red flag.