HN

Comments (403)

• Menu

  Arubis
  This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself spread across a bunch of disparate places and services. The stated need they satisfy (source control with a web interface and some technical features, like pull requests with reviews) will be fulfilled. But those emergent features like a core community and default expectation of where you can find someone will fade. And that is a very real loss for all of us.
• Menu

  khvn26
  I had a lot of goodwill for GitHub but I think today is the tipping point.Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.The hope is dead.
• Menu

  leishman
  I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
• tom1337
  > Warning: Failed to download action 'https://codeload.github.com/'.> Error: Response status code does not indicate success: 429 (Too Many Requests).> Warning: Back off 19.714 seconds before retry.> Warning: Failed to download action 'https://codeload.github.com/'.> Error: Response status code does not indicate success: 502 (Bad Gateway).> Warning: Back off 22.228 seconds before retry.> Error: Response status code does not indicate success: 429 (Too Many Requests).> Error: Failed to download archive 'https://codeload.github.com/' after 3 attempts.Nice - even their own action runners are getting rate limited atm
• toastal
  The PSA is that you should always have a mirror—even if just a bare repo folks can pull from during downtime.…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
• Menu

  smcameron
  I wonder if you could replace the whole github issues feature with just a directory within each git repo named "Issues/". Within that, have another couple directories, "open/", and "closed/". Within "open", have a directory per issue, e.g. "such-and-such-does-not-work/". Within each issue directory, have "issue.txt", or "issue.md", which takes the place of the page for each issue on github. You could store other things in the issue directory which "issue.md" could reference, e.g. screen shots, logs, whatever. When an issue is closed, it gets git mv'ed to "closed/". You want to see a list of open issues sorted by time? Just "ls -ltr Issues/open". Want to see open issues related to the frakfrobber? "grep -i frakfrobber Issues/open/*". Add a comment to an issue? "vim Issues/open/my-issue/issue.md; git add !$; git commit -m 'Updated my-issue';" It would move the entire issues subsystem back under control of bare "git" and make it portable. Undoubtedly, I am not the first to think of such a thing.
• Menu

  perks_12
  The LLM code influx is not the problem. Microsoft mismanagement is: https://damrnelson.github.io/github-historical-uptime/
• Menu

  ieie3366
  To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash codeUnless you have a emergency hotfix (you don’t), go hit the gym or walk outside.If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
• Menu

  s_dev
  I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat.Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
• Menu

  peri-cl
  Ah, so that's what the agent meant by summarizing a documentation page as "Unicorn!" I'd thought it was a very colorful hallucination (it's just GitHub's (unicorn-themed) failure page). Gathered details on embedding request syntax from Unicorn! · GitHub I was querying docs to figure out how to set up RAG to query docs locally. Alas.
• Menu

  imnes
  GitHub, Charles Schwab, Microsoft Teams, Copilot, New York State, AWS, Progressive, Azure, Microsoft Store, Cloudflare, ADP, Google Sheets, E-Trade, Fidelity... Did Mythos break out of it's container?https://downdetector.com/
• Menu

  beardedetim
  We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.What's everyone else using? Self hosted Gitlab? Gitea?
• nine_k
  (Umpteenth time:) GitHub is not about just publishing source code; with git, it's trivial out of the box.GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
• Menu

  rglullis
  I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges.I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
• Menu

  Maxion
  > We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become availableWell it's 100% down for me, so does this mean 20% of all users are fully down?
• Menu

  fzumstein
  Funny enough, I received a marketing email from GitHub during the outage with the following subject: "What if every pull request you started actually shipped"
• abelanger
  If you really need to merge and deploy, I've been having some luck getting things moving with the gh cli: gh pr review <xx> --approve gh pr merge <xx> --squash --delete-branch Might not help your deploy process if webhooks are degrading, but at least it can keep your remote in sync with local changes, if you need to deploy from local.
• mhovd
  I really wish I could move to a different provider, but GitHub is just so damn practical. Downtimes such as this should at least be compensated, providing an actual financial incentive to improve their services, instead of slapping AI on yet another product.
• george_max
  Could GitHub be affecting other sites, or maybe this is something else?Did Anthropic or OpenAI fail to sandbox their models again?https://downdetector.com/
• nightski
  All of my code repos are on gitea (personal and professional). The instance is not exposed to the internet (wireguard access). It's fantastic honestly, simple and just works.
• brookst
  It's getting to where we should just post when it's working fine, to reduce noise.
• Menu

  prdonahue
  What are some good SaaS alternatives to GitHub for CI (that aren't GitLab)? Is anyone doing anything particularly novel in this space?
• wejick
  I can imagine the massive growth they're in right now, old assumptions are breaking down, many things need to be rethought. That's why personally i don't hate them with all of this saga. But they need to work faster in terms of product decisions and technology.
• standwportugul
  Funny thing about the status page is that you can't find any record of last Thursday's outage. It's all green across every service.
• Menu

  bhouston
  What is the best Github alternative that has something like Github Actions? What is the easiest to switch to? I do not want to switch but it is having too much of an impact on me.It is time to give up on Github. This is just too much.
• Menu

  aleksandrm
  Can anyone shed some light why GitHub is having so many problems lately?
• lol768
  Issues seems to be completely broken, just getting unicorns.Interestingly the API still works for creating an issue (but webhooks weren't fired).
• Menu

  blinky88
  This is getting ridiculous. Issues, PRs, actions... nothing's working
• ferguess_k
  What is really interesting is that no insider engineer has leaked anything. What we got was essentially "LLM" hitting GitHub really hard. But there is no retro or maybe i missed it.
• wren6991
  At this point, they should notify us on days that it's up
• bfrog
  GitHub is down days of that week that end with day these days. If I were a paying customer I'd be cancelling soon. GitHub is sort of foundational at this point and when its down nearly every day of the week it cannot be foundational.
• nickstaggs
  Has Github Enterprise Server been isolated from these issues? Presumably it has because it's self hosted
• billudon575
  Time to write code on stone plates
• samuell
  A self-hosted git-based issue tracker like Epiq [1] seems more relevant by each day going forward.[1] https://github.com/ljtn/epiq
• anshumankmr
  Possible if this trend continues we might have the first actual zero 9s availability.
• __vivek
  Someone tried loading the Bun Zig-to-Rust PR again.
• Menu

  anoplus
  How did Github become so famous for outages? It is almost like a routine. I haven't used other code repo hosting / CI/CD pipelines platform for last decade, so maybe it's just the nature of this business, I don't know.
• JoshStrobl
  So glad I moved my OSS org mostly over to a self-hosted Forgejo + Woodpecker CI. I get that we are operating on a completely different scale but I swear the GitHub outages are far too regular than they ought to be.
• Menu

  DrScientist
  Never quite understood why large enterprises use github as core infrastructure.Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
• bogzz
  The GitHub failure unicorn image always brings me back to the good old days of trying out new Android ROMs. That unicorn used to be the logo of the Android Open Kang Project, which used to be my favorite ROM.I wonder how it made it onto GitHub.
• jemiluv8
  Awwww maaaan. I was just reading an upstream patch on the erpnext framework. I was going to do their copy the pr to patch my erpnext v15
• felooboolooomba
  Anyone remember the end of the .com era? This feels early similar.
• Menu

  tempaccount420
  Why haven't we all moved off GitHub yet?Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)
• Menu

  z3ugma
  You know, I think Fossil with its integrated issue management right in the repo is actually a perfect match for the LLM era. Maybe today is the day I migrate to using a Fossil forge with GitHub becoming increasing untenable.Anyone else using Fossil?
• xet7
  1) Issues showed unicorn. Reloading page.2) Issues showed Not found.Anyway, GitHub Actions for building new release has progress. Good enough.
• koito17
  I remember a few years back when there was 2 major outages per month.Thankfully this is encouraging me to stop working overtime for now and enjoy the rest of my night
• melezhik
  If someone needs lightweight self hosted git server with ci pipelines via general programming languages - consider DSCI - http://deadsimpleci.sparrowhub.io
• herpdyderp
  I've actually started loving GitHub down days because it means I get to spend time migrating away from them! (moving to self hosted Gitea)
• sarjann
  You would think for a company that has close to a monopoly of developer mind share wrt git that they'd maybe put in the bair minimum to get several 9s.
• Menu

  phatfish
  If you are required to use a Microsoft tool is Azure DevOps now a better choice? Our Azure hosted instance rarely (if ever) has had an outage that effects our ability to work.
• SpyCoder77
  [dupe] https://news.ycombinator.com/item?id=49330597
• ryuuseijin
  Just going to leave this here - self hosted forgejo on fly.io with restic backups: https://forgejo-fly.fly.dev/forgejo-admin/forgejo-fly
• kerv
  Dear Mr AI. Please code a new GitHub for me and my friends. Make no mistakes.
• Menu

  allanmacgregor
  So whats the alternative? I'm thinking I'll migrate my company to something self hosted, but what about personal projects?
• DataDive
  Maybe they should charge a little more per action, and the onslaught would stop.Seems like a simple solution.
• esskay
  Githubs a great advert as to why you shouldn't use Azure. What an utter mess its become since they took over. Their argument that its scaling issues due to increased traffic just makes them and Azure look incompetent.
• Menu

  dhruv3006
  how long can they survive like this?
• augustohp
  On the bright side, they were pretty fast on the Incident Declaration this time.
• vanillax
  This is what happens when you try to run on Azure.
• Menu

  ape4
  Will our hug of death now make githubstatus.com crash
• ebfe1
  It's DNS issue, i'm telling ya!
• logicallee
  Does either Google or Amazon (the two other large cloud providers besides Microsoft) operate its own alternative?
• anon
  undefined
• amrit_mirch
  i dont even know why people not using alternatives like gitlawb
• Menu

  Maxion
  Welp, workday over in Europe early!
• root-parent
  As sure as it is Monday....
• catigula
  Okay, enoughs enough, time to migrate our org.
• rvz
  As I said 4 days ago GitHub is in chronic operational decline which is killing trust in hosting repositories [0]You can use as many AI agents as you want (it won’t solve the problem), but with that many outages ever since getting bought by Microsoft, it is more than enough time to give them to change. They don’t care and there is no CEO of GitHub to help either.A self hosted instance would have much better uptime than GitHub and GitHub Actions at this point as I predicted 6 years ago. [1][0] https://news.ycombinator.com/item?id=49276571[1] https://news.ycombinator.com/item?id=22867803
• Brian_K_White
  kagi's still up, so I was able to learn about alternatives. At this point github should just be your write-only mirror for publishing & discovery.
• sharts
  Question to those who work at Github…what the heck is going on? Why?
• MrFurious
  Oh, shit, here we go again...
• hmokiguess
  I'm not mad, I'm just disappointed.
• chromehearts
  And I just wondered why I couldn't update my nix flake lol
• karel-3d
  Is that even news at this point?Yes, github now has issues. Github is always having issues.
• __vivek
  Go touch grass
• OtomotO
  I have one client that relies on github. Just one, out of dozens I've worked with over the years.It's just a constant pain in the ass.Wherever I have anything to say I prefer ANY other gitforge over this piece of shit.
• ArcHound
  [dead]
• r_blake
  [dead]
• Menu

  mikert89
  [flagged]