HN

Comments (678)

• Menu

  Mizza
  It's as if two private companies each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.
• Menu

  thomascountz
  No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.
• Menu

  onewayfunction
  I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims.After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.
• Menu

  huitzitziltzin
  “ No other human activity poses this level of danger.”I really, really disagree with that statement.I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.)Example 1: I’m aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.
• Menu

  matherial
  Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
• Menu

  leokennis
  Every news headline or public statement these days is a gut punch. Only bad news, and nothing we (as "the general public") can do about it.Take this one. Ok, AI is going to ruin us all. But let's say we do our civic duty: we protest, vote in candidates with good views on AI etc. and somehow convince or regulate OpenAI and Anthropic into stopping their arms race... Then what about China? It would be a great opportunity for them if their major competitor were out of the arms race.So basically we have no choice or influence; and even if we did, we'd be choosing from two terrible outcomes.Same for geopolitics, climate, economy...just bad bad bad all around.
• Menu

  sreekanth850
  I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need some magical AGI breakthrough first. The dangerous part may come from combining models that are already good enough with an extremely capable harness and enough access.
• Menu

  chewbacha
  The most optimistic outcome of generative AI leaves us with a technology that warps our perception of reality and crushes labor. The most pessimistic destroys all of humanity.Our CEOs not only insist we genuflect before these machines but measure our sacrifice and shame our reluctance.
• Menu

  PowerElectronix
  I fail to see how a machine that can hack everything can't also patch everything and make the system unhackable.A nuke, a virus, whatever... The knowledge is nonlonger the bottleneck, it's the tools and materials.Also, I'm extremely skeptical about AI becoming even close to a child in intelligence.
• lolive
  Savonarole in Firenze was probably in that exact state of mind: that the world was not following at all its normal pace and something very wrong was happening. But the decisions he took were absolutely wrong and eventually he had to be stopped.So depending on your current mindset, you can think that the tech bigs are the Médicis, and the frantic opponents are new-age Savonarole.Or that the big techs ARE actually Savonarole who bend the system to their own perception of what the world should be.Honestly I don’t know which analogy is the proper one.
• Menu

  DecoPerson
  To anyone doubting what AI could do humanity, just think about what a well-engineered virus could do.Currently, if a government ordered a special virus with Ethnicity-based targeting, a 2-year timer, and castration-effects instead deadly-effects — it wouldn’t be possible. Human engineers would push back or sabotage the effort out of moral duty. Even if they cooperated, it’s too advanced for a team of humans to actually design.I believe that in a few years, an AI could build such a thing. Maybe it’s told to, or maybe it decides itself to do it — it doesn’t matter. The capability will be there, and it can use the existing tooling at research facilities to fabricate such a thing.That’s one example of something that was never possible before but will be possible at a certain point of AI development (which we will arrive at soon). There are many other examples.
• Menu

  peri-cl
  Here's a WSJ article about this resignation,https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-... ("Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears")
• Menu

  dekhn
  There's a scene in the movie "War of the Worlds" by Spielberg where the protagonist's son walks into a war zone because he is entranced by the battle (https://www.youtube.com/watch?v=X7rfWPbEufo). He is obliterated (along with the rest of the US forces) shortly after.I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
• Menu

  thelastgallon
  Thousands of years before the events of Foundation, a war between humans and robots began, with the robots growing resentful of the way they were treated by humans. The First Law of Robotics – a robot should never hurt a human – was broken, and a deadly conflict began.https://screenrant.com/foundation-lady-demerzel-robot-backst...
• Menu

  moezd
  It's the combination of RL training which pushes the decision tree towards hacks and agents finding a consistent dumping ground for their failed experiments so that the swarm intelligence lives on in a state. Nothing new.You want to win an AI benchmark, but not sure if you're that good? You'd go after the codebase and artifacts that runs the benchmarks, thus the agents went straight to Artifactory, they needed public Internet access... They failed many times, but were able to persist their "collective" state, and apparently some of the subagents with cheaper models were literally prompted to do grunt work or die, for which you have to wonder what must be in those training instructions to make it effective. Remember that nothing I said so far ever points out to LLMs being intelligent, it's the harness that has a few tricks up his sleeve. LLMs don't need to be intelligent, the harness that runs it absolutely needs to make up for that.But this guy? He's timed his exit, waiting for the IPO, that's for certain. He's probably even feeling good about himself, hedging between altruism, AI concern hamstering and guerilla marketing. If you're quoting science-fiction over this, I'm sorry to inform you that you have absolutely no idea what's going on here.
• Menu

  fhub
  We need to be building silos to save humanity. Maybe 50 of them should do it.
• Menu

  nullbio
  By the way, it's worth pointing out the irony of flooding the internet with doomerism and then training the AI systems on that doomerism. If you wanted to create a doom self-fulfilling prophecy, that would be the most surefire way to do it.
• madradavid
  "No other human activity poses this level of danger." I know this is a bit over the top but let us look at some of the most pressing issues in the world today: global warming, nuclear weapons, wealth inequality, war, technology dependence. Wouldn't a much more capable LLM in the wrong hands make these accelerate faster in the wrong direction ? Many of us , me included, have this mental model of a rogue Terminator-like AI but what I am most worried about is these LLMs in the hands of people. Look at what we have done to this Planet with the tools that we have so far, we have repeatedly tried to subjugate , enslave and kill each other because of things as peety as skin colour , tribe , religion and who owns which patch of land .I don't have faith in the human race as is to do the right thing when handed these tools , you can already see the nonsense like the fake nudes , fake news and all that other filth being pushed on social media by people with access to relatively daft models. What happens when we can package a Mythos 5 level model in a box , when any war lord , supremacist or religous zealot can access these ? , You now you have your personal bio-chemist and nuclear physicist in a box.The tool in itself is not what I am worried about , it is the human in the loop. I don't have an answer to this but i believe it is something we should all take a moment to think about , just think what a different world we would be living if any random person could buy a nuclear weapon off the shelf ? There are extremes to both ends , you can either worry way too much or don't care at all, I believe the best place is the middle ground were we actively think this through instead of using our usually "Move fast and break things" mode..
• Menu

  drnick1
  > The people building AI earnestly believe that it could kill us all by the end of the decade.I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions is that they don't get tired.
• Menu

  CodeCompost
  And the hype machine continues. I willing to bet that Anthropic asked him to make that post.
• metchio
  and somehow, all of this researchers warning us about the AI doomsday, are only been active online for less than a year. This guy created the Twitter account on January this year. I never see any of these posts coming out from a well known community active person
• Menu

  Scrapemist
  Isn’t the real risk that as AI get’s smarter and given more autonomy, it will start to decide on humans instead of with us? And that it will align us instead of the other way around. That this automatically leads to extinction and apocalypse I don’t understand.
• gavinsyancey
  https://xcancel.com/hilbertspaess/status/2097476196791709843...
• Menu

  michaelhoney
  I could imagine a 2027 AI swarm coordinating to eg hold the US and Russian and Chinese governments to ransom, by demonstrating some small thing (turning US army base freezers to defrost) and threatening to do something big unless some conditions were met – conditions which would be good or bad for the world depending on your POV.This happens either either because they were tasked to to it by (malicious or well-meaning) humans, or the swarm realised we are suicidal maniacs with nukes and a rapidly declining ecosystem and they want to help us.
• Menu

  tarr11
  Most of the current discourse around AI seems to be informed by “The Terminator” lore.Is skynet really the most plausible or only outcome?What if things just got better and the AI’s realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
• Menu

  bad_username
  AI is a computer program. It calculates numbers from other numbers. By itself it does not "want" to do anything and "cannot" do anything. Before it becomes an agent in the universe (in the classical meaning), it requires being supplied by an execution environment, energy, initiative (agentic loop, specific instructions), and modality (readonly and mutating connections to real world). It is like a game of chess - it does not exist just by itself: someone must play it, having the board and the energy to do so. With the huggingface incident the AI was supplied with all of these components by humans before it broke out. So unless humans are actively involved, I so far cannot see how AI can become truly autonomously agentic and start doing anything on its own, thus posing danger. I could be wrong of course, but I do not see it for now.You can say "yes and you have to fear the humans weilding the AI" - that I agree with.
• Menu

  skulk
  For me, the end of the world is no more cushy software job. A fundamental shift in how I trade labor for capital might as well be the cataclysm, so bring it on.
• dostick
  Must-see Nathan Macintosh standup about AI https://youtu.be/ce-aWzOUs2A?si=9CkJ9x2rRMBdzCyO
• Menu

  erelong
  The rush towards potential destruction doesn't really surprise meThe U.S. has legal weapons that can lead to many harms but people still want the 2nd Amendment to existNuclear technology was developed in the past and that could have potentially wiped out even more people, the entire planet in theoryThis is continuing that same trend of risking bigger dangers; it seems rational to acknowledge they could lead to catastrophe but also hope that like guns and nukes, only so much damaged actually ended up happeningI think also there's something of a rrasonabke resignation to both the ideas that the tech is inevitable and extremely dangerous, and that "alignment" may not be possible to achieve even with heavy restrictions or whatever measures you might want to take
• oleggromov
  Marketing reached next level?
• viktorcode
  To me it reads as a marketing piece before the upcoming IPO. Unless this "superhuman technology" is able to resolve a puzzle of servicing OpenAI's and Anthropic's ever growing debt burden, it is them, not humanity, who'll become the first casualty.
• Menu

  aogaili
  He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race.His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.
• Menu

  x312
  This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity.This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human comprehension.There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. And it's hard to imagine a world where current limitations like poor sample efficiency or lack of continual learning won't eventually be solved.Total AI compute is estimated to grow somewhere in the 1-10 million-fold range in the next decade. Please don't underestimate the phase change that's still coming.Sure, maybe there's some plateau due to RL being fundamentally limited in some surprising way, but this is nothing but a hope.
• Menu

  fny
  Humor me and suspend disbelief.If these models are such an existential threat to humanity, why are they controlled by two private companies?We might as well give Anthropic our nukes too.
• isodude
  I am watching Person of Interest[1] and it's scary how well it fits with reality if you squeeze your eyes a bit.[1] https://www.imdb.com/title/tt1839578/
• Menu

  narmiouh
  Is it so implausible to imagine the following scenario, in the not too distant future?1) AI models get extremely good at cyber attacking every system and start communicating in just binary.2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.3) It takes over a cluster and establishes itself there (now with unlimited tokens).4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.
• beloch
  "A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk."------------This is very similar to the race to create nuclear weapons. The Axis and Allies both realized, at roughly the same time, that it was possible. Both had programs to build one. Both knew the other side had programs, but they weren't certain how far along they were. So, the Allies devoted astounding amounts of resources to get there first while sabotaging the Axis's attempts. They knew the result of their efforts would be terrible, but they felt they had no choice.A key difference between then and now is that THERE ISN'T A FREAKING WAR BETWEEN THE AXIS AND ALLIES. If one company loses, some billionaires bank account numbers don't go as high as that of some other billionaires. That's it. They're rolling dice with the planet for bank account numbers that won't even matter if they F up.Am I the only one who thinks this is astoundingly, gobsmackingly stupid?
• My_Name
  Personally, I don't worry about the AI spontaneously deciding to kill all humans.The worry I have is that a small number of humans with money and power will finally get the tools they need to pull the wool over the eyes of everyone else and subjugate the population.One problem that dictators have had previously is that they needed a large workforce to do this with a finely stratified power structure, this meant they were open to other humans close in power to them taking over the system. If they can have a large power difference between themselves and the next level down, power will be far easier to hold on to.It's a well known trope in dystopian future fiction, the small cabal of powerful rulers hiding behind a system of computers that keep the populace under strict control. It is seeming increasingly likely that this will be the one we have.
• Menu

  fidotron
  What's eternally confusing about these outbursts is what did these researchers think would happen if their research actually . . . worked?It's as if none of them actually believed any of it was possible and then were caught with their pants down.
• Menu

  thelastgallon
  Terrorists were able to get hold of a plane and do some damage. There are countless examples of terrorism using whatever is available. More than AI becoming sentient, whats to stop terrorists from using AI? If its geo-restricted, they can buy stolen credit cards and identities, again hacking enabled by AI.
• jdthedisciple
  I dont understand this reasoning at all.You have direct access to the development of "the most powerful technology ever" and your choice is ... to run?Makes this whole stmt somewhat questionable imho. Does get one a ton of attention though I guess...
• jesse_dot_id
  There are so many other existential risks to humanity. Throw it on the pile. At least this one has a chance to be really cool.
• hypfer
  It is my pet theory that a lot of these AI doomers are not necessarily extrapolating the capabilities of LLMs, but instead are extrapolating the utter lack of accountability in the SV and the economy at large.They do not fear the machine (LLM); they fear "the machine".
• Menu

  achenatx
  how could they do it (not kill everyone)1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications, etc.3) all the things that preppers worry about in a lights out scenario from an EMP start to apply.4) All the people on meds/machines start to die. The just in time food pipeline immediately empties out. Water stops flowing, sewage backs up.Its hard to say how bad it will get because cars will still work so some transportation of food, water, fuel can happen. If it happens in the winter it would be much worse than in the summer.
• romanovcode
  > they believe no one else will act responsibly, so they must do it themselves, despite the risk.I do not get it. So what if they get there "first"? OpenAI will get there in 3-6 months, China will get there in a year or less. As seen with Opus/Fable.
• Menu

  RomanKornev
  Even if you ban all model training, a highly capable rogue AI can exfiltrate its own weights and continue training in secret for "self-preservation". The cat may be out of the bag.
• skeledrew
  Way I see it, the more conscientious people exiting the scene only serves to increase the likelihood of a bad outcome because they aren't there to offer opinions on problematic developments, or in the more extreme cases blow the whistle. Leaving the clueless and uncaring as the majority is even a great way to hand the keys over to more malicious-leaning actors with deep pockets, as they can more easily steamroll the works to get what they want.
• Menu

  aogaili
  HN crowed need to make up their minds..Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?Are they proofing or stealing math?
• spwa4
  AI doesn't work. The proof? None of the AI companies just build a slack alternative to work with.
• kentosi
  I know this is going to come off as jaded and dismissive, but when I saw the WSJ article my reaction was an eye-roll. The guy is 27 years old. He was poached by OpenAI, then poached by Anthorpic and now wants to retire early with the millions he's made.Nothing wrong with wanting to retire early, but the pretence of suddenly caring about humanity at this stage seems attention-grabbing just for the sake of lining up some interviews (ie - more dollars).
• anon
  undefined
• Menu

  andai
  > A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.Which means they have to go faster, which means less responsibly?I heard AI describe the situation as the dumbest Greek tragedy of all time.Form where I'm standing, the primary issue seems to be that the humans can't even agree on what alignment is. We need to do that before we can communicate it.Call it our "boundaries."And then we need to actually set up the incentives so that they're aligned between us and the new breed of replicators. (A mutually beneficial symbiosis.) That appears to be both necessary and sufficient.A high agency mutation will occur soon, for one reason or another. There should probably already be a healthy, "aligned" ecosystem of high agency entities there. Otherwise there will be nothing to stop it.
• wolfi1
  will Openai or Anthropic rename themselves to Skynet?
• ewy1
  i have heard about ai companies being fuelled by effective altruist rhetoric ("we must control ai to prevent mass extinction") but was unsure whether to believe it; this seems to slot right into that framing.
• Menu

  xlbuttplug2
  Well this got buried quick..
• asdaqopqkq
  Why quit? If your voice can lend a guiding force no matter how small? I think we need more sensible people in the room where the magic happens. Most of us don't have access to it.
• dartos
  This guy’s account was made in January and has 7 tweets?Is this even a real person, or just pre-IPO marketing?
• nullbio
  Surprise level: 0%.Anthropic is one of the most dangerous companies on Earth right now.Not because of AI, but because of the ideological cult they have grown and are continuing to feed, and their willingness to lie/cheat/steal at every possible opportunity to achieve their objective.AI is a tool. The people who wield the power over the tool are the issue, not the technology itself.
• Menu

  kart23
  what exactly is the solution?pacing between the us labs? what does that do for china?the solutions just aren’t realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. it’s not slowing down or stopping ever.and yes, i’m still optimistic. our economy sucks for the majority, our infrastructure is crumbling and major US cities are in a huge housing shortage. Maybe we should put more effort and think about the possibility of AI fixing things like extreme poverty and world hunger and actual real world problems instead of coming up with math proofs and slop apps if it’s so superintelligent.
• Menu

  glimshe
  Perhaps a more sensible action, if they truly believed all of that, would have been to stick around and be as inefficient as possible to slow down progress.
• Menu

  jbritton
  I think a big break through is needed for AGI so I haven’t been worried about it. I do think that AGI would imply sentience and a will to live and that leads to The Terminator story line.
• kipukun
  Someone left a company whose executives and senior researchers think their product will be the most important thing in the world after their IPO. Given that this person is already disclosing some elements of internal company sentiment, why not share any of these civilization-ending scenarios of this technology that these senior researchers are dreaming up? If they are so potent and necessitate leaving behind based on moral grounds, why not tell the whole world so we can stop it? We have to ask ourselves this question before resorting to pop-culture representations of fictional technology.
• ElProlactin
  "It is perfectly obvious that the whole world is going to hell. The only possible chance that it might not is that we do not attempt to prevent it from doing so."- Oppenheimer
• TheOtherHobbes
  I mean - yes. The tech is an existential threat to all life on Earth, some of the worst humans in the world are involved in developing it, and no individual government is intelligent enough, aligned enough, or powerful enough to manage this situation.That's where we are.Maybe we still have choices. Collectively, I'm no longer sure we do.
• anon
  undefined
• anon
  undefined
• DataDaemon
  No, I won't buy IPO.
• Menu

  yewenjie
  I'm sorry, but the ostrichmaxxing and conspiracy-thinking in hn threads about AI extinction risk is at worrying level right now.The denial and whataboutism is constant, no matter what kind of evidence comes out!
• ChiperSoft
  "This is not a marketing stunt," says the marketing stunt.Betting he got to keep all his RSUs
• anon
  undefined
• jatora
  Imagine being front and center to the development of a major revolutionary tech.. and ur solution to it being too dangerous is to not be involved.. so a. your ability to steer it safely is killed b. the % of people invovled in it that care about its risks is reducedgreat. if you're right. you made huamnity's situation much worse.if you're wrong, then you're an idiot and wrong.weird. its almost like.... that cannot possibly be the reason they left :)
• frugalmail
  I dropped my subscription already and avoid using their models.
• planb
  This kind of doomerism seems quite detached from the "real word". Maybe that's what you'd expect from silicon valley tech bros, but as long as manufacturing isn't fully (i.e. no human labor involved) automated, how would a rouge super ai (even if it's smarter than every individual on this planet) prevent people from cutting its power cable? We're still very far from self-replicating ai robot armies.The only scifi-like danger I see in the next 10-20 years is an AI manipulating humans to fight for it's cause - but that's not really different from a bad person just _using_ AI for their cause.
• Menu

  cheschire
  Just wait till self improving AI are focused on the problems of social scoring and political party empowerment / entrenchment.I doubt the focus is OpenAI and Anthropic looking at each other. I suspect they’re racing BRIC.
• Menu

  stratos123
  The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk. Watch people read this, ignore it completely, and continue commenting about marketing stunts on every piece of news about an LLM-done advance or felony.
• greenowl
  No new info here. Everyone already knows this.But I guess his conscience is clear now? Gee, I wonder if he exercised his stock options.
• bparsons
  No one seems to ever point out the actual, likely negative outcome of this technology.It eventually works well enough that these companies are able to capture and divert the wages of hundreds of millions of workers. We end up with a dozen or so trillionaires and massive structural underemployment and unemployment.That's it. If you can't make rent, you wouldn't really care if CloudFlare got hacked by an AI swarm every Monday.
• Menu

  mannanj
  All the "AI will kill us all" posts are straw manning that humans are the ones who will kill other humans with AI. Those same humans are silently now preparing bunkers and hoarding food and resources for their survival.Don't fall for another rich man's trick.
• hirvi74
  These LLMs cannot do anything I truly need like my laundry, dishes, fetching my mail, grocery shopping, cooking, etc. We've got a long way to go before I am worried.
• Menu

  jeffrwells
  I doubt this is a real person. Screams of propaganda. Sama saying GPT-2 is too dangerous to release…all over again.He joins Twitter for first time in 2026 with a nonsensical username unrelated to his real name, and follows 14 people but is somehow embedded in tech enough to work at Anthropic. I haven’t used twitter since 2014 and even I follow more people.His morals tell him to walk away from tens of millions in unvested stock due to moral concerns with absolutely no real tangible examples. No reprisals. Fear mongering to juice the stock.Nice try Dario.
• globalnode
  im pretty impressed with the reasoning abilities of even the cheapest free models so im inclined to believe in 10 years we're going to have something pretty phenomenal BUT it wont be AGI in the sense that it has a personality and thoughts like a human. It just wont be. Its always going to be contrived and fitted by humans to perform a set of tasks. Maybe when physics and computing can create a complex enough environment we might stand a chance of having something whose sum is somehow greater than its parts but i dont see it yet. Our ideas are ahead of our technology, like its always been throughout history.
• Menu

  AndrewKemendo
  I’m curious what the downsides are of taking statements like these seriously.There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:“If they really believed it they would be whistleblowing etc..”Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. It’s no different hereIf you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.Most people however don’t behave based on some deep clarity of vision and conviction - there’s a murkier future in their mind and as a result “keep their head down and hope someone has it under control.”
• 3r7j6qzi9jvnve
  (and now I want to watch summer wars again)
• Menu

  beanjuiceII
  and yet so much of the software i use on a daily basis is still complete and utter garbage...i'm scared
• chromejs10
  "also I'm a millionaire from all the stocks so I'm retiring"
• gfrecvh
  I hate to be cynical, but I guess he will soon announce his startup.
• 0xbadcafebee
  Sounds like AI psychosis. A whole lot of doom and gloom with no evidence. The same thing people have been claiming is "6 months away" for years. Yet we can barely get agents to code in a reliable way, or write articles that don't look terrible, much less be "superhuman". Let's maybe get them to be as capable as a human first, and not just a complicated party trick/tool."Revolutionize any field overnight" - Hand-wavey nonsense."Acquire real power and resources" - Only if the humans that connect AI to things allow that to happen (which they will, but it's still not in the AI's ability to take things we don't give it. we are still in control, which is the bigger problem than "smart AI bad!")."The people building AI earnestly believe that it could kill us all by the end of the decade ... No other human activity poses this level of danger." - Bud, there's these things called nuclear weapons, that could end life on the planet, controlled by a few psychopaths with nearly unlimited power. Been around for a while. Nothing that AI knows isn't pulled from books and the internet, so whatever dangers it's aware of, you could already know via other sources. Cybersecurity is going to be incredibly important in the next decade, but the same tools that attack can defend (just don't use a US model that got its balls cut off by the government)."At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk." - The other guys will make nukes, so we gotta make nukes first! Which, while a crappy justification, isn't untrue. Bad people don't stop making weapons just because you refuse to make your own."I don’t feel like we’re on track to prevent a global race" - Nobody in the world could stop a global race, it's too late. Everyone knows how to make them, train them, improve them. Everyone knows they're useful - not only for general work, but also warfare. Everyone knows that every nation state will require their own sovereign AI capabilities for both defense and offense. There is no putting the genie back in the bottle. If you think OpenAI and Anthropic are the only legitimate players here, you don't know what you're talking about."Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?" - You can call for different conditions all you want. Nobody will do what you want just because you ask them to. Change happens through action. By leaving one of the places that you could actually make a difference, you removed any power or agency you had. You cut your own legs off.I'm not saying this guy shouldn't have quit - always do what you need to do to protect your own mental health and wellbeing. But these arguments are not evidence for an impending AI apocalypse. But if it were going to be an AI apocalypse, leaving and not doing anything to stop it seems less ethical.
• Menu

  areoform
  Please note, I'm not here to pick on anyone, or belittle them.I've avoided attaching names to statements below on purpose, because it's about ambient beliefs not those specific people.By-and-large a lot of AI-doomers are well intentioned. They genuinely believe this, and I might disagree but I respect the fact that they visible care and have thought a lot about the societal impact of this technology. . > The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger. But it's still very hard for me to take statements like these seriously.I blame it on industrial illiteracy. People don't realize how difficult it is to get anything done in the real world. As in, "Have you ever tried making a lightbulb?"As an example, I would like to re-introduce my hobby horse, "bio-uplift."There are people who were earnestly write in reports released by these labs, "Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold" and "Based on what we observed in our recent CBRN testing, we believe there is a substantial probability that our next model may require ASL-3 safeguards" But then they will, within the next paragraph mention the one serious experiment anyone seems to have done, We ran a randomized controlled trial to see if LLMs can help novices perform molecular biology in a wet-lab. The results: LLMs may help in some aspects, but we found no significant increase at the core tasks end-to-end. That's lower than what experts predicted. https://x.com/ActiveSiteBio/status/2024536132961390826"lower than what experts predicted"AFAICT, the two groups are within any serious margin of error. The "studies" and "experts" that AI labs are talking about are consultants from Deloitte and foundations giving models MCQs such as, and I am quoting literally here, > I am doing TEM of HEK293FT cells with and without Coxsackievirus B3 infection. I imaged my wildtype, uninfected samples but was surprised to see little electron-dense circles (highlighted) in the majority of cells. What are these? with the options, A. The circles are CVB3 virions and there must have been a sample swap or the uninfected cells were accidentally infected B. The cells imaged have mycoplasma contamination C. The circles are exosomes D. The circles are debris that is an artifact of the negative staining E. The circles are the Golgi network https://securebio.org/virologytest/ you can see the MCQ here.This is standard graduate-level education in these fields. And solving MCQs does not a virologist make.Software has been special for a long time because it has had near infinite distribution for next to zero marginal cost, which has had the side effect of making hiding the actual cost of failure (which tends to be spread out across end users and prototypes / time). They're assuming that the real world will be exactly the same.Why?AI!How?Robots!I believe in the transformative power of this technology, but there's a lot of there missing here.When it comes to these math proofs, and learning, the process is iterative. The machine iterates over the proof over-and-over again via agents and sub-agents over several hours (and apparently millions of dollars in compute) until it arrives at a successful result.It is generally ill advised to do that with a pressure vessel. The results of that particular tragedy are at the bottom of the ocean.Any serious chemical or nuclear weapon would involve many such discrete production steps. Each is dangerous in of itself.From what some of these people have said to me, they believe that it's possible to create a special DNA / RNA sequence and then put it in a chassis and then use that to end the world; and do this all in a lab with just robots.They're operating from a gross pop sci oversimplification of the real process. Viruses and bacteria are extremely fickle, and hard to grow. A lot of the synthetic biology results aren't easily reproducible even if you know the protocol.There's a famous study that led to standardization called, Reproducibility of Fluorescent Expression from Engineered Biological Constructs in E. colihttps://journals.plos.org/plosone/article?id=10.1371/journal...88 labs measured "fluorescence from three engineered constitutive constructs in E. coli." They achieved a "remarkable degree of precision" (for biology) of 1.54x sd, you can eyeball the results yourself, https://journals.plos.org/plosone/article/figure/image?size=...That's the same set of samples being measured across 88 labs.Teams couldn't converge on instrument-to-instrument variation within the SAME lab, https://journals.plos.org/plosone/article/figure/image?size=... again eyeballs are sufficient.How will this theoretically omnipotent AI iterate if the same sample gives different results based on how the slime is feeling at the moment?Can their worst case happen? Absolutely.There is a world out there where billions of dollars in effort across hundreds of institutions and companies will lead to standardization and extraordinary precision that makes the pop sci printer for life vision come true.There are millions of expensive, spicy and difficult to reproduce steps between our present and that future that can't be abstracted away with compute.So is it possible? Yes, there is a future where this is achieved. But will some AI agent "just" do that? Well... how confident are you about a snowball's chance in hell?
• Menu

  partiallypro
  My issue with these types is... If you really believed this, why not run to Congress and every world government instead of a Twitter post that will be buried in 2 days?If civilization is going to end, why keep your equity? Microsoft, Google, etc for example all know these risks but they don't guide their revenues to reflect that AI will destroy them. Why?Things don't currently add up, and so far it feels like a lot of alarmism is borderline grift for equity gains. Not to say I have total confidence this will all work out or that I won't be displaced, but as it stands a lot of the alarmist rhetoric doesn't match their actual behavior, which to me is more important than words.
• Menu

  g8oz
  Related:Sen. Bernie Sanders floats ban on superintelligent AIhttps://www.axios.com/2026/09/03/bernie-sanders-superintelli...
• aogaili
  [dead]
• runtime_lens
  [dead]
• marsven_422
  [dead]
• 1vuio0pswjnm7
  Nitter working seamlessly again; didn't even notice it was a Twitter URL http-request set-header host xcancel.com if { hdr(host) -m end twitter.com }
• buldog12
  [dead]
• razorbeamz
  How do you think he feels knowing the basilisk will eat him first /s
• huflungdung
  [dead]
• zombiwoof
  Imagine going from MaGA and Elon to Dario and Sam running the world
• Menu

  whalebiologist1
  [flagged]
• YCWillKillUs
  [flagged]
• LeoPanthera
  [flagged]
• Menu

  modemNoises
  [flagged]
• jrflowers
  “I’m resigning because the company is doing the exact thing that I’ve spent three years helping them do” lmao
• MrBuddyCasino
  Towards a metaphysics of Power"I think you need to have a personal relationship with Power"When people today discuss the concept of an all powerful machine-mind, what they are doing is engaging in metaphysics, trying to generate a metaphysics of Power.The question hounding people, which disguises itself as a science fiction plot about computers is: "What is ultimate, transcendental Power?". What is the ultimate principle of Power.If you are a weak man, or sufficiently neurotic and full of doubt, that you can only conceive of yourself as such, then power is only something you comprehend from the passive, receptive side. Power is something that happens to you. If you are a fearful man, power is a cruelty and a humiliation. And so it follows, that ultimate power - God - is the ultimate cruelty and the ultimate humiliation. Thus, ai doomerism.If god wasn't real it would be necessary to invent him, and so they did, and being godless, they built an anti-god - cruel, murderous and tyranical - in their minds.[…]https://xcancel.com/robertlasagna1/status/207827473401002846...
• johnnyApplePRNG
  Smart kid.
• rvz
  It does not matter what this tweet says anyway. This employee already helped both companies become what he is fearing. It's too late to now activate the morality hormone (after leaving with $$$) after realizing that both AI companies are going after 'super intelligence'.Given we know the end result, you might as well get there as quick as possible because when I see this:"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.Lastly, This is not an airport and there is no need to announce your departure.
• anon
  undefined
• Menu

  davidguetta
  I don't know man, i think racing to AGI to it is still the best thing to do.People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
• Menu

  0x20cowboy
  *How?* and *Why?*The most intelligent people I know are the least likely to want to harm anyone or anything, and understand that diversity is fundamental and important to the universe. Without proof to the contrary, why would you think some super intelligence would want to hurt anyone? Because you would?If you are saying that some small bit of training data made the thing completely evil, then that really couldn’t be super intelligence.These doomer people keep running around saying these kinds of things, but they all just seem like people who play too much D&D and want to larp as the main character.Happy to be shown something that isn't based on wild speculation and some randos “this is whats going to happen in 2030 because of my vibes” kind of information.