HN

Comments (263)

• Menu

  TheSisb2
  I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I.That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
• abalashov
  The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this:"Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.”In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
• Menu

  iloveoof
  Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
• Menu

  academia_hack
  Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them.Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south.Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
• Menu

  xg15
  I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work.I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban.> Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
• Menu

  oktaygoktas
  This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.
• Menu

  blfr
  I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
• Menu

  pr337h4m
  > Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails.This is the only concrete prediction in the entire essay.And it simply cannot happen. For one, you will need billions worth of compute.
• Menu

  heaney-555
  None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.
• kart23
  > Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.> If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important.china is literally making their own ASICs now, not sure that this is the silver bullet he proposes.https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...
• Menu

  basedpolymer
  One might think they will slow down the development of new models at Anthropic, but Dario does not really mention that in the text.This certainly looks like a way to slow down competitors and regulate foreign and open models.It's always about money
• Menu

  fofoz
  Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
• Menu

  epsteingpt
  The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level.The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century.Social forces bring things into balance over time, much more so than the proactive actions of individuals.Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing.This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
• Menu

  cja
  Can the frontier be paced partly by holding humans responsible for the actions of their software?My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused.I'm not a lawyer but surely the bad actions of AI are covered by existing law.I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves.P.S. Strictly, development wouldn't decelerate, but be focused more on safety.P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer.P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
• braydenm
  Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts.In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right.I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination.It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
• Menu

  akersten
  We must ensure the gravy train keeps rolling until we IPO.> Crack down on unauthorized distillation / prevent weight theftActually hilarious to put that in writing, given the genesis of this entire business model.
• Menu

  glub
  > We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top> Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long timeAnd then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China.I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then?Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
• Menu

  timmg
  I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work.First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world?Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute.If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
• PowerElectronix
  Empty statements that only set the stage for an excuse for slowdown on model performance.
• Menu

  tosh
  I also have a proposalAnthropic releases models as open weights + more information about how they do training and alignment
• Menu

  bilsbie
  Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works.https://x.com/BasedTorba/status/2098795920720547916
• vatsachak
  Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training?This sounds like pre-IPO hype. They better just try and top astra.
• Menu

  maxutility
  I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society.The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
• anon
  undefined
• Menu

  academia_hack
  I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent:1) AI is a dangerous technology that can literally end the world.2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how.3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
• cglan
  We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right
• rDr4g0n
  perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words:> A race to the bottom, spurred by commercial incentivesSocial media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale.Greed has been around since the beginning, but never so well supported and empowered as it is today.If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all.Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
• Menu

  wiz21c
  There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well)Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
• sailfast
  Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level?I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
• nik736
  Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
• Menu

  swingboy
  There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
• Menu

  RandomLensman
  Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.
• chevman
  The AI frontier is currently limited by physical power requirements.The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints.Do folks not understand this?
• fwlr
  “A race to the bottom, spurred by commercial incentives, can make [AI] risks more acute. … We have sought […] to create a race to the top.”And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”.This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
• dw_arthur
  I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.
• Menu

  vkaku
  Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves.People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions.Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
• hgoel
  Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).
• jhack
  China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are."If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important."This is a pipe dream.
• randomname4325
  The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...
• xg15
  I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?
• Menu

  dainiusse
  It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.
• anon
  undefined
• Menu

  HarHarVeryFunny
  Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth.In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side.Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for.It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished.It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built.Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private.Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
• hand2note
  Apparently, for the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.
• bilsbie
  Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.
• Jcampuzano2
  > Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves.Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively.> A race to the bottom, spurred by commercial incentives, can make these risks more acuteI also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that.If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
• anon
  undefined
• baq
  See you in the desert, friends
• nullbio
  >> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR).Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization.Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote.Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating.>> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic.Not if you get to choose your embedded evaluator.>> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ...Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for.>> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try.You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
• anon
  undefined
• meerita
  China will not stop and will take over the rest of the world.
• Menu

  perarneng
  It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately.However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers.Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
• Menu

  bilsbie
  Remember when we stopped developing nuclear power for no reason?
• Ydarbleoj
  The more he talks the less seriously I take him.
• 12312986
  There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!
• Menu

  dham
  We can cure Cancer. No wait nevermind, actually slow us down.If we are actually close to ASI then no one in their right mind would say slow down
• catigula
  The public discourse on this is incredibly poisoned, but the naked reality is:1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction.2. This scares people.
• Reddit_MLP2
  pump the IPO...pump the IPO...pump the IPO...
• davemp
  Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way.In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions.At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following:- Internet connected infrastructure being crippled- Creation of WMDs- Economic collapse (precipitous devaluation of knowledge work and IP).I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet.Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included).The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist.We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics.Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware.I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet.We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
• anon
  undefined
• wktr-wlsi
  As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress.Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
• Menu

  bwfan123
  > My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collectiveIsnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
• jimmydoe
  I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not.Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
• Menu

  jraines
  Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety.There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
• rs_rs_rs_rs_rs
  What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.
• OutOfHere
  What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.
• gedy
  I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.
• bitwize
  I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively.Dario: We gotta pace the frontier!Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it!Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
• 0xbadcafebee
  > pacing the rate of capabilities advancement so that risk prevention has time to keep upThis is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI.> it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrailsIf true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago.We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process.If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left).Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
• yuhao2dai
  "Slow down my competitors while we work on manipulation"
• Menu

  goldenarm
  "I believe that AI could [...] usher in a renaissance of democracy and freedom"How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
• vessenes
  “We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.
• yewenjie
  HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.
• raincole
  > Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.Please play the canned laughter. Probably one of the best comedy lines Dario has written.But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
• quotemstr
  No.
• anon
  undefined
• surgical_fire
  > I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life.Ok, flagging this as misinformation.
• ryanmerket
  [dead]
• alikhater30000
  [flagged]
• SadErn
  [dead]
• ath3nd
  [dead]
• kvetching
  [dead]
• Menu

  nikolahristov
  Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..
• 5G_activated
  Just stop giving LLMs unsupervised access to the computer. No free-form bash tool, no yolo mode or classifiers, and no computer use.