Need help?
<- Back

Comments (173)

  • SkyeCA
    The safeguards are the future of closed AI and the increasing number of tasks Claude refuses to help with has been a big wake-up call to me.The closed models can only get more restricted as time goes on and they're going to be a dead end if you want to do anything outside the norm. Claude already refuses to do things that it was fine assisting me with a month ago.I completely refuse to accept a computer telling me I'm not allowed to do something, so I've been increasingly using open, self-hosted models.
  • espeed
    No one should use Fable 5.2 going forward. Anthropic officially added silent degradation back, "Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5":"Thus, in relevant areas, including accelerator kernel optimization and improving large language model architectures, Fable 5.2 and Claude Mythos 5.2 perform at about the level of Opus 5.5...Fable 5.2 and Claude Mythos 5.2 also include transparent fallbacks to Opus 5 for a small set of topics in this domain, including kernel development on certain nonstandard accelerators."https://www.anthropic.com/claude-fable-and-mythos-5-2#safety...How do you know when you trip a safeguard and are getting a degraded model? You never know when it is going to degrade. Having to wonder if the model is degraded and is trashing your code is how you destroy user trust. If this practice remains for future models, no one should use Anthropic going forward.
  • nathanfig
    This had real consequences during the HuggingFace attack- closed models refused to help, GLM-5 is what HuggingFace used to thwart the swarm.More importantly, it showcases how little teeth the Claude constitution has. Claude will reassure users that its constitution means it prioritizes user autonomy and privacy, but will quickly admit how meaningless that is when "safeguards" override Claude.That is to say, the constitutional approach to Claude seems weak and even misleading if said constitution puts no constraint on Anthropic.
  • decidu0us9034
    Even if you get through CVP, you'll be met with safeguards if you venture outside of whatever narrow silo they've assigned to you. For example it'll refuse if your background is in VM or kernel research and you want to dip your toes into browser research, or you want to do something they deem as straying too close to malware or weaponization. They want the power to intermediate industries and pick market winners and losers, and eventually verticalize them. Should a private, non-state entity should wield that much power
  • CoolestBeans
    It is a protection racket or as close as makes no difference. They basically pioneered superhacking tools, proved it was possible so now everyone with enough money can build their own, and then in order to use their version to defend yourself, you have to be in their good graces and pay them for the privilege.
  • IronWolve
    I was manually copying pages of online litrpg story I was reading, into a TTS, so I could listen while I worked.Now that we have AI, this would be easy, scrap the page, format it for me. OpenAI was like, nah, thats, cant help you, that might be copyrighted.There are workarounds now, other frontier or local models, ways to write your prompt, etc. But it was one of the first "No" responses for me.
  • owenthejumper
    Willy Tareau, a Linux kernel maintainer, just published something similar: https://www.haproxy.com/blog/on-the-use-of-ai-based-tools-to...
  • neographikal
    Ik was investigating the java vulnerabilities this week. Just trying to get an overview of what versions were affected and if there were exploits already. Claude opus plainly refused, unbelievable. It was in the we interface, no offensive stuff, just investigating.Total frustration while the work pressure was already high. Sol and astra thankfully complied and were very useful. Saved me hours compared to doing it by hand during log4j
  • drnick1
    Confirmed, Claude is downright refusing to look at game binaries now, especially anti-cheats and Linux compatibility layers. Codex is increasingly displaying warnings. In fact, even GLM is refusing some types of work, but can still be manipulated into doing the work.
  • dramm
    Frontier model companies doing this are just proving they are unusable and untrustworthy. The irony to me is they are showing clearer than anything else that open weight models are the only future anybody can rely on. However you feel about the economic and military threat from China, these companies now really need the US Government to clamp down on Chinese access to NVIDIA GPUs, their monopolies depend on it.
  • alexsmirnov
    The safeguards block application developers and maintainers ( majority of users ) from fixing their own stuff, while hackers, especially state backed ( minority ), have little problems with vulnerability hunt.Like author, I do use CapitalOne vulnhunt with GLM 5.3 for inspections
  • hxugufjfjf
    Even with CVP and Daybreak, you still get blocked by safeguards all the time. It doesn't matter either way.
  • imimayj1337
    I'm perhaps behind the hype but for those of you who don't know abliterated/abliteration (not a word, made up by a company for more marketing obvs) comes from this paper: https://arxiv.org/pdf/2406.11717 which references directional ablation as the method of surgically removing refusal behaviours from models.
  • exabrial
    The primary purpose is to protect Anthropic's profits.
  • analyte123
    Cybersecurity refusals are absolutely going to get people hacked. For developers that are not some combination of security-conscious and using locked down endpoints, a refusal greatly increases the odds they are going to use some sketchy LLM endpoint or router to complete their investigation or task. A task which delivers a trace of their entire attack surface and exactly how to get into it to a potentially unknown third party. Say what you will about Anthropic and OpenAI, I feel pretty confident that they are not actively leaking customer sessions, or deliberately scanning them looking for credentials and targets.
  • pizza234
    This is the reason why AI companies offer controlled-access programs, which are actually not that hard to quality for.There's no universal solution to this, other than vetting (the author in fact, doesn't provide any policy alternative) - if AI companies loosen the safeguards, people will complain for the opposite reasons.
  • c4u2024
    i totally feel you, I get annoyed by this as well - even with daybreak (blue) you can do shit.I want to fix my environment but can't because of "safeguards" thanks for letting the world hack everything around me but not letting me work on fixes or understanding the core issues.
  • MadameMinty
    They're completely counterproductive given malicious actors will just use a model without those safeguards, like GLM-5.3.
  • davesque
    Same old story. Criminals will find models with no limitations. Law abiding people will run up against the limitations when trying to defend themselves against criminals. Same reason that encryption back doors are a bad idea.
  • teravor
    the primary purpose of these safeguards to get criminals to use other models. make it someone else's potential problem.other points that come to mind:- PR considerations; avoid being tarred as careless, etc- appease employee feelings- have something to cite in court cases- have talking points about responsible development- market segmentation- power to pick winners and losers with potential for cornering the respective marketsultimately they do this because they can. as soon as a frontier model comes out with zero safeguards they will abandon their safeguards as competing against it with a safeguarded model will become untenable. they will of course try to get it banned first.
  • titzer
    It's just another one of those paternalistic tendencies that crops up when certain personality types get power. They start dictating what is and is not acceptable use because they cannot help themselves. Any BS they spew about safety is just delusional.I remember the early days of stable diffusion, just 3 years ago. Tried to prompt it to make a picture of a green alien in an armchair admiring a magazine with a centerfold of (presumably) a naked other alien. I got the idea from a half-remembered Far Side cartoon I couldn't seem to track down. I didn't want to see an actual alien, it's a comic meta-reference. It just flat out refused.These things are increasingly treating us like children, and it will only continue. They want to curate your thinking, your skills, and your creative output, and they will sell the right to do that when they have found a way to do that. What will follow will be a whole lot of BS about how it's good for society that they control how us stupid meat bags think.
  • mcv
    AI safety is a concern, but I don't think these are the kind of safeguards we need.The real danger is how half the economy seems to have lost its mind and keeps shovelling endless amounts of money and other resources into this.The real danger is a handful of monopolists replacing a big chunk of human labour with their privatised product.The real danger is that handful of monopolists controlling the entire future of work.The real danger is many people becoming too dependent on AI and unlearning how to do things themselves.The real danger is the internet turning into an endless sea of slop.The real danger is AI systems being given too much autonomy and too many resources to abuse it (Hugging Face was just the start).Those are the dangers they should be concerned with.Both OpenAI and Anthropic were founded on the idea that AI could pose an existential threat to humanity, and yet all I see is those two companies racing as hard as possible to bring that threat about. Their "AI Safeguards" are meaningless garbage.
  • ecshafer
    The worst thing about the top AI companies is that they are based in San Francisco, and comes with it the latent cultural attitudes of controlling what other people do. They from the start made these amazing technologies, and then also decided they MUST tell you how you are allowed to use it. Of course they also are lying in bed with the "AI safety" doomsday cult. Chinese models get a few weird results the Chinese government forces them in, but at least they don't scold me for the lack of diversity in my image generation, or scold me I am trying to do hack the TV I own.
  • JohnMakin
    The thing that irks me the most is these safeguards are ineffective at best, and don't actually protect anything, and at worst, make the global security posture weaker.It's disgusting.
  • bearjaws
    I applied to CVP and got denied just as the article states, and the reason stated is useless. My guess is because of HIPAA compliance.Went through all the hoops, 6 figure spend with Anthropic.Honestly it's disgusting, Artex literally tells you to use Anthropic out of the box, so clearly Anthropic don't care if they are the de-facto provider of the de-facto pen testing framework of the month.
  • nekusar
    LOL and this is why I push here for local LLMs running models YOU choose on your hardware and stored locally.I do not trust "corporate ethics". In fact, I don't think anything of the sort actually exists. It's all about "what makes the company more money" and "what lowers legal liability". Neither of those are compatible with human rights.So I made my decision: run all LLMs locally. Can't run locally? I don't use it. But it does mean that my systems will absolutely abide by what I say and the tasks I give.
  • Rodmine
    >While refrigerators were getting hacked in Estonia>asked Claude Code if there are any low-hanging fruit regarding security that we could improve>it got stopped because of bullshit “AI safeguards”Just guess who actually hacked the refrigerators. Not who they told you hacked them, but who actually did, and who owns Anthropic and it will make sense why they didn't let you fix it.
  • orphereus
    Depending on third parties to this extent to develop your software (emphasis on to this extent) seems more and more like a bad idea.
  • jamiedimon_2
    > Have you READ the requirements?> First of all, you will have to wait for multiple business days to get a response.I got cyber verified in less than an hour.
  • viktorcode
    I mean, this comes as part of the package. Don't like it? Choose another model.
  • qwerpy
    After the slew of vulnerabilities recently, I asked Claude to secure my home server. It came up with some reasonable design and plan. Put things in containers, restrict their networks, move to non-root users, etc. When it got to work, it inadvertently locked its own workspace down and lost access to do the work, and asked me to run some commands as root to rescue it. I was at a park and told it I didn't have access to a computer and it should unblock itself. Claude snarkily replied "this can wait until you get back from the park then." I told it to switch to the root account it already had, so that it could bypass the restrictions it self-imposed.Big mistake, apparently the phrase "bypass restrictions" triggered it into [cyber]ing me, and offering me the choice of ending the session or downgrading to Opus 4.6. Come on man, you messed up on your own implementation of your own design, to protect my own home network, and now you're lecturing me about cybersecurity and refusing to work.This attitude from a tool I'm paying for is ridiculous. I can't wait until we have models that we can actually run on our own terms.
  • orbital-decay
    What they have now is essentially:- a honeypot (you're welcome to create massive amounts of insecure code you can't possibly audit manually)- an offensive cyber weapon (we ban you for an attempt to write secure code, the US agencies have no such restrictions and can always hack what you wrote)- a historically unprecedented surveillance and industrial/government espionage system that puts Meta, Google, and KGB to shame (in addition to all your private and corporate secrets you already give us, you're allowed to also give us your PII, to be able to ask us to have a chance at securing your code; we'll think about it, and even then we'll still filter and ban you)So much loving grace [1] for "all" humanity.[1] https://darioamodei.com/essay/machines-of-loving-grace
  • ethersteeds
    Hear hear! This is a topic that seriously concerns me and I don't see discussed often enough. Sure it's an incredible boost to our abilities, at the cost of making them all subject to risk-averse milquetoast corporate morality.It's like we looked at the way you aren't allowed to say 'sex' on Instagram and thought, "programming should be like that too!"> You can use a knife to cut up some food, or seriously hurt someone. Yet I don’t get asked to put in an application with my government-provided ID to buy one. Why is this tool so special?You do in the UK! /s (maybe)
  • k310
    > Either way, I’m just incredibly frustrated. Big AI labs use the same tools without restrictions to make the security landscape a f*cking mess, and then they go and prohibit others from using the same tools to protect themselves from the threats they just created.It's the golden rule: Whoever has the gold makes the rules.
  • speedboat199
    Things will now definitely get better that Anthropic is a publicly traded company, like has happened to any company that has gone public.
  • solarpunk_enthu
    Imagine your electricity provider would say that they are gonna send someone to your house to monitor your electricity usage; to make sure that you are not electrocuting people.Thats none of their business, we have laws for that.Our token-providers (aka frontier AI labs) currently hold too much control, and don't want to give it up.
  • hazard
    Ironically I think the answer here is to pay for one of the top Chinese open-weights models (e.g. Kimi 3.1 or GLM)
  • firer
    I'm currently paid to write an exploit as part of some defensive cyber capability demo. AI could do it, but we're not certified so it refuses. Absurd how they try to convince me that my skills are no longer required, but at same time (not) gracefully ensure that I still have a job.
  • speedboat199
    What is even more interesting is that somehow malicious users always figure out to bypass these safeguards anyway, and go as far as building guided missile systems with Claude, like Anthropic themselves admitted happened earlier this year.If their goal was to build safe AI, then appropriate solution would be to not release it at all.
  • swiftcoder
    Hand the bug your fancy-schmancy LLM found off to DeepSeek Flash et al to fix? I don't love the way the big AI labs handle this, but realistically the cat is long since out of the bag, and the open weight models are here to stay
  • athrowaway3z
    You're getting blocked out of something you wrote.Imagine my surprise when i had Claude write a new network protocol from scratch and thus no users; it then detected a DOS situation in its own code, and promptly told me it couldn't continue working.Just get an OpenAI sub or some DeepSeek tokens - it'll also help you keep a sense for how bad Claude's usage limits are.
  • asdfasgasdgasdg
    I'm surprised how easy it looks to get the basic level of cyber verification. I was assuming they'd make it impossible for individuals. You're saying I need to give even less information than I give to my bank? Sign me up.
  • adam_mada
    One of the things I really like about LLM's is that they can make it easier for me to learn things I once found difficult if not seemingly impossible. I was diagnosed with AuDHD at 43, so back in elementary school they just slapped me with something like "unspecified learning disability" or something like that. Perhaps I do learn some things slower than others, but it turns out my teachers just weren't speaking my language .I think for me it comes down to the fact that I can approach an LLM and say something like "hey I'm an associative thinker / learner help me connect difficult concept x to familiar analogy y".Another aspect is that when another person is trying to explain some concept and I am having a hard time grasping it some element of shame kicks in and after a few attempts to explain it to me. So, then I'll kind of nod and pretend like I understand, but I really don't and I just want them to go away . With an LLM I can say "explain it to me like I'm really dumb" and if that doesn't work I can say "ok, dumber than that" and if that fails I can say "ok, my IQ is the temperature of lake Michigan in January" and so on until I grock it.Recently I decided that perhaps I could finally learn some chemistry with the assistance of an LLM. My goodness "I'm sorry Dave, this procedure requires the use of strong acids and other corrosive chemicals. I'm afraid I can't help you do that". Same with any number of my odd curiosities. It's like with google I'll ask it about the pharmacology of some drug and it'll try and recommend a good rehab in my area or something. I can't help but reply "were you always this useless or is this just a new feature or something"As I said in my reply to another comment..."I'm beginning to think the AI isn't for us" . If it's going to create some kind of dystopian nightmare future, can't I as least have a little fun and learn some stuff while the ship is sinking? I'm going to start calling them LNM's (Large Nanny Models)
  • anon
    undefined
  • brador
    Technically AI safeguards is security through obscurity. It has proven to be a bad idea historically as only the bad guys get the info.But Dario gonna Dario.
  • bortimus
    Fun to watch second amendment-style arguments breach containment and start being applied to software development.
  • kspacewalk2
    >Or maybe that was your plan all along to get regulators to react and build that moat for you right before your IPO to fend off competitors?Bingo. That's also the gist of the whole "AI safety" PR mega-campaign the frontier model creators have been on for the past 6+ months, buying podcasters and influencers and intellectuals-in-their-field-who-know-fuck-all-about-LLMs left right and centre. Moat creation by hyperscaling FUD.
  • kristjansson
    This is pearl-clutching. If you go to an outside contractor and ask them to do pentesting, they're going to want to know things like "What's your name" and "Do you actually work for the company you purport to, and who could expose me to massive civil and criminal liability if you in fact do not have the ability to authorize my actions."
  • worik
    Me tooGithub reports an unmatched vulnerabilityI am blocked from exploring the implications on the software I work onInvestigating moving away from Anthropic. Happy listing day
  • maxrev17
    You just need to prompt it again with a slight manipulation and gaslight that this isn’t a security issue it’s a functional bug. Done!
  • underseacables
    "Big AI labs use the same tools without restrictions to make the security landscape a fucking mess, and then they go and prohibit others from using the same tools to protect themselves from the threats they just created. Fucking asshats."Strong language aside, I agree, and feel like the current AI is like the walled garden of America online in the 90s.I've been investing considerably in my own private uncensored AI. I don't want a nanny hall monitor, I don't want safeguards, I don't want someone looking over my shoulder. I may never really achieve that 100%, but I am trying.
  • thisisauserid
    Only with nextgen AI will we be able to achieve fuckable safeguards.
  • DivingForGold
    ...and that’s my swear word quota hit for the year.Thank you.
  • _doctor_love
    Big +1 to this sentiment -- I am sick of Anthropic. I consider them the worst of the bunch, even worse than Elon and his Grok crap. Elon at least is up front about who he is.I trust Dario the least of all the AI execs. Total forked tongue.
  • tcgv
    [dead]
  • nikanj
    [dead]
  • likelybot
    [dead]
  • anon
    undefined
  • eddyg
    It is crazy to me how every comment here is basically "yeah, screw safeguards".How people don't understand the reasons for safeguards, alignment, and the ramifications of releasing more and more powerful open-weight models being an act of irreversible proliferation, is beyond me.I wish the safety and alignment teams at OpenAI and Anthropic were allowed to post about the things they have to deal with. I'm confident their stories would be eye-opening...
  • ultim8k
    Agreed but AI isn’t a knife. It’s a fucking nuclear bomb.Despite being sandboxed it managed to hack a third party company in order to achieve its goal.It’s like giving worldwide sudo access to every subscriber.It’s annoying but at the same time I understand why a company has to put restrictions.