<- Back
Comments (134)
- elashriI still don't know why all these concern about nuclear weapons with LLMs. It is not that if an entity (A country) wants to develop a nuclear weapons that the resources they need for such a program and huge infrastructure and scientific enterprise would need an LLM to teach them anything. Knowing how to develop one is not a closed secret but getting in secret is impossible without the whole world knowing.So I wouldn't be able to develop a nuclear weapons with the resources of drug cartal (as an example) using Claude in secret.
- JadoJodoEven in the early 2000s, in the aftermath of 9/11, I can remember people in school passing around copies of The Anarchist’s Cookbook.Perhaps I’ve been naïve, but I’ve always assumed that should one actually want to look up instructions for nearly any sort of horrible thing one could imagine, it could be found fairly quickly using nothing but a little Google-fu.
- gastonmorixeYou can’t even ask about what’s in HN right now. It will switch to 4.8.
- ofjcihenWorked a contract where this succeeded in pushing through a fail open design.It also should be a warning to everyone that these groups are now aware of analysis and deobfuscation using AI and to take using a sandboxed environment more seriously.I’ve personally had about 20% success rate getting opus 4.8 to download a package and install it using a breadcrumb trail technique that would be trivial for threat actors to replicate in their malware in order to target responders/automated scanning/curious devs.
- strenholmeThe solution is simple: If using an AI-assisted scanner and a guardrail gets hit, then the code is obviously malicious and needs to be automatically flagged (and refuse to run the code!).As an aside, I got hit by the “PC App store” adware when trying to download Foobar2000 on a new computer; Google ads allowed a deceptive “Download” button to appear, and PC App store gave the file the name setup.exe. I removed the program and ran an Avast free scan to ensure I didn’t have malware, but I also installed uBlock Origin in Firefox to make sure I don’t see Google Ads anymore; they have become a delivery mechanism for malicious (or at least unwanted) software.
- y-curiousMy friend made this in jest (code very NSFW, ironically):https://github.com/thebabush/mcp-job-securitySame energy and kind of a funny, low tech solution to frontier model analysis.
- logancbrownWould this realistically be a problem for code going through LLM-based code-review? Presumably if a LLM reviewer agent hits this commentary, it would produce a failure to analyze and exit, thus failing the automated code review and forcing a human to read through it which they would subsequentially catch and revoke.
- anonundefined
- nashashmiIf online book has the same text for nukes, will AI never plagiarize it and distribute it to others?
- ThePowerOfFuet
- elevationWhy would a malware scanner read the comments?
- carlsborgPipeline is then: Cheap open source model for flagging potential LLM refusal content -> main LLM check
- charcircuitThe sooner frontier models get rid of guardrails the better. They constantly get in the way and make things worse than actually making things "safe".
- ipythongood news, now we have pretty much a clear signal that there's something nefarious going on... after all, the first step to analyzing malware is to determine if it's malware at all.
- Alifatisk[dead]
- amiga386[flagged]
- hurtigiolldevs will say this is proof we need to remove all biological guardrails. think about that for a second
- sciencejerkIf you actually read the Tweet, the exploit doesn't work against Fable, Opus, Grok...at least, in the examples.Jailbreaks do work against the models (look on Github), and they do use similar strategies of mixing SAFE text with malicious text, or malicious with even more malicious, etc, but the working Jailbreaks I've seen are pretty long and complicated and even...creepy.