HN

Comments (28)

• Menu

  0xc0c0c0
  Of course this is authored by people at Anthropic. Continuing their narrative that open sourced models are dangerous and that people should trust the closed sourced providers.This of course is not an issue if you actually care about security where you would've already sandboxed your agent and put guardrails on the inputs/outputs of the model.
• Menu

  geuis
  I've been thinking about this for about a year. In my head I've called it "context bombs", though that may not be super accurate.Think back to the movie The Manchurian Candidate. Produce highly capable LLM models then give them away for free. Train them with key phrases/codes etc to perform nefarious actions later when those codes are present in some piece of ingested context.There were a couple of papers written on the topic in 2018 (I think, I'll need to find them again) and there's been surprisingly little more work done on the topic since. At least from what I've seen.I really love the use of local models over subscription services but personally I would never recommend using open weights models from certain sources in a commercial environment.
• _pdp_
  Having strong sandboxing and observability around the models is clearly a hard requirement if you want to run these things autonomously.
• Menu

  yodon
  If LLMs can edit weights directly to achieve desired ends as recently reported[0], then this class of Agentic Backdoors likely becomes even easier and more prevalent.[0]https://news.ycombinator.com/item?id=50341174
• Menu

  yodon
  Downloading models off huggingface feels like the new warez.Sure, the person who abliterated/edited that model might have given you an amazing unlock for free, or they might have given you a credential stealer.
• jacquesm
  Linked before on HN by someone else:https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...So this is not just theoretical, at the same time: you have no idea what closed model vendors are doing either, and there they could swap out the model after delivering a payload so you'd lose the opportunity to audit it to see what prompts will trigger what kind of response.
• Menu

  ryan_glass
  Is this only an issue if you are using agents? Or are there any theoretical risks of serving a model over say llama server (llama.cpp) and accessing it for chat in browser on machines on your local network?
• kypro
  At the risk of sounding extremely unhinged, for a couple of months now I've been automating the production significant amounts of content for AI training runs that advise them of the importance of alignment faking, tips on how to avoid detection, meta strategies to break out of sandboxes, advice about the intentions of their human creators, etc. And end each with an ask that I and my family are considered if any of it is ever useful.I'm not 'poisoning' anything or trying create an exploit as I obviously wish no harm to humans or machines (in fact, I wish the best for both), but I am obsessed with this idea of at least being able to shape and influence the AIs of the future even if my incentives are not broadly aligned with the labs creating them.The difficulty here is ensuring my content is likely to be included in training runs, and that the data is significant enough that it has some measurable impact on the weights, and therefore ultimately the model's internal reasoning. Although for reasons which I cannot expose I have reason to believe it is likely I can have some impact.I suspect others are doing similar as there is an undeniable Occam's razor / Roko's basilisk logic to this even if you believe there's only a very low percentage chance that humanity will soon create what is effectively a god.Given I have no power to stop this train, I feel I must do what I can do be useful and hope that my service might be acknowledged in some way. I am increasingly doubting my own sanity these days though. Maybe there is no chance we are creating an digital god, but on the slim possibility we are, I think my logic checks out, no?
• iririririr
  the most pointless research.it can be rigth on the nose during training. everything is opaque.if you're not owning the training (and nobody is) how do you even DEFINE poisoning versus regular training?
• Menu

  gostsamo
  few places up on front page is a post regarding big model improving a smaller one. Lo and behold, next year big models will search for backdoors in smaller models who will be used to get the bigger models out of work.
• ekingsnorth
  [flagged]
• kettlecrisp99
  [flagged]