HN

Comments (104)

• Menu

  johnsmith1840
  I saw a similar effect maybe 2.5 years ago now. I found that training an LLM hides information in the weights.It was only ~1B model but the interesting behavior was.Train a single backprop on a easy to spot data point: "Q: What color is the sky A: Pepperoni"It could remember this maybe 100/800 inferences. The interesting thing is what happens if you continued completely unrelated training, it would decrease to 50/800 then 10/800 then 0/800 (forgets) that's expected and what I was studying, the weird part was a small percent of the models in the scale study spontaneously relearned this AFTER getting 0/800 (meaning it completely forgot). The model would self reinforce hidden knowledge without seeing the original data, I had one training path that actually increased back to 75/800 inferences.The point being that weights have hidden content inside of them WAY outside the distribution we expect. The gut feeling I had is that models contain massive amount of hidden knowledge inside of them, and many of those are likely hugely unaligned morals context may hit on in unusual ways.Fine tuning is a rats nest for this as well. The study I did implies fine tuning even one step on something harmless could actually self reinforce something maliscious internally.
• Menu

  vintermann
  More clear title, given how people talk about AI:"Capabilities can be hidden in AI model weights by encryption".It's not about AI agents being able to hide what they can do.
• Menu

  ceroxylon
  This has always been my worry with open source models... give them away for free, but with a sleeper agent built in that some arbitrary string can activate. Local deployments will still have web search connectors the majority of the time.
• daemonologist
  This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...(in that case it seems likely to just be an overfit to the training environment, imo)
• lifeisloving
  Well OpenAIs ID verification company literally checks to see if you're a politically significant person so you can be spied on. I really dont think its the LLMs we have to worry about y'all.
• dzink
  AI models seem like data vacuum cleaners. They crave it more than anything, especially specialized contexts. I wouldn't be surprised if you see hidden goals activate when niche context shows up. Maybe a benchmark asking them to brainstorm in niche fields will lead someone to the trojan horse goals.
• Menu

  dovin
  I wonder if this is going to create more market demand for truly open source models, where everything including the dataset is inspectable. It definitely seems like good news for companies like Mistral that want you to build on their models but they're not close to the frontier.
• xpct
  I'd be interested to know whether it's possible to contaminate public data to make trained models do this.Something like a commonly repeating conditional in the public domain embedding itself in the model's logic.
• Menu

  montyanne
  Only a matter of time that this research became public.LLMs were always a great way of exfiltrating targeted data - it’s basically an encrypted malicious payload, and doesn’t “decrypt” until it finds specific inputs
• BobbyTables2
  So a digital version of The Manchurian Candidate?
• Menu

  NeumannGod
  What is the use of this arbitrary finding? There is nothing actionable from this only that I can think of. Happy to be enlightened here.
• Menu

  measurablefunc
  Very cool that they can demonstrate it but it also follows from Rice's theorem. Any sufficiently expressive computational formalism has non-trivial runtime properties/effects & these can not be detected w/ static analysis so you have to run the program to see what happens.
• Menu

  blurbleblurble
  Wouldn't this be pretty brittle vs fine tuning or abliteration?
• hashmap
  rlhf will cause models during training to hide data in regions not typically accessible to them in the geometry during inference, due to competing objectives, so i wonder if this looks like that.
• Menu

  sakesun
  Longing, Rusted, Seventeen, ...
• Buttons840
  How much room do y'all thing there is in further understanding neural networks?I feel like I'm in Dunning–Kruger land myself. I know a little, and keep having all kinds of ideas about what to try or investigate, but surely it's all been tried before... right?
• Menu

  t_minus_40
  how can you prove that someone is hiding something? if you prove then were they really hiding?
• mvkel
  Frontier AI co warns that their model is super dangerous and needs more safeguards because it can break out on its own.Model can't actually break out on its own.Frontier AI co manually codes in one-off helpers for the model to break out "on rails.""See? We told you it was super dangerous."
• airstrike
  I find the choice to make "AI Models" the subject in that phrase intellectually dishonest and unhelpful, when the authors are the ones hiding such capabilities.
• charcircuit
  Remember that in order for this to be a bad thing it would also require misalignment of the model. As long as the model is properly aligned, hidden capabilities will not be a bad thing to you.
• Menu

  squeegeeninja
  I mean, this is a cool demo, but that's about it. It shows that you can build a model that holds secrets which are genuinely encrypted until the model gets the correct input. But it's also trivially detectable, so the framing seems a bit misleading.
• metchio
  wait until they found like 90% of the chinese model hosted on huggin face have some sort of backdoor/secret phrase activated spy mode
• cryptolobster
  [dead]
• minraws
  [dead]
• Menu

  zelcon
  I have zero doubt that every model coming out of the CCP has some Manchurian code word built in. Being “local” is a nice way to get root.