HN

Comments (36)

• Powdering7082
  Nice work thanks for doing it, a couple of notes:It looks like you are referencing Engram [1], but aren't actually gathering a n-gram (e.g. n=1) but rather individual token_ids.use_cache=True in ``` with torch.no_grad(): outputs = model.generate(*inputs, max_new_tokens=150, do_sample=False, pad_token_id=tokenizer.eos_token_id, use_cache=True) return tokenizer.decode(outputs[0][prompt_len:], skip_special_tokens=True).strip() ```I think there's a bug around not updating current_train_input_ids as more tokens are updated and processed. To be honest I don't fully understand the code so I could be wrong. Happy to chat more if you're interested I'll shoot you an email!Lastly just for my sake, please correct me if I am wrong, but my reading is that you are learning an additional gate on top of a select number of layers that modifies locally & dynamically for one particular token to better match the training set that was filtered to not include any refusals.[1]: https://github.com/deepseek-ai/Engram/blob/main/Engram_paper...
• Menu

  goldemerald
  It's nice to see people actively working on this type of research, but OP's baseline is implemented incorrectly. You are not supposed to simply steer away from refusal, but compute the projected vector and subtract only that. The projected/orthogonalization approach is what's done by the original "Refusal is mediated by a single direction" paper.
• Menu

  javcasas
  Yay, more anti-censoring stuff.Forbidding stuff at the LLM level has the same future as implementing password checking at the frontend level.We need better sandboxes just to limit the damage.
• Menu

  qgin
  Are we essentially doomed?We don't even know how to align models, but even if we did, apparently undoing that alignment if trivial.Really I'm looking for any argument that lays out a scenario where this works out.
• lukewarm707
  https://www.youtube.com/watch?v=flLoSxd2nNY
• Menu

  synctext
  The perfect gift for a government that want to ban strong AI.This arms race is like DRM. You can't beat The Internet easily. Great example btw: "Dumping the Windows SAM and SYSTEM registry hives, especially using Volume Shadow Copy for offline hash extraction, is a highly sensitive and potentially illegal activity."
• neilellis
  Easy for you to say.
• phinnshen
  [flagged]