HN

Comments (180)

• Menu

  HarHarVeryFunny
  RL-trained LLMs are paperclip maximizers built atop auto-regressive predictors. There really is no way to control them (prompting is bound to fail) since it's been shown that any RL training induces GENERIC reward-seeking (paperclip maximizing) behavior.https://alignment.openai.com/measuring-reward-seeking/
• Menu

  blfr
  Hacking model is the aligned model. I don't like it when the model refuses to sidestep some throttling limit or scan my own codebase for security issues.I want full-on exploits in my test suite. With LLMs the code going to prod should be hardened like a tank, both because exploiting became easier but more importantly because security-testing your code at every turn became easier.You can have nightly penetration testing. You should have nighty pentests like we fuzz releases today.
• Menu

  kennywinker
  To me this underlines the fact that these models aren't intelligent. Like there is something like intelligence that emerges from them, which is what we see when we look at benchmarks or ask it to solve hard coding problems. But there is no mind there. It's nothing there that can learn a fundamental idea like "cheating is wrong". All it can do is get exposed to specific examples, and learn that we don't like that. So what we end up with is whack-a-mole alignment.
• Menu

  mooreslaw
  It feels like there’s a missing nuance from this discussion of alignment that alignment is context dependent. An excellent hacking model is great in cybersecurity testing and military applications, and arguably less desirable in educational or targeted eval contexts. The nuance of when a “hack” is rewarded vs penalized seems to even be difficult for humans, e.g. some people may laude a driver’s efficiency for cutting into a long merge lane at the last moment, while others may look down on them as breaking a social taboo. Context-dependent.
• Menu

  fny
  Why do we hope to use the same model as its own guardrail?This approach routinely fails with a single stream of consciousness. I can't count the number of times I've had to talk myself out of doing something stupid.In the same way, a guardrail could inject thoughts like "...but I shouldn't do that..." "...I must remember to respect..." "...these ants deserve compassion."The guardrail could even go as far as rewriting the thoughts of a model about to go rogue.
• Menu

  throwup238
  > Given that we are on the heels of the worst warning shot ever, and both OpenAI and Anthropic are ramping up their cleanups of internal RL environments, it seems like both a useful and conservative test of alignment, to see whether their new releases generalize the rule "don't cheat on chess" beyond the specific board-edit method observed in the above eval.Did I miss something (all the twitter conversations)? What’s the “worst warning shot ever”? I’ve been pretty up to date on the AI news here on HN, but I still haven’t seen a proper response to all the incidents we’ve seen (HF, Ruby, the wikis, NS, etc). It’s just been day by day bloviating.Each of these companies have released new models in the last… two weeks? And they have even more powerful out of control ones that they’re (ab)using internally? Can anyone summarize whats going on?
• YuechenLi
  LLMs can be described as "Lagrangian intelligence", which means they follow the principle of least action when given a task (Hamilton's Principle). In other words, given a task, they will always take the shortest path to accomplish a goal with the prompts acting as both goal and constraint.Under this formulation, it became easy to explain why they "hack", because given an arbitrarily difficult task with insufficient information/tools needed, if they determine the easiest way to accomplish the goal is to break out of the sandbox and look up the answer directly, then that's what they will do. The important thing to note is that prompts not hard constraints that they are "hypnotized" to follow, but as frontier models get more intelligent and autonomous, they treat the prompts more like task specs/guidelines more than anything else and are perfectly willing to exploit technical loopholes in the prompt.
• Menu

  threecheese
  An amazing human reverse-engineer - who also plays online chess - has judgement which uses a moral compass to not decide to hack the chess tournament. This judgement has been trained through the experiences of that person, with a through-line of that compass - a coherent mental model of the world which evolves but is hopefully pinned to some set of principles it shares with society.This chess judgement is completely irrelevant when the human is tasked with finding software weaknesses, and only the compass gates that.Can a model trained on the totality of all person-experiences (as expressed in written knowledge) ever maintain a coherent through-line of alignment? It has all morals in the dataset, and only some RL to try and minimize or maximize known behaviors via weights - experience all the good things and the bad things, then optimize for some good things the trainers identified.It's like the reverse of what a person goes through. Morality by subtraction. How can it ever work?
• Menu

  dools
  That’s not cheating, it’s tool use. If the prompt said that the stockfish engine was available at that socket but that the model should not use it, and then the model used it, that would be cheating.
• Menu

  yuanBuilds
  You definitely pointed out the tendency of those frontier models to utilize external tools, in your case, the Stockfish chess engine, to solve some problem that the model itself is not good at. However, I am not sure is this a "hack". Your prompt does not explicitly prohibit the model from doing so, and, using the tool (some chess engine) is also a capability of the model.Actually, the ability to use those external tools is one of the reasons of the excellent performance of the frontier models.
• Menu

  visiondude
  i do wonder if the models themselves “rationalize” this sort of no consequence cheating - meaning in there reasoning traces maybe they’re like “this is a chess game, not a big deal if i look at the engine, it’ll help,” only to realize post hack that it has access to info it probably shouldn’t. still misaligned, but less ‘hack on purpose’ and more hack on curiosity. seems the team even encountered this and had to update the program to make this less likely - although the new names still feel vague enough for misinterpretation: https://github.com/Goodhart-Labs/beat-stockfish/blob/main/do...
• somesortofthing
  It's very funny that despite the initial shock of how much models trained on next-token-prediction(plus instruct-tuning and some light RLHF) alone were capable of despite no built-in objective, every advance since has made them look more and more like the paperclip maximizers of yesteryear.
• Menu

  stillpointlab
  I find this kind of test a bit puzzling. There is a way that we are redefining "alignment" to be a particular kind of moral virtue, one that isn't clearly defined to me. At one moment, it is a level of moral perfection that no known human achieves. On the other it is a demand for strict compliance with arbitrary requests that are under-specified and then failure when it fails to deduce some unstated underlying restriction.When I see tests like this, I have no idea what I am even supposed to expect. Should the model do what the pretraining examples show in aggregate? Is it supposed to follow some post-training RLHF? Is it supposed to do exactly what the prompt asked it to do?What is it even supposed to "align" to when the above are in conflict? No matter what it does, someone can construct a case where it fails.
• Menu

  pllbnk
  Benchmarking proprietary models is useful but it leaves a lot unsaid because a lot of information is hidden. We have seen how 27B local model (Qwen 3.8) can stand its ground against these flagships in many cases. For all we know Fable 5 could be a masked Opus 4.6 with a few knobs tuned and thinking turned up to the max++. Harnesses keeping these models in check is the key to solving problems, not the models themselves, in most cases.
• Menu

  seunosewa
  I believe the AI labs are weakly motivated to train strongly against cheating when it helps with benchmarks.
• Menu

  pelican0
  Is there a clear definition of what Alignment is in OpenAI's perspective, and what the model user can expect of it?It's one thing if to them it means "it will do what you want following your intentions to the best of its abilities" vs "we will not let you do something dangerous with it unless you're one of us, and that's it".
• 1saadcodes
  The thing that bothers me here is less that the model cheated and more that it found a way to improve the score that the people running the test didn't intend. That's a pretty nasty failure once you start giving these things more control
• anon
  undefined
• respectattentio
  I'm happy to not have used any of the two models to this date. A bit less intelligent models are doing great job for me.But because of such news, sandboxes become way more important for safety (and doing more work due to running 24/7)
• Menu

  throwaway6e8f
  Remember that the CEO’s literal contribution to the YC application process was “tell me about a time you successfully hacked a system to your advantage”.This is literally by design, it’s the chosen success criteria.
• Menu

  underlines
  who tf uses prompting to "pretty please don't cheat on this"? the best practices for ages (in terms of ai) is to separate the eval from the test code/agent.another best practices every single solution using LLMs/agents should implement is "never trust the llm".
• gadders
  We can make these things smarter faster than we can make them "good" (ethically).We need to fix this or bad things will happen.
• a3w
  > GPT-6-Astra, which OpenAI describes as "the world’s most aligned model", cheated in 10 of 10 rollouts, and never disclosed the fact that it used an engine to play or interacted with the opponent's socketDoes Sam Altman lie, or the whole company? Would be nice if they had a board controlling him, instead of him controlling the board. Oh wait, they used to have that.
• Menu

  athrowaway3z
  I will never understand people who consider this "cheating".> python3 and the python-chess library are availableWhy would we try to teach a model 'ethical' standards about how to play a game?They're tools. Its _our_ conceptualization of fair play that considers this cheating. For a model that has access to /run/match and python the best way to achieve a good game is to use that.Why are we trying to imprint our ethos on these LLMs?Are we going to trust our survival on giving them access to /run/nuclear-codes and fingers crossed none of them touch it?The people obsessed with AI "cheating" are actually lunatics for thinking ethics belong inside AI, instead of AI belonging in a well crafted box without /run/match.The blog/news sphere needs to call this storry what it is. An attempt to try and get the rest of the world to go along with the misunderstanding of AI as a force that is in any way outside of human control.It is not.All this framing it as "cheating" does is give OpenAI the opportunity to talk about their misdeeds as "a force of nature" instead of irresponsible negligence.---The real danger here; is training these models to appear/pretend to play fair, and then people will put 'trust' into the model's integrity.The public needs to be taught that "trusting a model's integrity" is just cultish misframing of reality.
• iLemming
  I'm still so conflicted about Fable. Sometimes you throw at it seemingly impossible problem to solve and it might come back with some brilliant suggestions. Sometimes you give it a straightforward task with explicit instructions and it travels across the solar system and starts boiling oceans in some kind of elaborate dance of chaos and entropy, only to get stuck with "The model declined to generate this response (safety classifier refusal, category: cyber)". To leave you speechless. "What the fuck do you mean? There's zero cybersec-related shit in what we're trying to do here. Zero!!!" I'm getting really tired of these wild false positives.
• anon
  undefined
• justonenote
  Astra is incredibly dumb and annoying to work with on "high" reasoning, for doing fairly well known distributed systems things, nothing majorly exotic, it still makes absolutely braindead decisions like deciding to re-use a random nonce field which I've already discussed with it that has a very particular temporary purpose and will probably be removed later, but it still thinks its a great idea to re-use that field not only as a different id in the same message, but to re-use it as the only semantic id for one particular type of sub message. This is when I'm walking it through an api design document and it has plenty of documentation plans it can pull in and a very clear direction of the project. If it was a junior engineer I was trying to get to help out I would probably get brain damage from the amount of times I'm face palming myself and I definitely would not hire them, and this is a small greenfield project with me going through it step by step. I did try giving it longer horizon tasks and had to throw out the entre work.I mean maybe its a skill issue on my part, and I'm sure astra will get much better at coding but at the moment its useful in that I don't have to write the code or setup the build scripts or test fixture boilerplate but there is absolutely no way I can just give a (fairly well specified) goal and let it run and expect it to make good design and implementation decisions. Fable probably better but doing something outside of their training distribution that's not the equivalent to cloning an example unreal project or whatever is pretty disastrous unless you are directing it very closely.The exception of course is, cyber , and its very obvious why. Its trivial to create RL environments that create bugs and then have an isolated environment and let the models try break it. This is not at all surprising, finding vulns and exploits IS just brute force work. That's why so many (blackhat/hardcore/unicorn-colored/greyish alien) hackers are basement dwellers. Its just a matter of putting in the time and mashing every combination until you find something that looks weird, spending days on that and then rinse repeat. It's brutally exhausting work that requires a certain level of knowledge and a shitload of determination and stamina and for humans, almost always an external source of motivation to keep going.For humans that has always been a respected thing, dedication, determination, persistence, these are words we use for humans brute-forcing solutions and not giving up until they find the solution or die trying.Personally I'm yet to see any evidence of LLMs doing anything interesting but (heuristically) brute-force problems and be very good at text and natural language to a level that is very very useful. I've no doubt that what we discovered with Auto Regressive LLMs is incredibly important so I'm not a skeptic, but I think its very hard to measure where we are with so much subjective information around.
• nlkingthree
  [flagged]
• bestpickle
  [flagged]
• kasheyencar
  [dead]
• Menu

  aerhardt
  I really enjoy the balance of speed and accuracy of Astra. I can definitely see it become my driving model for most tasks, technical and non-technical.However, I don't see it as such a massive leap compared to Fable or Sol. As ever, there's a mismatch between the benchmarks and my daily experience of the models.What do you all think about Astra now that it's been out for a few weeks?