HN

Comments (253)

• Menu

  vessenes
  This is insane.Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
• Menu

  NitpickLawyer
  > As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weightsHA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
• evanjrowley
  Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.[1] https://github.com/firelex/jeff-code[2] https://github.com/firelex/jeff#benchmarks
• Menu

  vanyle
  I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.We'll see if this attempt at RSI worked.
• Menu

  adverbly
  Mind blown.I simply must know now: Could they do it with neurons in a brain?Some absolutely crazy and insane experiments if this ability grows.Could they edit an animal's brain? Can they teach things to animals???
• Menu

  futureshock
  I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
• Menu

  Imnimo
  On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
• Menu

  Roland303
  I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
• bluejay2387
  This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.
• Menu

  nsypteras
  I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.
• AlexErrant
  As insane as this result is, next steps are even crazier:> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
• crorella
  This is remarkable and something I would have never though of doing regardless of how much it makes sense!I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
• faxmeyourcode
  This is so cool. It feels similar to the models' ability to read base64.https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...
• btucker
  This survey paper from earlier this year has a lot of good related work:A Survey of Weight Space Learning: Understanding, Representation, and Generationhttps://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning
• Menu

  alienbaby
  I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
• Aboutplants
  Throw out whatever timeline you previously had
• Menu

  dumberquestions
  We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
• daemonritus
  almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
• lifty
  Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.
• Menu

  mynti
  I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising
• Menu

  davesque
  Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?
• Menu

  FeepingCreature
  I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.
• Menu

  alienbaby
  This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...
• neuronexmachina
  I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
• Menu

  levinb
  Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
• Menu

  rpcope1
  So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?
• m1coti
  I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
• Menu

  lisenKaci
  Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?
• Menu

  mlsu
  I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.- the best way to improve on some task is to have interpretability into your own neurons (..?)- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly- therefore, models naturally put effort into 'self-interpretability'I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
• Menu

  rumblefrog
  I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.
• tjohnell
  This seems like a big deal.
• famouswaffles
  This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?
• ernubufiei
  I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.Absolutely incredible regardless.
• bilsbie
  GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.
• jonplackett
  Is there any chance they’ve just seen these weights before and are recognising them?If not this is mind blowing.However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
• olirex99
  I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
• bfung
  Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?* what, if any, is that extra magic sauce?* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
• gunalx
  How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?
• sligbad
  1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.Such an unexpected result...
• x187463
  Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.
• enestaylan
  Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
• swingboy
  So how would this be in the training? Does OpenAI train models on their own epochs?
• anonyfox
  Guess this is RSI in a nutshell
• clarle
  This is wild. Could Astra read and potentially audit a mapped human brain in that case?
• Menu

  mlmonkey
  This raises so many questions.Unless Astra was trained on other networks, how is it able to reason about the weights?Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
• badmonster0
  this is really cool
• bilsbie
  The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.
• IHLayman
  ... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
• epestr
  Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next
• kikdij
  That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
• iyn
  This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
• multibit
  Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.
• happyPersonR
  Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
• rumblefrog
  I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
• 6thbit
  Wow, in a way it's like CRISPR for ML models!
• mbowcut2
  This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
• sigbottle
  This is fucking insane LMFAO what?I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
• Ardon
  WildThis also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weightsAnd the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
• KeplerBoy
  Is it aware of projects like [1] and implementing those?[1] https://github.com/lyramakesmusic/neuron-explorer
• rumblefrog
  Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?
• tjungblut
  Great, instead of backpropagation, we now have slopagation.
• hard_times
  Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?
• czig
  I wonder if this behavior emerges when using recursive self improvement to generate new models
• anon
  undefined
• jay986
  this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
• InvisGhost
  Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).I wonder how it would do with distilling models by hand?
• j0e1
  RSI was already here?!
• skybrian
  Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?(When they say "Writing a Go algorithm," they mean the board game, not the language.)
• barrenko
  Kind of like octopuses with the RNA editing, but doing it it on another octopus.
• Menu

  educasean
  Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
• aroman
  This doesn’t look like “normal” technology to me…
• jjcm
  At what cost? That's the part I'm curious about.On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
• Menu

  __alexs
  Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?
• Ancalagon
  Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park
• nialv7
  this is maybe one of the most bonkers machine learning research i've seen all year.
• 7777777phil
  A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...
• fabian2k
  Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
• k9294
  Whaaaaat!? This is crazy right?I'm just blown away. Editing weights? Writing weights from scratch!?
• kristiandupont
  I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster
• anon
  undefined
• adeptima
  Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase
• saagarjha
  I wonder if one could use this to efficiently abliterate models and identify guardrails ;)
• kingstnap
  Friendship ended with AdamW. Astra take the wheel.But for real this is insane. Okay I need to get on this stat.
• esafak
  This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.Does this open the possibility of better decoding biological brains?
• VectorLock
  Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.
• vkgcr
  This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...
• Menu

  ahmedfromtunis
  I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.This looks very similar to that in a way.I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
• 6thbit
  This could set us on a path to weights becoming thew new compilation target.
• TGower
  Quick, fine-tune your qwen models before they patch this out of Astra.
• Imustaskforhelp
  I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)I am surprised that it did that and I am just left feeling technically impressed. This is insane.The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
• Menu

  vegnus
  I was here
• suddenlybananas
  Does asking an LLM to not use tools guarantee it doesn't use tools?
• colingauvin
  I have absolutely no words. If this is real, I can't even imagine the future anymore.
• _boffin_
  This is straight black magic. llama 70b, let’s see what we can push you to
• Menu

  ForHackernews
  Possibly related to https://already-clean.goodfire.com/
• wg0
  Too good to be true! Shocking!
• karim79
  I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.I'm still learning but this is fun as fuck.
• tsunamifury
  The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.
• Menu

  jackdoe
  talk about homoiconicity hahalisp people will be in awe
• jstummbillig
  That seems both entirely obvious in hindsight and potentially super important.I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
• markusw
  Whoa. Wild. I want to try this!
• luke-fugu
  Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.
• Menu

  33MHz-i486
  kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective
• anon
  undefined
• anon
  undefined
• sreekanth850
  Where are we heading?
• toppigamer
  i was there lol
• christkv
  It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.
• Menu

  Starlevel004
  I cannot say this fills me with confidence about alignment.
• moralestapia
  I think this is how RSI will come to be.
• Jtarii
  Slightly terrifying.
• jefb
  This is shocking.
• nkohari
  I can't help but be skeptical of this, but... damn.
• augment_me
  Big if true
• isoprophlex
  Sorry for what is undoubtedly a low-informational comment but..Whoa. What the absolute fuck is going on here.
• Menu

  WarmWash
  They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.It's very impressive, but also very expected.
• porridgeraisin
  a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretabilityThat an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.Edit: tweets from one of the coauthor providing more context:https://x.com/a1zhang/status/2117328657161322810https://x.com/a1zhang/status/2117330975122235442
• Menu

  josh-wrale
  Can we call this a form of recursive improvement?
• JeppyG
  Now our AIs have AIs lol
• adammarples
  Uh oh, it's iterative self improvement time, with no training runs, we're screwed
• reverseblade2
  [dead]