HN

Comments (42)

• Menu

  evilmathkid
  Hi! Author here. Surprised to see this on HN now. Happy to answer any questions!Some context about this:- This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be tackled without LLMs- Till the v1 of this result, this benchmark was only scaled by LLMs or their finetunes (ofc w enormous training costs). Other attempts performed okayish but used v complex architectures or extremely high amounts of training compute. No one expected a simple AR transformer to perform this well, at this low cost and w these few training samples.- Sample Efficiency is one of the most important unsolved problems today in AI. That's what I was targetting with this work. We know it is easy to increase SE by increasing compute/params, so it was important to constrain cost as much as possible (also why OpenAI's Parameter Golf had fixed compute and why Modded NanoGPT is considered very sample efficient)- Can the perf be improved? Yes but the competition is ongoing so can't talk about it- Personally I think today's frontier models can be beat by training from scratch. Haven't proved this yet tho- Fun: I was new to ML when I posted this first (dec '25). I basically used ARC as a way to learn ML
• Menu

  xeonax
  Even cooler is his about me mention of saving his own life https://mvakde.github.io/ > Saved myself in a medical emergency (doctors didn't know what rhabdomyolysis was)
• dhruv3006
  > The mentioned approach is fundamentally flawed, since the inputs are used during pretraining constituting to a leakage, a universally recognized flaw of ML training.I saw this on the community note for the last blog you wrote - anything to do here.
• kriro
  Thanks for motivating me to work a bit on non-LLM things again :)
• kvn95ss
  > Also, I’m not sure whether “general reasoning” even exists in the first place? Maybe humans are specialised tooI have been wondering the same. We are now exposed to so many stimuli, we are tricked into thinking this is the norm - to have a reasonable understanding about everything, unless specialization is called for.
• hn45e7pbij
  Nice to see arc-agi-1 framed this way — I'd been circling the same idea without the right words.
• pwmglenn
  Really impressive and creative research. I wonder if the leading labs do anything similar with their models? It doenst look like the open source labs do?
• Menu

  westurner
  How does it perform on ARC-AGI-3?There was this a few weeks ago:"Schema Harness Achieves ~99% on Arc‑AGI‑3 Public" https://news.ycombinator.com/item?id=48938163>> Schema, the harness we introduce today, reaches 99% on the ARC_AGI_3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 SolWhat does that do with 5.6 Luna instead of the expensive models?What of 'schema' would improve the performance of mdlARC?mdlARC: https://github.com/mvakde/mdlARCThere's an updated ARC-AGI-1 chart with 5.6 Luna in each thinking level in this video from last week: "A New Architecture [..] | MOONSHOTS " https://youtube.com/watch?v=qQfUbo7Ldc0&t=2m5s
• Menu

  eis
  > Increases in LLM scores are now mainly driven by post training (evidence in next section) and are probably a function of amount of synthetic data. They are learning to solve ARC tasks, not learn general abstract reasoningAgreed and that's for any benchmark. Private tests are better but you still have to trust the provider to not log and use them for training.That's why I like when a new set of tests like a new ARC-AGI version is published, that's where you can see which of the models abstracted to more general capabilities instead of being focused on the previous tasks. Most models completely fail new ARC-AGI tests.The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. You hit a ceiling very fast and investing into more compute will give you diminishing results. So yes, you can train a custom model to do somewhat decently on a specific set of tasks but then what?
• GovPulse
  [flagged]
• larodi
  "I don’t understand why others didn’t figure this out"- how about we allot the possibility that so many of presumed ML experts don't have any clue what they be doing, and are eventually API bitches, nothing more.
• Menu

  embedding-shape
  Is the author only running their model against one benchmark? I don't think anyone finds that difficult to achieve, the difficulty comes when you want to make the model not benchmaxxed to a specific benchmark, and generalize so it can solve problems not part of the training data, but seems this model is specifically for not this? How useful is that?If you just wanted to pass these specific tasks in this specific benchmark, and wanted to do so cheaply, I'm sure a non-LLM-based approach would yield better results for even cheaper, since what the author's model does, seem to basically be "solve ARC puzzles", not a general LLM or "coding" LLM.