HN

Comments (39)

• Menu

  denysvitali
  Better link: https://iquestlab.github.io/But yes, sadly it looks like the agent cheated during the eval
• Menu

  sabareesh
  TL;DR is that they didn't clean the repo (.git/ folder), model just reward hacked its way to look up future commits with fixes. Credit goes to everyone in this thread for solving this: https://xcancel.com/xeophon/status/2006969664346501589(given that IQuestLab published their SWE-Bench Verified trajectory data, I want to be charitable and assume genuine oversight rather than "benchmaxxing", probably an easy to miss thing if you are new to benchmarking)https://www.reddit.com/r/LocalLLaMA/comments/1q1ura1/iquestl...
• Menu

  brunooliv
  GLM-4.7 in opencode is the only opensource one that comes close in my experience and probably they did use some Claude data as I see the occasional You’re absolutely right in there
• anon
  undefined
• anon
  undefined
• Menu

  adastra22
  A 40B weight model that beats Sonnet 4.5 and GPT 5.1? Can someone explain this to me?
• squigz
  This is a lie, so why is it still on the front page?
• simonw
  Has anyone run this yet, either on their own machine or via a hosted API somewhere?