<- Back
Comments (39)
- cesarvarelaPlease add Fable; a good benchmark should show that Fable is less prone to just autocomplete and instead pushes back or is at least more tasteful.
- sothatsitThis matches my experience of Opus 5 being a nice improvement over Opus 4.8, but not being revolutionary like Fable felt.I’ve now replaced my use of Opus 4.8 xhigh with Opus 5 medium, and I’m using less tokens and it’s quicker. I can understand people being annoyed by its writing style but for getting work done that really doesn’t bother me. I’ve been really enjoying using it.
- VgooseNice! I actually ran across this paper+benchmark recently, too. It's the first I've found that start to aim at some of the non-functional and longitudinal requirements that I think have always been an important part of writing production code.It's especially relevant now that models are good enough to solve ~most point-in-time problems.Some relevant but disconnected thoughts:- deterministic scores are so nice- what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is- another signal I've been thinking about and I'm seeing increasingly get brought up is the state space of a system; I'm seeing formal methods pop up a lot recently
- Johnny_BonkI haven't joined your chats in a while but glad to see you put this together, I truly feel as though opus 5 is not much of an improvement. The only time i ever felt a wow factor was opus 4, 4.6 and fable pre trump admin lobotimizing
- killingtime74Did you not benchmark latest GPT 5.6 or GLM 5.1/Kimi K3 because of cost? I can run them if you share how you ran them
- delberttyMostly a harness problem in my experience. Slop accumulates when the agent can touch anything, so constraining it to one seam and having it add alongside rather than edit in place does more than model choice.
- ajwinTo what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.
- akurilinReally hoping that all of the attention you're bringing to the longitudinal sloppification of codebases makes it back to the labs and creates some pressure to improve that trait of the models. This new benchmark seems promising.At the same time, I imagine it will be hard for them to prioritize this over improving flashy one-shots of impressive zero-to-one feats that demo so well and attract more customers.Can we have simonw make "pelican on a bicycle after 1000 requests for iteration" popular?
- dclfinally the benchmark for me
- tamimioSo many benchmarks more the models themselves.. just make one unified standard to benchmark all or stop calling it “benchmarking” as this word lost its meaning.
- knighthackerThis is where Opus 5 shines
- cute_boiOpus 5 is an overconfident stupid model. It tries to generate too much slop, tries to act like everything will fall. I have reversed back to fable and codex sol.