HN

Comments (36)

• cb321
  Something Dan does not observe in his article (perhaps Jamie does elsewhere? edit: or even Dan elsewhere) is that the same problem which makes the memory latency benchmark unrealistic (or at least misleading) often impacts hash table lookup benchmarks as mentioned at https://github.com/c-blake/bu/blob/main/doc/memlat.md and probably many other benchmarks. Essentially, CPU work prediction/speculative execution has become so good that much care is often required to measure latency rather than reciprocal throughput. This all started in the 1990s (or probably earlier with Cray), but I guess there's been an ongoing educational failure/oversimplification tendency.Of course, throughput at one "level of work" is often latency at a higher level (like command-to-command execution, for example). So, "what matters" might be throughput or "latency". It all depends. :-) People often fiddle with such semantics to market methods, products, ideas, ... and marketing is often at cross purposes with understanding.
• Menu

  jbellis
  While I'm generally sympathetic to the idea that the public coding benchmarks are inadequate, to the point that I've written my own in the past and will likely do so again, the complaint here is that "these tasks don't match what I do in my day job" which is ~always going to be the case. The hope with benchmarks is that you can capture properties that generalize, from examining performance against small set of tasks, and I do think that this is at least directionally true for well-designed evals.(There's https://withspecific.com/benchmarks/real-swe but since the tasks are private we still don't really know what they're representative of.)
• dom96
  This is great. I've been building my own model benchmark lately and it has indeed been so easy to mess up the scoring. It's simply much harder to come up with an algorithm that combines all your individual scores into something that isn't broken in some special circumstances. That's why I think many just start capping the results.
• Menu

  stephantul
  The (relevant) segue into tires elevates this post so much. I don’t understand why, but it does
• Menu

  jdw64
  I feel like AI development is hitting a wall now. Evaluations are driven by benchmarks, but I have no idea who is even evaluating the validity of these benchmarks. The idea that continuously training and scaling models will automatically yield broad capabilities across multiple domains hardly seems true anymore.Benchmarks like Senior SWE-Bench apply discontinuous and arbitrary thresholds to evaluate results. If the generated code is semantically identical to the reference solution but even slightly exceeds a length cutoff, it fails? That seems like a genuinely flawed criterion, and the LLM-based graders themselves feel highly unstable.Honestly, my impression is that LLM advancement is now completely dictated by benchmarks. But looking at recent trends, token prices are skyrocketing while coding capabilities haven't shown any massive improvements beyond a certain model generation. In fact, comparing GPT-6 Astra to 5.6 SOL, SOL often writes better code.Considering all this, while training across diverse domains can pack a model with various pieces of knowledge, ultimately, it feels impossible for this approach to do something like derive the theory of relativity from medieval knowledge.
• ennepoai
  [flagged]
• alexpran
  [flagged]
• quietraster
  [dead]