HN

Comments (25)

• Menu

  jumploops
  > Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].Everything old is new again (:[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf[2] https://www.microsoft.com/en-us/research/publication/distrib...
• bit_rot73
  My RTX 3090 is still laughing at my attempts to run 70B models efficiently.
• clem_rw
  My daily struggle is trying to make a 7B model respond in under 500ms without breaking the bank. This hits home.
• Menu

  ttoinou
  Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate. This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.
• Menu

  brrrrrm
  this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there
• censor25
  Nice read. I was wondering what can one do to get into inference engineering as simple theoretical knowledge is not sufficient and switching profiles is tough for someone with years of experience.
• Menu

  arjie
  You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.
• Menu

  qingcharles
  > A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.
• Menu

  datadrivenangel
  The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.
• calclavia
  good recap on the recent inference techniques!
• fsckboy
  "the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio".The efficient frontier of LLM inference is a line, not a frontier.this is a frontier:https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.If you want to steal an idea for this, steal indifference curves, they'd make more sense.
• pan_lid
  [dead]
• yeasin-arafat
  [flagged]
• soricus
  [dead]
• paidx
  [flagged]
• jing09928
  [dead]
• nedo_var
  [dead]
• killerdog10
  [dead]