HN

Comments (22)

• herpderperator
  I'm curious: If someone wanted to serve models off hardware/silicon directly (like Cerebras or Taalas, and soon Google I think) rather than GPUs, would these parameters still be adjustable at request time? Or would they have to decide that before the model can even start serving and it would be locked in until they reload it (which would make it briefly unavailable)?
• salamo
  Possible reasons:- They might be dynamically adjusting these at inference time [1]. For example, start with a low temperature and generate samples with increasingly high temperatures until one of them passes some quality gate.- They don't want you to fine-tune on high temperature completions (rejection fine-tuning). You could call this "rejection fine-tuning rejection".[1] https://rlhfbook.com/c/09-rejection-sampling#related-best-of...
• franky47
  > To improve determinism, define a system instruction with explicit rules for your specific use case."Please be deterministic".
• Menu

  aesthesia
  My guess is that RL training being done with particular generation parameters makes models much more brittle to changes in these parameters, and that's why we're seeing changes like this across model providers. But I don't really know.
• Menu

  nichohel
  "Last" or "latest"? Those are rather different.
• Menu

  kouteiheika
  Obligatory "The Conspiracy Against High Temperature Sampling":https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...
• Menu

  tolugenius
  > To improve determinism, define a system instruction with explicit rules for your specific use case.Is this guaranteed to work any better than top_k or top_p? This just sounds like making a smaller version of a Agent.md doc.
• tough
  fwiw sonnet-5 also drops temperature (sonne-4 had it)
• charcircuit
  Along with everything else. These parameters can make speculative decoding less accurate increasing the inference cost.
• impulser_
  Good. These have been basically useless for the past few generations of models, and most of the time made the model perform worst.
• gdiamos
  thank god, these parameters are so confusing
• greatgib
  1. Sampling parameter deprecation (temperature, top_p, top_k)temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error. Remove these parameters from all requests.