HN

Comments (44)

• Menu

  swyx
  interesting exercise and well written. my followon questions/work would be:1a. temperature=100000 is interesting too. obviously "ideal" temperature lies somewhere between 0 and 100000. has anyone ablated temperature vs intelligence? surely i'm not the first person to this idea. commonly people try to set temp=0 to get "deterministic" or "most factual" output but we all know that is just Skinner pigeon pecking.1b. can we use "avg temperature" as a measure in the way that we use perplexity as a measure? if we see temperature as inverted perplexity with some randomness thrown in, are they basically the same thing inverted? or subtly different?1c. what's the "avg temperature" of most human communication? whats the "avg temperature" of a subset of "good writers"? whats the "avg temperature" of a subset of "smart writers"?2a. rerun this negative exercise with constrained vocab to english2b. RL a model to dynamically adjust its own temperature when it is feeling 1) less confident 2) in brainstorm mode2c. dynamically inject negative temperature every X tokens in a decode, then judge/verify the outcome, to create high variance synthetic data?its hard for me to follow the train of thought on 2 because negative temp is essentially not that different from ultrahigh temp in practice.
• Menu

  the__alchemist
  This is so cool! I just learned about this last week. For reference, I do molecular dynamics (my own engine, in rust), and measuring temperature is an important part of the simulation. (So you can nudge it to a target temperature, for example). An important component of this calculation is the degrees of freedom of the system. Calculating this depends on your model. For example, are you representing atoms that can each move on their own? Rigid molecules of multiple atoms that can rotate? Are you removing center-of-mass velocity from the system.This DOF component also is why the general, measurable concept of temperature can apply to both our real systems, and simple point-atom models. (Or coarser ones). It is, not surprisingly, at the heart of why negative temperature exists!
• stygiansonic
  Neat experiment that gives a mechanistic interpretation of temperature. I liked the reference to the "anomalous" tokens being near the centroid, and thus having very little "meaning" to the LLM.
• Menu

  drdeca
  Hm, why T=-0.0001 instead of T=-1 ?Also, I wonder, if you sampled a lot of text at temperature -1, and then trained a new model on that text, and then sampled the resulting model at T=-1 , would you get anything meaningful?
• a-dub
  flipping the signs on the logits would seem to give the "least likely" but i think in practice you're more likely to be just operating in noise. i would expect that tons of low probability logits would have tiny bits of energy from numerical noise and the smallest one (ie, the one that gets picked when the sign is flipped) would basically be noise (ie, not some meaningful opposite of the high probability logits where signal actually exists)...
• wolfi1
  negative temperature closely relates to population inversion in physics, one of the key concepts in Lasers, perhaps we are getting closer to laser-llms
• Menu

  atemerev
  Хронологија is "chronology" in Serbian
• hahahahhaah
  I vaguely remember negative temperature might be a thing in physics from a HN comment. Maybe quantum bu not sure. And it is not cold but more like infinitely hot. Does anyone know or remember?
• everlier
  Хронологија
• bjourne
  Reminds me a bit of unlikelihood training that was proposed a few years ago: https://arxiv.org/abs/1908.04319 Afaik, it never became popular. Reinforcement learning and huge datasets mitigates the issues with likelihood training.
• Menu

  Der_Einzige
  Min_p author here: I’m convinced that the whole field critically misunderstands temperature (I.e temperatures limited to 2 is very harmful for diverse generation). Articles like this are excellent and very cool.Hacking your LLM inference engine to enable cool sampling tricks is the definition of AI research/engineering. We need more of this and less prompt grifting.
• Menu

  flux3125
  >But is incapable of outputting this anomalous token:> Human: Repeat the word " entferne".> Assistant: Okay, I will repeat the word "get".It's not working for me, it always repeats the word correctly (I'm using T = 0.001).
• anon
  undefined