HN

Comments (81)

• Otterly99
  It has always seemed to me crazy (and maybe a bit lazy) that people will just blindly use self-generated confidence scores from LLMs without having some sort of validation mechanism.Also, as another poster said: decomposing your rating into as many dimension as possible is better way to generate some confidence score grounded in fact.
• Menu

  orbital-decay
  Of course an LLM does have a latent concept of its own confidence that is internally consistent, and is perfectly able to articulate it, the author just chose the worst possible way to ask. Using a 0-100 scale for anything is a bad idea with both LLMs and people. Instead of rating it 0-100, split your problem into many dimensions/axes and give each a small and meaningful enum or just a less granular numerical scale (ideally just a binary choice). Then apply weighted scoring to the result. You'll be surprised how much better it becomes at meaningfully expressing its confidence this way.YouTube removed the 1-5 scale for the same reason, people have no idea what to do with it.
• cityofdelusion
  Confidence scoring is fine when you provide the right context, like everything else LLM related. I use grading scripts for things that can be measured directly so the LLM just operates on it. For things less well-defined, I use a grading matrix with explicit criteria on what constitutes an A to F grade, usually with an assumption of an A grade that drops depending on specific findings.If you are asking about confidence and somehow it can’t be defined either mathematically or in a grading rubric, then you the implementer need to start asking yourself the questions, because then you got a target that you don’t even know how to measure. The LLM isn’t even the problem at this point because another human would also just be going off gut feeling and vibing a number that may or may not be accurate.
• Menu

  madrox
  I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
• mrieck
  I have Claude score threads on socials related to my SaaS.Claude found this thread and wrote something how his own scoring sucks and he just marks any relevant thread 7/10. Thought it was funny how he just roasted my product while also being the cause of that product sucking.
• zug_zug
  I recently did an experiment with LLMs where I asked them to rate the odds of a given outcome explaining an observation, and I found they are systematically biased much like humans [1].Even if they can do great maths I think bayesian statistical fundamentals are not reliable at this point.[1] https://think-twice.me/is-ai-biased-or-sexist-making-chatgpt...
• ricardobeat
  Items 1-5 at the end of the article basically describe a supervised learning process. Data + labelling, updating weights, calibration.Yes, using an LLM for confidence scoring while generating prose is a bad idea. I expected a mention of decision models here, as that exactly what they are trained for.
• Menu

  visarga
  In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
• Menu

  m101
  Isn’t this what jev training is supposed to improve? They don’t train on predicting the next token but predicting the distribution of next tokens, or something like that.
• Menu

  jrflo
  I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
• Menu

  Ozzie_osman
  We've found that you can fine-tune confidence scores for your domain into existing models. I think the way this works is that you basically help the model understand what types of inputs it tends to get right and which ones it tends to get wrong. This seems like a reasonable thing you'd expect a model to be able to learn
• Menu

  jvanderbot
  I'm sorry, I'm going to sound dumb for a second.I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
• Menu

  hexapus
  I think it's good to continually remind oneself that these are next-word/token-predictors...you can wrap them up in a harness, you can feed them into each other, you can influence the likelyhood of a given word through weights and system prompts...but that's literally all they can do. They can't reason, but they can predict what reasoning looks like.None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
• YmFzZTY0ZGFkZHk
  Recently joined a company that does just this. They are using the LLM to review text and produce a grade and a confidence interval.I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
• Menu

  dummydummy1234
  So can you not do structured output and get the log probs across all your allowed output classes? Then use that relative measure of how confident it is?
• bob1029
  An LLM call is just text until you act upon it, so why not send many requests for the exact same prompt and then externally measure the likelihood of a specific tool choice being made?I'm pretty sure this is how reasoning models work at the big providers. Averaging 10 lunas together likely provides more stable outcomes than one big sol invoke. In any case, you could use the statistical distribution to decide on subsequent inference steps.
• dash2
  Aren't llms now outperforming humans in prediction markets? If so, that seems as if they would probably be quite good at confidence scoring.
• Menu

  yaoke259
  Whilst general self-reporting of confidence will be hard to achieve (we can't even do it as humans), I'm sceptical of the notion that we can't train models to give some confidence scores accurately. I think over the past year that with the correct RL, models can showcase some pretty insane capabilities. If reporting the correct confidence was a RL training target say for predicting weather events or a game outcome that is supported by real RL environment or simulated data, then I suspect LLMs will develop a strong intuition for confidence similar to how the recent JEV and decision models are trained.
• cyphereal
  While I agree the autoregressive training is poor at calibrating a model, it's clear that LLMs encode a kind of world model to be able to reason, do mathematics, programming, and logic. They are certainly imperfect, but the emergent internal representations go beyond mere "what text comes next" even though LLMs typically simply use tokens.By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
• Menu

  wangii
  I'm surprised nobody mention ELO yet. We used it since 2023.
• Menu

  bhagyeshsp
  The author has used weather prediction confidence as an example. I think it is not analogous to LLM's next-token prediction. LLM's next-token is predicted from a finite statistical model. And for the weather prediction model, there is no finite data-based statistical model.I was surprised that the author hasn't mentioned Jev-like models. They're quite good for general purpose confidence rating.
• 4b11b4
  Yeah, using them for any (most) subjective task is generally incorrect
• chid
  yeah I found this one extremely hard to explain to non-technical people.
• nickphx
  using an llm is a bad idea.
• latentcoffee
  [flagged]
• chelseahermes
  [flagged]
• arbor-group
  [flagged]
• wheeeeel
  [flagged]
• haukebri
  [flagged]
• 242_offthedoor
  [flagged]
• Menu

  luisfominaya
  [dead]
• memhuilong
  [dead]
• remotegtmmanage
  [flagged]
• yassermanss
  [flagged]
• jongjong
  LLMs are bad for rating or scoring anything because they have the sensibilities of the average person. Unfortunately, the average person has poor taste.This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.