I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
I would guess it tends towards more likely because it can't not think of something mentioned.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
I’ve run those experiments. Commercially, so I can’t share data. But I can say what the article describes roughly matches what I’ve seen. How well calibrated the scores are varies by model, application, and subtle wording changes. I’m a little more optimistic than the author in that I have gotten it to work reasonably well. But I would not recommend productizing it with a SaaS LLM because it will break every time your vendor sunsets the model you were using, and getting it working again on a new model will be an unholy PITA.
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
You can, but it is also fraught and strange and hard to trust. The logprob on one particular token in light of all previous tokens is just that: a token probability. That’s not really the same thing as an assessment of all the reasoning or chain of thought or whatever that preceded that token.
That is the opposite of dumb, you can use the next token likelihood to macgyver together a jev in a pinch exactly that way, you just would also want to take some steps to scramble up the choices to smooth over bias and tokenization artifacts. llms have favorite letters for example. Qwen likes A and D.
I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.
When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
I think it's good to continually remind oneself that these are next-word/token-predictors...you can wrap them up in a harness, you can feed them into each other, you can influence the likelyhood of a given word through weights and system prompts...but that's literally all they can do. They can't reason, but they can predict what reasoning looks like.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
It’s equally useful to continually remind yourself that a Turing machine is just a spool of tape with a magnetic head, and that a human brain is just fatty lipids in a chemical bath.
Recently joined a company that does just this. They are using the LLM to review text and produce a grade and a confidence interval.
I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.
How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
Whilst general self-reporting of confidence will be hard to achieve (we can't even do it as humans), I'm sceptical of the notion that we can't train models to give some confidence scores accurately. I think over the past year that with the correct RL, models can showcase some pretty insane capabilities. If reporting the correct confidence was a RL training target say for predicting weather events or a game outcome that is supported by real RL environment or simulated data, then I suspect LLMs will develop a strong intuition for confidence similar to how the recent JEV and decision models are trained.
In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
While I agree the autoregressive training is poor at calibrating a model, it's clear that LLMs encode a kind of world model to be able to reason, do mathematics, programming, and logic. They are certainly imperfect, but the emergent internal representations go beyond mere "what text comes next" even though LLMs typically simply use tokens.
By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.
I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
LLMs are bad for rating or scoring anything because they have the sensibilities of the average person. Unfortunately, the average person has poor taste.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.
This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.
I was hoping the writer would follow the interesting framing of the quality of an LLM's confidence score with some data on how good it is, rather than just proclaiming it's bad because of sycophancy tendencies. If someone collected the data on something measurable, I would be willing to bet the LLM's confidence number isn't too far off.
You mean like this recent preprint?
https://arxiv.org/html/2607.20526v1
From that paper, model accuracy is not perfect, but it's overall pretty favourable. There are some encouraging findings in there, too, including that accuracy of model confidence scales with reasoning effort.
I would guess it tends towards more likely because it can't not think of something mentioned.
But I agree, I would8 much rather have some data than reckons.
It's with noting that people do this too. They tend to vastly overetimate a low probability event that they know is possible. Often by more than one order of magnitude.
It is something worth keeping in mind when you are people throwing around personal p doom scores.
Interestingly when humans have to make a decision based upon the probability they feel, they tend to act far closer to the actual probability than what they would say the probability is.
I’ve run those experiments. Commercially, so I can’t share data. But I can say what the article describes roughly matches what I’ve seen. How well calibrated the scores are varies by model, application, and subtle wording changes. I’m a little more optimistic than the author in that I have gotten it to work reasonably well. But I would not recommend productizing it with a SaaS LLM because it will break every time your vendor sunsets the model you were using, and getting it working again on a new model will be an unholy PITA.
I'm sorry, I'm going to sound dumb for a second.
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
You can, but it is also fraught and strange and hard to trust. The logprob on one particular token in light of all previous tokens is just that: a token probability. That’s not really the same thing as an assessment of all the reasoning or chain of thought or whatever that preceded that token.
That is the opposite of dumb, you can use the next token likelihood to macgyver together a jev in a pinch exactly that way, you just would also want to take some steps to scramble up the choices to smooth over bias and tokenization artifacts. llms have favorite letters for example. Qwen likes A and D.
I got a degree in statistics, and what I remember from those times is that we spent maybe 10% of our time learning how to build the model and 90% of the time correctly interpreting the model output. Of course, in the time I learned statistics it was almost exclusively applied in hard science, finance, and actuary. The wild west of machine learning hadn't gotten started. I wonder if they teach it differently now.
When we start using big datasets where n went into the tens of millions we kinda stopped bothering because margins were so small it was effectively certain you found the mean of some range. However I think in that time we forget what confidence scoring actually is and how quickly multiplied alpha can balloon on you.
I think it's good to continually remind oneself that these are next-word/token-predictors...you can wrap them up in a harness, you can feed them into each other, you can influence the likelyhood of a given word through weights and system prompts...but that's literally all they can do. They can't reason, but they can predict what reasoning looks like.
None of this means they can't be useful tools, but when they fall down, we should remind ourselves of why that is, and be honest with ourselves about what they could ultimately be capable of.
What is the difference between reasoning and predicting what reasoning would look like?
The same as the difference between a thing and a statistical model of a thing.
They don't always match up.
In theory, theory and practice should be the same; in practice, they are not.
Human reasoning also does not match up with reality. Isn't it statistical modeling all the way down?
One has a human behind it and one has a matrix latent space. Usually human understanding is more "real"
Stallman vehemently states llms are bullshit generators with no true understanding.
But sometimes you can approximate your problem and unfold an llm interlocator to producing the strings you need for your task
It’s equally useful to continually remind yourself that a Turing machine is just a spool of tape with a magnetic head, and that a human brain is just fatty lipids in a chemical bath.
Recently joined a company that does just this. They are using the LLM to review text and produce a grade and a confidence interval.
I'm no expert in the field but it is clear that the model outputs are not reliable. To my frustration this was designed by "experts" so I originally trusted that this approach was sound.
How does this compare with descriptive confidence? For example, major v.s. minor? Intuition tells me it's a different flavor of the same illusion.
Whilst general self-reporting of confidence will be hard to achieve (we can't even do it as humans), I'm sceptical of the notion that we can't train models to give some confidence scores accurately. I think over the past year that with the correct RL, models can showcase some pretty insane capabilities. If reporting the correct confidence was a RL training target say for predicting weather events or a game outcome that is supported by real RL environment or simulated data, then I suspect LLMs will develop a strong intuition for confidence similar to how the recent JEV and decision models are trained.
The first half of your first sentence is correct
In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.
I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.
While I agree the autoregressive training is poor at calibrating a model, it's clear that LLMs encode a kind of world model to be able to reason, do mathematics, programming, and logic. They are certainly imperfect, but the emergent internal representations go beyond mere "what text comes next" even though LLMs typically simply use tokens.
By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.
I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.
Yeah, using them for any (most) subjective task is generally incorrect
LLMs are bad for rating or scoring anything because they have the sensibilities of the average person. Unfortunately, the average person has poor taste.
This is a tough one to explain because some people think the average person has good taste by definition. I used to think that way but over the years I've realized that a lot of what the average person likes is derivative 'slop' - This was true before AI and still true with AI.
Once you start deducting points for lack of originality, you will realize that people have really poor taste... Very narrow taste. Most people literally consume trashy variations of the same thing over and over. We have to start deducting points for this because the more intelligent among us can't take it anymore! We're tired of being exposed to thousands of variants of the same slop... It might be sufficiently different to fool the average pleb's simple mind but to us it's like eating the exact same dish every meal of every day with different amounts of salt and pepper.
This comes through when you ask an LLM to rate/score something. It's frustratingly conventional and not based on first principles thinking and it ADDS extra points for being derivative instead of SUBTRACTING as it should. We can see it play out in the workplace. LLMs have poor judgement. It's the one area where they lag behind.