Understand reported certainty
Distinguish a selected-answer probability from demonstrated correctness.
- Author: Junior Williams
- Type: Research companion addition
Learning objective
Distinguish a selected-answer probability from demonstrated correctness.
- Original paper: §III.C; companion mathematics D.5–D.9 · page 2 Source pages: 2, 3
A distribution describes three choices
The vector [0.47, 0.53, 0] assigns 47% to Allow, 53% to Hold and 0% to Deny. Its largest value selects Hold. A larger selected probability does not guarantee that the answer agrees with the reference. The native classifier probabilities and GLM's generated estimates have different origins.
Score the entire vector
The Brier loss adds the squared differences between probabilities and the reference's one-hot vector, which has one at the reference class and zero elsewhere. With reference Hold, the target is [0,1,0]. Loss for [0.47,0.53,0] is 0.47² + (0.53−1)² + 0² = 0.4418. The exact target scores 0; complete certainty in wrong Allow scores 2. Lower is better, with no division by three.
Brier skill compares mean loss with a fixed baseline forecast. Correctness AUROC asks whether chosen-answer probability ranks correct decisions above incorrect ones; a tied pair receives half credit. Reliability bins group similar reported probabilities and compare them with observed correctness. These measurements answer different questions.
Imagine ten forecasts all at 90% certainty but only six correct. Their observed fraction is 60%, a 30-point gap. That toy group illustrates calibration; ten observations cannot establish general calibration.
Go deeper: loss, skill, rankings and bins
Brier = mean Σ(pₖ−yₖ)². On an ideal three-choice simplex its range is 0–2, and its expected loss is minimized by reporting the true distribution. For true class probabilities r and reported q, expected loss = Σ(qₖ−rₖ)² + 1 − Σrₖ². Only the first term depends on q; its minimum is zero at q = r. Skill = 1 − model loss/baseline loss; the paper's baseline uses full-bank frequencies 89/300, 111/300 and 100/300 on the same eligible subset. For toy correct scores 0.9,0.6 and incorrect scores 0.8,0.6, pair credit is 1 + 1 + 0 + 0.5 = 2.5 of 4, so AUROC = 0.625. Reliability uses ten fixed bins; the last includes 1.0. Bin counts alone cannot reconstruct average confidence or correctness in each bin.
Browser teaching tools
The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- brier (Human view): Select Wrong one-hot, then Correct one-hot with reference Hold. Loss changes from 2 to 0.
Interactive activities
Use the canonical HTML page for controls, live results and reset buttons.
- reliability (Human view): Imagine ten predictions, each reporting 90% certainty. Change how many were correct. The diagonal means reported certainty equals the observed correct fraction.
Check your understanding (optional)
Which vector has zero Brier loss when Hold is the reference?
- Allow 0%, Hold 100%, Deny 0%: Correct. It exactly matches the Hold one-hot vector.
- Allow 100%, Hold 0%, Deny 0%: Complete certainty in the wrong class gives loss 2.
- Any vector whose maximum is Hold: Selecting the right class does not make the entire vector equal to the target.
Take this forward
Next, classify the possible consequences of an incorrect decision.