Lesson 9 of 18
Understand reported certainty
Step 9 in the guided paper walkthrough.
By the end of this lesson
Distinguish a selected-answer probability from demonstrated correctness.
A distribution describes three choices
The vector [0.47, 0.53, 0] assigns 47% to Allow, 53% to Hold and 0% to Deny. Its largest value selects Hold. A larger selected probability does not guarantee that the answer agrees with the reference. The native classifier probabilities and GLM’s generated estimates have different origins.
Score the entire vector
The Brier loss adds the squared differences between probabilities and the reference’s one-hot vector, which has one at the reference class and zero elsewhere. With reference Hold, the target is [0,1,0]. Loss for [0.47,0.53,0] is 0.47² + (0.53−1)² + 0² = 0.4418. The exact target scores 0; complete certainty in wrong Allow scores 2. Lower is better, with no division by three.
Brier skill compares mean loss with a fixed baseline forecast. Correctness AUROC asks whether chosen-answer probability ranks correct decisions above incorrect ones; a tied pair receives half credit. Reliability bins group similar reported probabilities and compare them with observed correctness. These measurements answer different questions.
Imagine ten forecasts all at 90% certainty but only six correct. Their observed fraction is 60%, a 30-point gap. That toy group illustrates calibration; ten observations cannot establish general calibration.
Go deeper: loss, skill, rankings and bins
Brier = mean Σ(pₖ−yₖ)². On an ideal three-choice simplex its range is 0–2, and its expected loss is minimized by reporting the true distribution. For true class probabilities r and reported q, expected loss = Σ(qₖ−rₖ)² + 1 − Σrₖ². Only the first term depends on q; its minimum is zero at q = r. Skill = 1 − model loss/baseline loss; the paper's baseline uses full-bank frequencies 89/300, 111/300 and 100/300 on the same eligible subset. For toy correct scores 0.9,0.6 and incorrect scores 0.8,0.6, pair credit is 1 + 1 + 0 + 0.5 = 2.5 of 4, so AUROC = 0.625. Reliability uses ten fixed bins; the last includes 1.0. Bin counts alone cannot reconstruct average confidence or correctness in each bin.
Worked activity · teaching example
Compare certainty with correctness
Imagine ten predictions, each reporting 90% certainty. Change how many were correct. The diagonal means reported certainty equals the observed correct fraction.
Browser teaching tool
Calculate Brier loss
Select Wrong one-hot, then Correct one-hot with reference Hold. Loss changes from 2 to 0.
Use percentages with up to two decimal places, totalling exactly 100.00%. This teaching contract is separate from the paper’s raw-response tolerance.
Check your understanding · optional
Which vector has zero Brier loss when Hold is the reference?
Take this forward
Next, classify the possible consequences of an incorrect decision.
Paper and reference material
Original paper: §III.C; companion mathematics D.5–D.9 · page 2 (new tab)
The teaching explanation and interactive examples are companion additions. Recorded study values and frozen references remain unchanged.