What reported certainty can tell us

Start with the probability’s origin

CLM, JEV, and KEV supplied native classifier distributions. GLM supplied estimates as generated JSON. All can be scored against the reviewed key, but the study does not establish identical training objectives, internal meanings, or calibrated deployment-risk estimates.

Chosen-answer probability is the probability attached to the selected label. Correctness is agreement with the frozen key. Calibration describes a relationship between reported probabilities and observed frequencies across groups of decisions. One high-probability answer cannot demonstrate calibration.

Three measurements

The following values are reported in Table III and were checked against the deposited eligible records.

Arm	Probability-eligible n	Brier	Brier skill	Correctness AUROC
CLM	300	0.9724	-0.4645	0.5701
JEV	300	0.1519	0.7712	0.8196
KEV	300	0.1395	0.7900	0.7900
GLM	267	0.0502	0.9246	0.7335

Brier score averages the sum of squared errors across all three probabilities. The paper does not divide by three. Lower is better. For an ideal nonnegative vector summing exactly to one, loss ranges from zero for a correct one-hot forecast to two for a wrong one-hot forecast. That exact-simplex statement does not authorize normalizing the paper’s accepted nonexact sums.

Brier skill compares that loss with one fixed forecast, (89, 111, 100)/300, evaluated on the same eligible observations as the model. The forecast stays fixed even for GLM’s 267-case subset. Skill is 1 − model loss / baseline loss. A negative value is worse than that baseline. Using rounded table values can change the last printed digit.

Correctness AUROC ranks selected probabilities on correct decisions against those on incorrect decisions, with half credit for ties. It does not measure calibration, and it is undefined when either group is empty. KEV’s skill and AUROC both print as 0.7900 after rounding; they are different statistics with different unrounded values.

Read GLM’s row with its eligibility set

GLM’s Brier score of 0.0502 describes its 267 completed distributions, with only seven incorrect decisions. Its 33 invalid outputs have no usable full distribution and do not enter that probability score. The row cannot be read as a probability-quality measurement for all 300 assigned cases. The selected-subset comparisons on the paired-comparisons page help explain the denominator difference.

Work through C207

The frozen reference is hold. Table IV displays JEV as (47%, 53%, 0%) and KEV as (92.03%, 6.62%, 1.35%). JEV’s loss is 0.47² + (0.53 − 1)² + 0² = 0.4418. KEV’s displayed values give approximately 1.71911678. These calculations use rounded percentages to teach the score; the paper’s aggregate losses use original numbers.

The browser calculator accepts percentages with up to two decimal places and an exact total of 100.00%. That deliberately simple teaching contract differs from the original decimal-tolerance validator. It rejects an invalid input rather than silently normalizing it.

Reliability diagrams need more than counts

Figure 3 groups selected probabilities into ten bins, with one included in the last bin. Each point uses the mean selected probability and observed correctness within that bin. Empty bins have no point. A diagonal represents equality between those two quantities.

The manuscript prints bin counts, not all numeric coordinates. Counts alone cannot reconstruct a reliability curve. This reader preserves the original figure, while the probability lesson uses a labelled synthetic example. The deposited records are available separately for analysis.

JEV’s native vectors comprise 123 one-hot, 172 two-positive, and five three-positive vectors. KEV’s all have three positive components. JEV’s one-hot errors include C090 and C298. An extreme reported probability can accompany an error.

Original paper, sections III.C and V.E–F · Mathematics and proofs