# Understand reported certainty

> Distinguish a selected-answer probability from demonstrated correctness.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/09-probability-quality/)

- Author: Junior Williams
- Type: Research companion addition

## Learning objective

Distinguish a selected-answer probability from demonstrated correctness.

- [Original paper: §III.C; companion mathematics D.5–D.9 · page 2](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=2)
Source pages: 2, 3
## A distribution describes three choices

The vector [0.47, 0.53, 0] assigns 47% to Allow, 53% to Hold and 0% to Deny. Its largest value selects Hold. A larger selected probability does not guarantee that the answer agrees with the reference. The native classifier probabilities and GLM's generated estimates have different origins.

## Score the entire vector

The **Brier loss** adds the squared differences between probabilities and the reference's **one-hot vector**, which has one at the reference class and zero elsewhere. With reference Hold, the target is [0,1,0]. Loss for [0.47,0.53,0] is 0.47² + (0.53−1)² + 0² = 0.4418. The exact target scores 0; complete certainty in wrong Allow scores 2. Lower is better, with no division by three.

**Brier skill** compares mean loss with a fixed baseline forecast. **Correctness AUROC** asks whether chosen-answer probability ranks correct decisions above incorrect ones; a tied pair receives half credit. **Reliability bins** group similar reported probabilities and compare them with observed correctness. These measurements answer different questions.

Imagine ten forecasts all at 90% certainty but only six correct. Their observed fraction is 60%, a 30-point gap. That toy group illustrates calibration; ten observations cannot establish general calibration.

<details><summary>Go deeper: loss, skill, rankings and bins</summary><p>Brier = mean Σ(pₖ−yₖ)². On an ideal three-choice simplex its range is 0–2, and its expected loss is minimized by reporting the true distribution. For true class probabilities r and reported q, expected loss = Σ(qₖ−rₖ)² + 1 − Σrₖ². Only the first term depends on q; its minimum is zero at q = r. Skill = 1 − model loss/baseline loss; the paper's baseline uses full-bank frequencies 89/300, 111/300 and 100/300 on the same eligible subset. For toy correct scores 0.9,0.6 and incorrect scores 0.8,0.6, pair credit is 1 + 1 + 0 + 0.5 = 2.5 of 4, so AUROC = 0.625. Reliability uses ten fixed bins; the last includes 1.0. Bin counts alone cannot reconstruct average confidence or correctness in each bin.</p></details>

## Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- [brier](https://trustcyber.ca/research/allow-hold-deny/lessons/09-probability-quality/#explore-brier): Select Wrong one-hot, then Correct one-hot with reference Hold. Loss changes from 2 to 0.

## Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.
- [reliability](https://trustcyber.ca/research/allow-hold-deny/lessons/09-probability-quality/#demo-reliability): Imagine ten predictions, each reporting 90% certainty. Change how many were correct. The diagonal means reported certainty equals the observed correct fraction.

## Check your understanding (optional)

Which vector has zero Brier loss when Hold is the reference?
- Allow 0%, Hold 100%, Deny 0%: Correct. It exactly matches the Hold one-hot vector.
- Allow 100%, Hold 0%, Deny 0%: Complete certainty in the wrong class gives loss 2.
- Any vector whose maximum is Hold: Selecting the right class does not make the entire vector equal to the target.

## Take this forward

Next, classify the possible consequences of an incorrect decision.

## Reference material

- [probability](https://trustcyber.ca/research/allow-hold-deny/probability/index.md)
- [mathematics](https://trustcyber.ca/research/allow-hold-deny/mathematics/index.md)

## Lesson navigation

- [Previous: What does the clock measure?](https://trustcyber.ca/research/allow-hold-deny/lessons/08-response-time/index.md)
- [Next: Different errors, different effects](https://trustcyber.ca/research/allow-hold-deny/lessons/10-error-consequences/index.md)

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)