Accuracy and uncertainty
Compute accuracy with the right denominator and interpret uncertainty.
- Author: Junior Williams
- Type: Research companion addition
Learning objective
Compute accuracy with the right denominator and interpret uncertainty.
- Original paper: §III.C; companion mathematics D.1–D.4 · page 2 Source pages: 2
Count what was assigned
Primary accuracy is correct valid decisions divided by all assigned cases. Coverage is valid responses divided by assigned cases. Conditional accuracy uses valid responses only. Missing responses stay in primary accuracy.
Work the numbers
Suppose 300 cases are assigned, 250 answers are valid and all 250 are correct. Primary accuracy and coverage are both 250/300 = 83.33%; valid-only accuracy is 250/250 = 100%. Change both counts to 240: primary accuracy and coverage become 80%, valid-only accuracy remains 100%, and 60 responses are invalid.
A confusion matrix counts each reference class against each returned decision. Recall for a class is its correctly returned count divided by all cases in that reference class. Balanced accuracy averages the three recalls. Always returning Hold gets all Hold cases right and all Allow/Deny cases wrong: (0 + 1 + 0)/3 = 33.33%, even though ordinary accuracy is 111/300 = 37%.
A Wilson interval describes uncertainty around a case-level proportion under its statistical assumptions. Here, 95% describes the method: across repeated samples meeting its assumptions, about 95% of intervals would cover the underlying success rate. It is not a guarantee for the next action. More observations at the same success rate generally narrow it. It does not account for a different case author, alternate reference readings or repeated inference sessions.
Go deeper: identities and Wilson calculation
Primary accuracy = coverage × conditional accuracy when at least one response is valid. With zero valid responses, conditional accuracy is undefined. For p = correct/n and z ≈ 1.96, Wilson centre = (p + z²/(2n))/(1 + z²/n); half-width = z√(p(1−p)/n + z²/(4n²))/(1 + z²/n). The browser calculates the two bounds. See D.1–D.4 for notation and derivation.
Browser teaching tools
The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- accuracy (Human view): Select GLM and change the denominator to valid responses. Compare 97.38% conditional accuracy with 89% coverage.
- matrix (Human view): Select GLM. Trace correct cells on the diagonal and keep the invalid column in each reference-row total.
Interactive activities
Use the canonical HTML page for controls, live results and reset buttons.
- counts (Human view): Change both correct and valid from 250 to 240. Then try zero valid responses with zero correct. These are hypothetical counts, separate from the study.
- wilson (Human view): Keep the same 90% success rate but change 90/100 to 270/300. Observe how the interval narrows with more observations.
Check your understanding (optional)
If 240 of 300 assigned cases are valid and all 240 are correct, what is primary accuracy?
- 80%: Correct. Primary accuracy retains every assigned case: 240/300.
- 100%: 100% is valid-only accuracy; it omits 60 invalid outputs.
- 60%: 60 is the invalid count, not the accuracy percentage.
Take this forward
Next, define exactly when the response clock starts and stops.