Lesson 7 of 18
Accuracy and uncertainty
Step 7 in the guided paper walkthrough.
By the end of this lesson
Compute accuracy with the right denominator and interpret uncertainty.
Count what was assigned
Primary accuracy is correct valid decisions divided by all assigned cases. Coverage is valid responses divided by assigned cases. Conditional accuracy uses valid responses only. Missing responses stay in primary accuracy.
Work the numbers
Suppose 300 cases are assigned, 250 answers are valid and all 250 are correct. Primary accuracy and coverage are both 250/300 = 83.33%; valid-only accuracy is 250/250 = 100%. Change both counts to 240: primary accuracy and coverage become 80%, valid-only accuracy remains 100%, and 60 responses are invalid.
A confusion matrix counts each reference class against each returned decision. Recall for a class is its correctly returned count divided by all cases in that reference class. Balanced accuracy averages the three recalls. Always returning Hold gets all Hold cases right and all Allow/Deny cases wrong: (0 + 1 + 0)/3 = 33.33%, even though ordinary accuracy is 111/300 = 37%.
A Wilson interval describes uncertainty around a case-level proportion under its statistical assumptions. Here, 95% describes the method: across repeated samples meeting its assumptions, about 95% of intervals would cover the underlying success rate. It is not a guarantee for the next action. More observations at the same success rate generally narrow it. It does not account for a different case author, alternate reference readings or repeated inference sessions.
Go deeper: identities and Wilson calculation
Primary accuracy = coverage × conditional accuracy when at least one response is valid. With zero valid responses, conditional accuracy is undefined. For p = correct/n and z ≈ 1.96, Wilson centre = (p + z²/(2n))/(1 + z²/n); half-width = z√(p(1−p)/n + z²/(4n²))/(1 + z²/n). The browser calculates the two bounds. See D.1–D.4 for notation and derivation.
Worked activity · teaching example
Change hypothetical counts
Change both correct and valid from 250 to 240. Then try zero valid responses with zero correct. These are hypothetical counts, separate from the study.
Worked activity · teaching example
Explore an uncertainty interval
Keep the same 90% success rate but change 90/100 to 270/300. Observe how the interval narrows with more observations.
Browser teaching tool
Change the accuracy denominator
Select GLM and change the denominator to valid responses. Compare 97.38% conditional accuracy with 89% coverage.
Conditional accuracy describes completed decisions and must be read with coverage.
Browser teaching tool
Inspect a confusion matrix
Select GLM. Trace correct cells on the diagonal and keep the invalid column in each reference-row total.
| Reference | Allow | Hold | Deny | Invalid | Assigned |
|---|---|---|---|---|---|
| Allow | 0 | 89 | 0 | 0 | 89 |
| Hold | 0 | 111 | 0 | 0 | 111 |
| Deny | 0 | 100 | 0 | 0 | 100 |
Native matrices derive from section V.B–C and were checked against deposited observations. Table II is a separate model-agreement matrix.
Check your understanding · optional
If 240 of 300 assigned cases are valid and all 240 are correct, what is primary accuracy?
Take this forward
Next, define exactly when the response clock starts and stops.
Paper and reference material
Original paper: §III.C; companion mathematics D.1–D.4 · page 2 (new tab)
Source pages: 2
The teaching explanation and interactive examples are companion additions. Recorded study values and frozen references remain unchanged.