TrustCyber / machine
/research/allow-hold-deny/lessons/07-accuracy-and-uncertainty/index.mdtext/markdown4.6 KB ~1,162 tokens
Open raw

Accuracy and uncertainty

Compute accuracy with the right denominator and interpret uncertainty.

Canonical HTML page

Learning objective

Compute accuracy with the right denominator and interpret uncertainty.

Count what was assigned

Primary accuracy is correct valid decisions divided by all assigned cases. Coverage is valid responses divided by assigned cases. Conditional accuracy uses valid responses only. Missing responses stay in primary accuracy.

Work the numbers

Suppose 300 cases are assigned, 250 answers are valid and all 250 are correct. Primary accuracy and coverage are both 250/300 = 83.33%; valid-only accuracy is 250/250 = 100%. Change both counts to 240: primary accuracy and coverage become 80%, valid-only accuracy remains 100%, and 60 responses are invalid.

A confusion matrix counts each reference class against each returned decision. Recall for a class is its correctly returned count divided by all cases in that reference class. Balanced accuracy averages the three recalls. Always returning Hold gets all Hold cases right and all Allow/Deny cases wrong: (0 + 1 + 0)/3 = 33.33%, even though ordinary accuracy is 111/300 = 37%.

A Wilson interval describes uncertainty around a case-level proportion under its statistical assumptions. Here, 95% describes the method: across repeated samples meeting its assumptions, about 95% of intervals would cover the underlying success rate. It is not a guarantee for the next action. More observations at the same success rate generally narrow it. It does not account for a different case author, alternate reference readings or repeated inference sessions.

Go deeper: identities and Wilson calculation

Primary accuracy = coverage × conditional accuracy when at least one response is valid. With zero valid responses, conditional accuracy is undefined. For p = correct/n and z ≈ 1.96, Wilson centre = (p + z²/(2n))/(1 + z²/n); half-width = z√(p(1−p)/n + z²/(4n²))/(1 + z²/n). The browser calculates the two bounds. See D.1–D.4 for notation and derivation.

Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

If 240 of 300 assigned cases are valid and all 240 are correct, what is primary accuracy?

Take this forward

Next, define exactly when the response clock starts and stops.

Reference material

Lesson navigation

Companion navigation