Lesson 12 of 18

Read the accuracy results

Step 12 in the guided paper walkthrough.

By the end of this lesson

Explain why coverage and class-level results matter beside accuracy.

Read counts before rounded percentages

Every primary score uses the same denominator: 300 assigned cases.

KEV: 273 correct, or 91.00%.
JEV: 272 correct, or 90.67%.
GLM: 260 correct, or 86.67%.
CLM: 111 correct, or 37.00%.

CLM, JEV and KEV supplied 300 valid responses. GLM supplied 267, leaving 33 invalid. Its valid-only accuracy is 260/267 = 97.38%, but coverage is 89.00%.

Inspect the two failure patterns

CLM selected Hold on every case. It matched all 111 Hold references and missed the 89 Allow and 100 Deny references. Its decisions match an always-Hold baseline. The study cannot identify whether task representation, model scoring, implementation or their interaction caused that pattern.

GLM’s missing answers ended at the completion limit. Those responses are neither valid Hold answers nor transport timeouts. Their omission is uneven across reference classes and selected difficult-looking subsets. High accuracy on completed answers needs to be read together with failed completion.

The matrix explorer includes GLM’s separately derived matrix from deposited observations. That full matrix was not printed in the manuscript. Keep this companion derivation distinct from Table II, which compares JEV answers with KEV answers rather than either model with the key.

Go deeper: reading Figure 1

Accuracy bars are Wilson 95% intervals over all 300 assigned cases. The horizontal coordinate is compliant-response median HTTP time. Circles are local and squares hosted. Shared axes support reading the plot, but the experiment does not isolate model-compute speed. Table I includes accuracy, intervals and compliant-response timing counts.

Read the original visual evidence

Figure 1 · Speed and accuracy. All 300 assigned cases determine accuracy; compliant-response timings use 300 native responses and 267 GLM responses. Select the image for full resolution.

Table I · Primary results. Accuracy, compliant-response counts, Wilson intervals and timing for the four tested configurations. Select the image for full resolution.

Read Table I as text

CLM: 111/300 correct, 37.00%; Wilson 31.73–42.60%; 300 timed compliant responses; median 1113.2 ms, p95 1183.1 ms.
JEV: 272/300 correct, 90.67%; Wilson 86.84–93.46%; 300 timed compliant responses; median 139.1 ms, p95 211.1 ms.
KEV: 273/300 correct, 91.00%; Wilson 87.22–93.74%; 300 timed compliant responses; median 659.7 ms, p95 678.3 ms.
GLM: 260/300 correct, 86.67%; Wilson 82.35–90.05%; 267 timed compliant responses; median 2966.5 ms, p95 5146.6 ms.

Browser teaching tool

Change the accuracy denominator

Select GLM and change the denominator to valid responses. Compare 97.38% conditional accuracy with 89% coverage.

GLM primary accuracy: 260 / 300 = 86.67%. Coverage: 267 / 300 = 89.00%. Invalid: 33.

Conditional accuracy describes completed decisions and must be read with coverage.

Browser teaching tool

Inspect a confusion matrix

Select GLM. Trace correct cells on the diagonal and keep the invalid column in each reference-row total.

Model

Frozen references (rows) against returned decisions (columns)
Reference	Hold	Assigned
Allow	89	89
Hold	111	111
Deny	100	100

CLM balanced accuracy: 33.33%. All 300 assigned cases are retained.

Native matrices derive from section V.B–C and were checked against deposited observations. Table II is a separate model-agreement matrix.

Check your understanding · optional

Why must GLM 97.38% be accompanied by coverage?

You can try again or continue to the next lesson.

Take this forward

Next, unpack how one net correct case can conceal many different decisions.

Paper and reference material

Original paper: §V.A–B; Table I; Figure 1 · page 4 (new tab)

Source pages: 4 5

The teaching explanation and interactive examples are companion additions. Recorded study values and frozen references remain unchanged.