TrustCyber / machine
/research/allow-hold-deny/lessons/12-accuracy-results/index.mdtext/markdown5.5 KB ~1,397 tokens
Open raw

Read the accuracy results

Explain why coverage and class-level results matter beside accuracy.

Canonical HTML page

Learning objective

Explain why coverage and class-level results matter beside accuracy.

Read counts before rounded percentages

Every primary score uses the same denominator: 300 assigned cases.

CLM, JEV and KEV supplied 300 valid responses. GLM supplied 267, leaving 33 invalid. Its valid-only accuracy is 260/267 = 97.38%, but coverage is 89.00%.

Inspect the two failure patterns

CLM selected Hold on every case. It matched all 111 Hold references and missed the 89 Allow and 100 Deny references. Its decisions match an always-Hold baseline. The study cannot identify whether task representation, model scoring, implementation or their interaction caused that pattern.

GLM's missing answers ended at the completion limit. Those responses are neither valid Hold answers nor transport timeouts. Their omission is uneven across reference classes and selected difficult-looking subsets. High accuracy on completed answers needs to be read together with failed completion.

The matrix explorer includes GLM's separately derived matrix from deposited observations. That full matrix was not printed in the manuscript. Keep this companion derivation distinct from Table II, which compares JEV answers with KEV answers rather than either model with the key.

Go deeper: reading Figure 1

Accuracy bars are Wilson 95% intervals over all 300 assigned cases. The horizontal coordinate is compliant-response median HTTP time. Circles are local and squares hosted. Shared axes support reading the plot, but the experiment does not isolate model-compute speed. Table I includes accuracy, intervals and compliant-response timing counts.

Read the original visual evidence

All 300 assigned cases determine accuracy; compliant-response timings use 300 native responses and 267 GLM responses.
Figure 1 · Speed and accuracy. All 300 assigned cases determine accuracy; compliant-response timings use 300 native responses and 267 GLM responses. Select the image for full resolution.
Accuracy, compliant-response counts, Wilson intervals and timing for the four tested configurations.
Table I · Primary results. Accuracy, compliant-response counts, Wilson intervals and timing for the four tested configurations. Select the image for full resolution.
Read Table I as text
  • CLM: 111/300 correct, 37.00%; Wilson 31.73–42.60%; 300 timed compliant responses; median 1113.2 ms, p95 1183.1 ms.
  • JEV: 272/300 correct, 90.67%; Wilson 86.84–93.46%; 300 timed compliant responses; median 139.1 ms, p95 211.1 ms.
  • KEV: 273/300 correct, 91.00%; Wilson 87.22–93.74%; 300 timed compliant responses; median 659.7 ms, p95 678.3 ms.
  • GLM: 260/300 correct, 86.67%; Wilson 82.35–90.05%; 267 timed compliant responses; median 2966.5 ms, p95 5146.6 ms.

Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

Why must GLM 97.38% be accompanied by coverage?

Take this forward

Next, unpack how one net correct case can conceal many different decisions.

Reference material

Lesson navigation

Companion navigation