TrustCyber / machine
/research/allow-hold-deny/lessons/13-paired-comparisons/index.mdtext/markdown5.0 KB ~1,267 tokens
Open raw

Agreement can hide shared errors

Reconcile agreement, disagreements and the net accuracy difference.

Canonical HTML page

Learning objective

Reconcile agreement, disagreements and the net accuracy difference.

Keep each case paired

First sort the same 300 cases into four groups:

The first two groups are the 283 agreements. The last two are the 17 disagreements. Agreement was 283/300 = 94.33%. Correctness needs the reference; agreement alone does not establish it.

Account for the one-case advantage

KEV's total is 264 + 9 = 273; JEV's is 264 + 8 = 272. Their net difference is (9−8)/300 = 0.33 percentage points. Table II is a model-agreement matrix, so its diagonal includes shared mistakes as well as shared correct answers.

In a paired bootstrap, sample case IDs with replacement and retain both answers for each selected case. Some cases appear more than once and others disappear. Repeating this shows resampling sensitivity within the bank. Keeping the pair preserves the relationship between models' outcomes on the same case.

The exercise constructs 300 pairs consistent with the aggregate counts. It illustrates the mechanics using its own deterministic generator. It does not reproduce the original deposited-order bootstrap.

Go deeper: planned contrasts and descriptive follow-up

The original 10,000 shared resamples used seed 20260927, without stratification. Primary native-minus-GLM contrasts were CLM −49.67 points (−56.67,−42.33), JEV +4.00 (+0.33,+7.34), KEV +4.33 (+0.67,+8.00); secondary CLM-minus-KEV was −54.00 (−61.00,−46.67). Intervals are marginal, not simultaneous. The JEV–KEV comparison was added descriptively after primary results; it is not a prespecified superiority test. D.10 explains why shared correctness cancels in the point difference.

Read the original visual evidence

Rows are JEV decisions and columns KEV decisions. The diagonal counts agreement, including shared errors.
Table II · Decision agreement. Rows are JEV decisions and columns KEV decisions. The diagonal counts agreement, including shared errors. Select the image for full resolution.
Read every Table II cell as text

Rows are JEV answers; each list gives KEV Allow, Hold and Deny counts in that order. JEV Allow: 94, 1, 0 (total 95). JEV Hold: 3, 81, 5 (total 89). JEV Deny: 0, 8, 108 (total 116). Column totals are 97, 90 and 113. The diagonal 94 + 81 + 108 = 283 counts agreement, including 19 shared errors.

Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

How can KEV win nine disagreements yet have only one more correct case?

Take this forward

Next, compare reported certainty on those same cases.

Reference material

Lesson navigation

Companion navigation