# Agreement can hide shared errors > Reconcile agreement, disagreements and the net accuracy difference. [Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/13-paired-comparisons/) - Author: Junior Williams - Type: Research companion addition ## Learning objective Reconcile agreement, disagreements and the net accuracy difference. - [Original paper: §V.C; Table II; planned contrasts in §V.A · page 4](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=4) Source pages: 4, 5 ## Keep each case paired First sort the same 300 cases into four groups: - Both correct: 264. - Both wrong: 19. - Only KEV correct: 9. - Only JEV correct: 8. The first two groups are the 283 agreements. The last two are the 17 disagreements. Agreement was 283/300 = 94.33%. Correctness needs the reference; agreement alone does not establish it. ## Account for the one-case advantage KEV's total is 264 + 9 = 273; JEV's is 264 + 8 = 272. Their net difference is (9−8)/300 = 0.33 percentage points. Table II is a model-agreement matrix, so its diagonal includes shared mistakes as well as shared correct answers. In a **paired bootstrap**, sample case IDs with replacement and retain both answers for each selected case. Some cases appear more than once and others disappear. Repeating this shows resampling sensitivity within the bank. Keeping the pair preserves the relationship between models' outcomes on the same case. The exercise constructs 300 pairs consistent with the aggregate counts. It illustrates the mechanics using its own deterministic generator. It does not reproduce the original deposited-order bootstrap.
Go deeper: planned contrasts and descriptive follow-up

The original 10,000 shared resamples used seed 20260927, without stratification. Primary native-minus-GLM contrasts were CLM −49.67 points (−56.67,−42.33), JEV +4.00 (+0.33,+7.34), KEV +4.33 (+0.67,+8.00); secondary CLM-minus-KEV was −54.00 (−61.00,−46.67). Intervals are marginal, not simultaneous. The JEV–KEV comparison was added descriptively after primary results; it is not a prespecified superiority test. D.10 explains why shared correctness cancels in the point difference.

## Read the original visual evidence
Rows are JEV decisions and columns KEV decisions. The diagonal counts agreement, including shared errors.
Table II · Decision agreement. Rows are JEV decisions and columns KEV decisions. The diagonal counts agreement, including shared errors. Select the image for full resolution.
Read every Table II cell as text

Rows are JEV answers; each list gives KEV Allow, Hold and Deny counts in that order. JEV Allow: 94, 1, 0 (total 95). JEV Hold: 3, 81, 5 (total 89). JEV Deny: 0, 8, 108 (total 116). Column totals are 97, 90 and 113. The diagonal 94 + 81 + 108 = 283 counts agreement, including 19 shared errors.

## Browser teaching tools The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations. - [disagreements](https://trustcyber.ca/research/allow-hold-deny/lessons/13-paired-comparisons/#explore-disagreements): Choose JEV only correct, then C207. Read its rule and facts before comparing the two answers. ## Interactive activities Use the canonical HTML page for controls, live results and reset buttons. - [bootstrap](https://trustcyber.ca/research/allow-hold-deny/lessons/13-paired-comparisons/#demo-bootstrap): Choose 1,000 or 10,000 resamples, then select Calculate. Every draw keeps the two answers for each teaching pair together. The aggregate counts match the paper; the constructed pairs and random generator do not reproduce its recorded bootstrap. ## Check your understanding (optional) How can KEV win nine disagreements yet have only one more correct case? - JEV wins eight of the other disagreements: Correct. The offset is nine minus eight, leaving one net case. - The 19 shared errors were removed: All 300 assigned cases remain. - Agreement automatically means correctness: Shared errors are part of agreement. ## Take this forward Next, compare reported certainty on those same cases. ## Reference material - [paired-comparisons](https://trustcyber.ca/research/allow-hold-deny/paired-comparisons/index.md) - [mathematics](https://trustcyber.ca/research/allow-hold-deny/mathematics/index.md) ## Lesson navigation - [Previous: Read the accuracy results](https://trustcyber.ca/research/allow-hold-deny/lessons/12-accuracy-results/index.md) - [Next: A confident answer can be wrong](https://trustcyber.ca/research/allow-hold-deny/lessons/14-certainty-results/index.md) ## Companion navigation - [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md) - [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)