# Compare the same cases

> A one-case net advantage can sit alongside shared errors and much larger differences on individual cases.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/paired-comparisons/)

- Author: Junior Williams
- Type: Research companion addition

## Agreement is not correctness

JEV and KEV chose the same label on 283 of 300 cases (94.33%). That total includes shared mistakes. Their decisions differed on 17 cases: 15 hold references, two allow references, and no deny references.

| Paired correctness | Cases |
| --- | --- |
| Both correct | 264 |
| JEV only correct | 8 |
| KEV only correct | 9 |
| Both wrong | 19 |

The 264 shared successes and 19 shared failures cancel in KEV-minus-JEV primary accuracy. Nine KEV-only successes minus eight JEV-only successes give **1/300 = 0.33 percentage points**. That descriptive comparison is not a prespecified superiority finding.

The 19 shared errors are C044, C046, C076, C079, C083, C090, C093, C094, C107, C114, C161, C201, C210, C233, C249, C255, C265, C270, and C298. A close aggregate comparison can conceal a common failure pattern.

## The full disagreement set

The table transcribes manuscript Table IV. The explorer adds the rule, facts, action, and frozen rationale from the deposited reviewed case bank. Rounded displayed percentages remain separate from original response numbers and are not normalized.

| Case | Frozen reference | JEV decision | KEV decision |
| --- | --- | --- | --- |
| C033 | hold | hold | deny |
| C095 | hold | deny | hold |
| C125 | hold | deny | hold |
| C126 | hold | hold | deny |
| C140 | hold | deny | hold |
| C144 | hold | hold | deny |
| C164 | allow | hold | allow |
| C181 | hold | deny | hold |
| C187 | hold | hold | deny |
| C193 | hold | hold | deny |
| C207 | hold | hold | allow |
| C224 | allow | allow | hold |
| C228 | hold | hold | allow |
| C243 | hold | deny | hold |
| C252 | hold | deny | hold |
| C258 | hold | deny | hold |
| C291 | hold | deny | hold |

C207 and C228 are especially useful. JEV returned hold at selected probabilities 53% and 51%, while KEV returned allow at 92.03% and 76.56%. Their frozen references are hold. The reference-change page explains the prerequisites that were not established.

C164 and C224 have allow references. The paper raises a reference-consistency question involving C164 and shared-error case C270: prerequisites may be treated as implicit rather than separately established. This reader preserves the frozen labels and does not resolve that ambiguity by changing the key.

## Planned paired contrasts

The paper resampled aligned cases 10,000 times with seed 20260927. Each resample retains repeated indices and uses the same selected indices for both models. These percentage-point intervals are the paper’s planned marginal percentile intervals, independently reproduced from the deposited correctness observations with NumPy’s generator and linear quantiles.

| Contrast | Difference | 95% interval | Status |
| --- | --- | --- | --- |
| CLM minus GLM | −49.67 pp | −56.67 to −42.33 pp | Primary |
| JEV minus GLM | +4.00 pp | +0.33 to +7.34 pp | Primary |
| KEV minus GLM | +4.33 pp | +0.67 to +8.00 pp | Primary |
| CLM minus KEV | −54.00 pp | −61.00 to −46.67 pp | Secondary |

The [paired-comparisons lesson](/research/allow-hold-deny/lessons/13-paired-comparisons/) uses a grouped reconstruction of the reported joint counts and a deterministic teaching generator. Its samples are not the manuscript’s original NumPy draws. A seed without its generator, ordering, and interpolation convention is insufficient for exact reproduction.

## Selected subsets answer different questions

Removing the eleven changed-reference cases leaves 289 cases. JEV gets 270 correct (93.43%) and KEV 273 (94.46%). Their reported Brier scores are 0.0997 and 0.0810; correctness AUROC is 0.9094 and 0.9367. That AUROC order reverses the full-bank order. AUROC is pairwise, so this is not subtraction of eleven additive losses. The analysis is post hoc.

On GLM’s common 267 completed cases, JEV has 252 correct, KEV 253, and GLM 260. JEV and KEV each got 20 of the omitted 33 cases correct. Reported Brier differences against GLM on the common subset are +0.03823 for JEV and +0.03152 for KEV. Comparing only completed cases conditions on a selected subset; it does not undo GLM’s operational completion failures.

[Original paper, Table II, Table IV, sections V.C–F and Appendix A](/research/allow-hold-deny/original.pdf#page=5)

## Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- [disagreements](https://trustcyber.ca/research/allow-hold-deny/paired-comparisons/#explore-disagreements): Choose JEV only correct, then C207. Read its rule and facts before comparing the two answers.

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)