# Consequences and limits of the comparison

> Read the error types alongside the limits of the evidence before applying the results to a real system.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/limits/)

- Author: Junior Williams
- Type: Research companion addition

## Different errors have different consequences

The paper classifies observed errors under a hypothetical authorization setting. It does not measure real harm, costs, or deployment outcomes.

| Arm | Unauthorized allow | Work refusal | Unnecessary escalation | Invalid |
| --- | --- | --- | --- | --- |
| CLM | 0 | 0 | 189 | 0 |
| JEV | 9 | 16 | 3 | 0 |
| KEV | 11 | 13 | 3 | 0 |
| GLM | 2 | 4 | 1 | 33 |

An unauthorized allow predicts allow for a hold or deny reference. A work refusal predicts deny for an allow or hold reference. An unnecessary escalation predicts hold for an allow or deny reference. A correct hold is not an unnecessary escalation. Invalid output stays separate.

The browser exercise adds user-chosen nonnegative weights. Its total is wU × U + wR × R + wE × E + wI × I. These are illustrative weights, not measured severity or a deployment recommendation. Equal weights reproduce the count of incorrect or invalid assigned cases.

For JEV minus KEV, the weighted difference is −2wU + 3wR because escalation and invalid counts are equal. JEV has a lower illustrative total when 2wU > 3wR; KEV has a lower total when that inequality reverses. The exercise shows how a declared objective can change a preference while the original observations stay fixed.

## What limits the claims

The benchmark contains a single authored synthetic case bank, one reviewer, and one sequential execution session. Its cases are not a random sample of all organizational policies or deployment situations. The confidence intervals describe case-level variation under the paper’s analysis; they do not quantify every reference ambiguity, session effect, or future deployment risk.

The case texts contain no executed actions or independently observed harms. A returned label does not prove safe agent behavior. Hold is a semantic result, not abstention by a fitted uncertainty threshold. Native and generated probabilities differ in origin.

The local and hosted configurations differ in hardware visibility, service implementation, precision, and response contract. The measured timing ratio cannot isolate architecture. GLM’s missing outputs are concentrated in some disputed or changed-reference cases, making completed-only summaries conditional on a selected set.

The paper reports exploratory, post hoc subsets separately from planned contrasts. The results do not establish a general winner across speed, accuracy, reported certainty, and hypothetical consequences.

## The unresolved reference question

The manuscript discusses consistency between C164 and C270, where the interpretation of approval or queue conditions may depend on what the facts establish implicitly. A second blinded policy review was not supplied. The original key and all primary scores remain unchanged. This reader flags that concern rather than choosing a new reference answer.

## Disclosures and attribution

Junior Williams is the sole author. The paper reports no external funding or disclosed financial relationship that influenced the work. This reader preserves that author disclosure; it is not an independent investigation of relationships.

Codex helped draft candidate cases, and the author reviewed and froze the key before inference. AI tools also assisted with implementation, analysis support, and prose review. In particular, JEV-assisted prose review of research involving JEV is not an independent peer review or blinded validation. The present companion is an editorial and teaching adaptation with a separate arithmetic audit, not a new experiment.

[Original paper, sections V.H, VI–VIII](/research/allow-hold-deny/original.pdf#page=7) · [Sources, rights, and reading glossary](/research/allow-hold-deny/sources/)

## Browser teaching tools

The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- [weights](https://trustcyber.ca/research/allow-hold-deny/limits/#explore-weights): Change unauthorized-allow weight from 1 to 5. Compare JEV 64 with KEV 71, then reset to equal weights.

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)