# What the evidence cannot show

> Explain the major limits without changing the recorded results.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/16-limits-of-the-claims/)

- Author: Junior Williams
- Type: Research companion addition

## Learning objective

Explain the major limits without changing the recorded results.

- [Original paper: §VI.A–C · page 7](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=7)
Source pages: 7, 8
## Separate an observation from its cause

CLM always chose Hold in the tested MPS configuration. That observation does not isolate whether the model, task representation, implementation or their interaction caused it. Hosted and local timing mixes hardware, framework, networking and provider conditions. A larger GLM completion cap could alter several outcomes, but this run does not estimate them.

## Inspect the reference limitation

All erroneous allows fell among eleven pre-run changed references. The review treats an unestablished condition as requiring Hold. Some unchanged Allow cases also leave conditions implicit: C270 mentions owner-approved purchase in its rule without separately affirming approval in the facts; C164 requires verified queue length but states that the trigger is exceeded. The review does not establish consistent treatment of every such wording.

These examples identify an unresolved reference-consistency issue. They do not supply newly adjudicated labels. The reference remains frozen, and a scored error means disagreement with that key.

The case bank was AI-drafted and single-author reviewed. One case-level bootstrap does not account for shared authorship, scenario similarity, alternative reasonable readings or repeated-run variation. No proposed action was executed, and neither correctness nor certainty certifies deployment safety.

<details><summary>Go deeper: what another study would need</summary><p>Independent human adjudication, new cases and repeated sessions could address different sources of uncertainty. Randomizing arm order and observing provider load could improve timing interpretation. These are future design choices; none should be described as completed in this experiment.</p></details>

## Explore the idea

Choose a claim and inspect its limit.
- KEV had one more correct decision: Supported on the frozen 300-case bank; not general superiority.
- JEV will always respond faster: Not established: one session mixes service, host and network conditions.
- Changing GLM budget fixes its failures: Unmeasured counterfactual: another cap would be another arm.
- All references have independent consensus: Not established: one human reviewed the key.

## Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

## Check your understanding (optional)

What should we do with C164/C270 while explaining this paper?
- Describe the unresolved issue and preserve the frozen labels: Correct. The paper identifies a possible inconsistency without re-adjudicating those cases.
- Silently relabel them to improve consistency: That would alter the benchmark being explained.
- Hide them because they complicate the result: They are a central limit on interpreting the reference key.

## Take this forward

Next, state the conclusion with its evidence and disclosures.

## Reference material

- [limits](https://trustcyber.ca/research/allow-hold-deny/limits/index.md)

## Lesson navigation

- [Previous: Compare speed and consequences](https://trustcyber.ca/research/allow-hold-deny/lessons/15-speed-and-tradeoffs/index.md)
- [Next: State the conclusion responsibly](https://trustcyber.ca/research/allow-hold-deny/lessons/17-conclusion-and-evidence/index.md)

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)