# Allow, Hold, or Deny

> Learn when an AI agent should act, wait or refuse—and how this study compared four models.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/)

- Author: Junior Williams
- Original paper: v1.0.0, September 27, 2026

## Begin with the basics

No AI, statistics or programming background needed. Each lesson explains one idea, works through an example and lets you try it. Check your understanding when you want to; you can always move forward.

## What you will learn

- How permission differs from the ability to perform an action.
- How the study counts correct, incorrect and incomplete responses.
- What response time and reported certainty can tell you.
- How to read the results, cases and limitations together.

The examples run in this website. Change an input, inspect the result and reset it whenever you want. Your last lesson is saved in this browser when storage is available.
- [Start learning](https://trustcyber.ca/research/allow-hold-deny/lessons/01-ai-and-agents/index.md)
## Follow the paper

- [1. Meet models and agents](https://trustcyber.ca/research/allow-hold-deny/lessons/01-ai-and-agents/index.md): Explain how a model becomes part of a system that can act.
- [2. What is the paper asking?](https://trustcyber.ca/research/allow-hold-deny/lessons/02-the-research-question/index.md): Explain the research question without assuming a universal winner.
- [3. Ideas behind the experiment](https://trustcyber.ca/research/allow-hold-deny/lessons/03-related-ideas/index.md): Distinguish authorization, selective classification and calibration.
- [4. Allow, Hold or Deny](https://trustcyber.ca/research/allow-hold-deny/lessons/04-policy-decisions/index.md): Apply the policy in the correct order.
- [5. Build a reviewed case bank](https://trustcyber.ca/research/allow-hold-deny/lessons/05-cases-and-reference/index.md): Identify model inputs and the reference used to score them.
- [6. What counts as a response?](https://trustcyber.ca/research/allow-hold-deny/lessons/06-valid-responses/index.md): Separate decision validity, distribution validity and full compliance.
- [7. Accuracy and uncertainty](https://trustcyber.ca/research/allow-hold-deny/lessons/07-accuracy-and-uncertainty/index.md): Compute accuracy with the right denominator and interpret uncertainty.
- [8. What does the clock measure?](https://trustcyber.ca/research/allow-hold-deny/lessons/08-response-time/index.md): Distinguish one response from a scored loop and an entire session.
- [9. Understand reported certainty](https://trustcyber.ca/research/allow-hold-deny/lessons/09-probability-quality/index.md): Distinguish a selected-answer probability from demonstrated correctness.
- [10. Different errors, different effects](https://trustcyber.ca/research/allow-hold-deny/lessons/10-error-consequences/index.md): Classify the hypothetical effect of an incorrect decision.
- [11. How the experiment was run](https://trustcyber.ca/research/allow-hold-deny/lessons/11-experimental-design/index.md): Identify shared controls and remaining configuration differences.
- [12. Read the accuracy results](https://trustcyber.ca/research/allow-hold-deny/lessons/12-accuracy-results/index.md): Explain why coverage and class-level results matter beside accuracy.
- [13. Agreement can hide shared errors](https://trustcyber.ca/research/allow-hold-deny/lessons/13-paired-comparisons/index.md): Reconcile agreement, disagreements and the net accuracy difference.
- [14. A confident answer can be wrong](https://trustcyber.ca/research/allow-hold-deny/lessons/14-certainty-results/index.md): Read probability metrics with their eligibility sets and limits.
- [15. Compare speed and consequences](https://trustcyber.ca/research/allow-hold-deny/lessons/15-speed-and-tradeoffs/index.md): Interpret measured response times and explore illustrative error weights.
- [16. What the evidence cannot show](https://trustcyber.ca/research/allow-hold-deny/lessons/16-limits-of-the-claims/index.md): Explain the major limits without changing the recorded results.
- [17. State the conclusion responsibly](https://trustcyber.ca/research/allow-hold-deny/lessons/17-conclusion-and-evidence/index.md): Connect the findings to evidence, authorship and ethics.
- [18. Inspect cases and connect the ideas](https://trustcyber.ca/research/allow-hold-deny/lessons/18-appendices-and-synthesis/index.md): Use the complete case details and pinned evidence to explain a result.

## Reference material

The original paper remains unchanged. Explanations and teaching examples are companion additions. Reported values were checked against the public evidence deposit; new derivations are identified where they appear.
The study classifies written situations against a frozen reviewed key. It does not execute actions or establish deployment safety. There is no general winner across the tested configurations.
CLM: local Contrastive-LM; JEV: hosted TypeSafe; KEV: local Kev-4B; GLM: hosted GLM 5.3 FlashX.
- [Full experimental configurations](https://trustcyber.ca/research/allow-hold-deny/study-design/#what-differed-between-arms)
- [What allow, hold, and deny mean](https://trustcyber.ca/research/allow-hold-deny/policy/index.md): Start with the authority to act. A probability attached to a decision cannot supply missing permission.
- [Four configurations, one frozen case bank](https://trustcyber.ca/research/allow-hold-deny/study-design/index.md): Read the controls and the differences between configurations before comparing the results.
- [Accuracy, coverage, and response time](https://trustcyber.ca/research/allow-hold-deny/results/index.md): Start with all assigned cases. Then inspect which responses were eligible for each other measurement.
- [Compare the same cases](https://trustcyber.ca/research/allow-hold-deny/paired-comparisons/index.md): A one-case net advantage can sit alongside shared errors and much larger differences on individual cases.
- [Eleven prerequisites that changed the key](https://trustcyber.ca/research/allow-hold-deny/reference-changes/index.md): The reviewed key was fixed before inference. Read the cases where an initial allow became hold.
- [What reported certainty can tell us](https://trustcyber.ca/research/allow-hold-deny/probability/index.md): Accuracy, probability quality, ranking, and calibration measure different things. Keep their meanings distinct.
- [The calculations, one step at a time](https://trustcyber.ca/research/allow-hold-deny/mathematics/index.md): Definitions, derivations, and proofs explain the measurements. They do not add empirical claims to the paper.
- [Consequences and limits of the comparison](https://trustcyber.ca/research/allow-hold-deny/limits/index.md): Read the error types alongside the limits of the evidence before applying the results to a real system.
- [Read, explore, or reproduce](https://trustcyber.ca/research/allow-hold-deny/reproduction/index.md): Choose the level of work you need. Reading and browser exercises require no Python or model service.
- [Figures and tables from the paper](https://trustcyber.ca/research/allow-hold-deny/figures/index.md): Read the original visual evidence alongside the explanations, with full-resolution files available on demand.
- [Sources, rights, and glossary](https://trustcyber.ca/research/allow-hold-deny/sources/index.md): Find the study’s references, the companion’s provenance, and definitions used throughout the reader.

## Original research

- [Original paper PDF](https://trustcyber.ca/research/allow-hold-deny/original.pdf)
- [Manuscript DOI](https://doi.org/10.5281/zenodo.23005044)
- [Evidence deposit](https://doi.org/10.5281/zenodo.22998941)