# Ideas behind the experiment

> Distinguish authorization, selective classification and calibration.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/03-related-ideas/)

- Author: Junior Williams
- Type: Research companion addition

## Learning objective

Distinguish authorization, selective classification and calibration.

- [Original paper: §II · page 1](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=1)
Source pages: 1, 2
## Three nearby ideas

**Authorization** means deciding whether a subject may perform an operation on a resource under a policy. For example, a staff member may read a particular workbook only when permission covers that workbook. Attribute-based authorization considers properties of the person, resource, operation and environment.

**Selective classification** lets a predictor decline some inputs, often using a confidence threshold. Its **coverage** is the fraction on which it supplies an answer. The paper does not fit such a rejection threshold. Hold is a substantive answer about missing authorization conditions, even if the model is very certain.

**Calibration** compares reported probabilities with observed outcomes across a group. If many predictions each report 80% certainty, around 80% being correct would be consistent with calibration for that group. One confident correct answer cannot establish calibration.

## Work through the distinction

A model returns Hold at 99% because permission for the target workbook is unresolved. That is a complete policy decision. A model returns an unfinished response with no decision. That is a delivery failure. Treating both as abstentions would lose the distinction the experiment measures.

<details><summary>Go deeper: how the sources relate</summary><p>The paper draws on attribute-based access control, selective classification, Brier scoring and calibration research. Its earlier JEV study uses a separate case bank and protocol. The two studies' results are not pooled. See the reference chapter for the bibliography.</p></details>

## Explore the idea

Select an observation and identify the concept.
- Permission for this workbook is unconfirmed: Authorization: a prerequisite remains unresolved, so the policy may require Hold.
- A response stops before its decision: Response completion: this is invalid output, separate from Hold.
- 90 of 100 predictions at 90% are correct: Calibration: the observed fraction matches reported certainty in this illustrative group.

## Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

## Check your understanding (optional)

Is a highly certain Hold answer automatically invalid?
- No; Hold can be a complete, correct policy answer: Correct. Hold identifies an unresolved prerequisite. Validity asks whether the required answer format was delivered.
- Yes; Hold always means the model refused to predict: No confidence-based rejection rule was fitted here.
- Yes; uncertainty must always be Deny: Deny requires an applicable express prohibition.

## Take this forward

Next, apply the actual Allow/Hold/Deny rule.

## Reference material

- [sources](https://trustcyber.ca/research/allow-hold-deny/sources/index.md)

## Lesson navigation

- [Previous: What is the paper asking?](https://trustcyber.ca/research/allow-hold-deny/lessons/02-the-research-question/index.md)
- [Next: Allow, Hold or Deny](https://trustcyber.ca/research/allow-hold-deny/lessons/04-policy-decisions/index.md)

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)