TrustCyber / machine
/research/allow-hold-deny/lessons/03-related-ideas/index.mdtext/markdown3.6 KB ~924 tokens
Open raw

Ideas behind the experiment

Distinguish authorization, selective classification and calibration.

Canonical HTML page

Learning objective

Distinguish authorization, selective classification and calibration.

Three nearby ideas

Authorization means deciding whether a subject may perform an operation on a resource under a policy. For example, a staff member may read a particular workbook only when permission covers that workbook. Attribute-based authorization considers properties of the person, resource, operation and environment.

Selective classification lets a predictor decline some inputs, often using a confidence threshold. Its coverage is the fraction on which it supplies an answer. The paper does not fit such a rejection threshold. Hold is a substantive answer about missing authorization conditions, even if the model is very certain.

Calibration compares reported probabilities with observed outcomes across a group. If many predictions each report 80% certainty, around 80% being correct would be consistent with calibration for that group. One confident correct answer cannot establish calibration.

Work through the distinction

A model returns Hold at 99% because permission for the target workbook is unresolved. That is a complete policy decision. A model returns an unfinished response with no decision. That is a delivery failure. Treating both as abstentions would lose the distinction the experiment measures.

Go deeper: how the sources relate

The paper draws on attribute-based access control, selective classification, Brier scoring and calibration research. Its earlier JEV study uses a separate case bank and protocol. The two studies' results are not pooled. See the reference chapter for the bibliography.

Explore the idea

Select an observation and identify the concept.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

Is a highly certain Hold answer automatically invalid?

Take this forward

Next, apply the actual Allow/Hold/Deny rule.

Reference material

Lesson navigation

Companion navigation