# What counts as a response?

> Separate decision validity, distribution validity and full compliance.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/06-valid-responses/)

- Author: Junior Williams
- Type: Research companion addition

## Learning objective

Separate decision validity, distribution validity and full compliance.

- [Original paper: §III.B · page 2](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=2)
Source pages: 2
## Completion comes before correctness

An answer can be correct only if the required decision arrives in a valid response. A response contract is the set of required fields and rules for an answer. JSON is a text format for named fields and values. Decision validity checks that the complete answer comes from the expected model, identifies the right case and contains an unambiguous lowercase decision.

Distribution validity adds three finite numeric probabilities between zero and one, an acceptable sum, and a selected decision at a reported maximum. Full compliance adds format rules, including GLM's instruction to choose the first supplied label on an exact tie. These are separate checks even though their counts happened to coincide in this experiment.

## Try a small contract

The toy object below uses case_id TEACHING and decision hold. Changing Hold to Allow changes the decoded decision. Removing the closing brace makes the JSON incomplete. It supplies no valid decision, even if a human can guess the intended text. This example checks the JSON text only. The study also checks whether a complete response arrived from the expected model and whether its probabilities meet the required rules.

The study repaired no JSON, normalized no probabilities, salvaged no prefixes and made no automatic retries. GLM's 33 incomplete outputs stayed invalid in the assigned-case denominator.

<details><summary>Go deeper: decimal tolerance</summary><p>The recorded rule is τ(d) = max(1.5 × 10⁻ᵈ, 10⁻¹⁵), and |sum(p) − 1| must not exceed τ(d). The terms are compared, not added. Precision comes from original expanded decimal spellings, with native floors of two places for JEV and four for KEV. Accepted nonexact vectors remain unchanged. The mathematics reference explains the calculation in D.12.</p></details>

## Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.
- [validity](https://trustcyber.ca/research/allow-hold-deny/lessons/06-valid-responses/#demo-validity): Change the toy decision to allow, then remove its closing brace. Compare a decoded answer with a malformed response.

## Check your understanding (optional)

An output ends at its token cap before completing JSON. How is it counted?
- Invalid output: Correct. No complete contract was delivered; this is separate from a valid Hold decision.
- Hold: Hold must be a recognized decision in a valid response.
- Correct if its beginning sounds right: The protocol prohibited partial-prefix salvage.

## Take this forward

Next, calculate scores while retaining missing responses.

## Reference material

- [study-design](https://trustcyber.ca/research/allow-hold-deny/study-design/index.md)
- [mathematics](https://trustcyber.ca/research/allow-hold-deny/mathematics/index.md)

## Lesson navigation

- [Previous: Build a reviewed case bank](https://trustcyber.ca/research/allow-hold-deny/lessons/05-cases-and-reference/index.md)
- [Next: Accuracy and uncertainty](https://trustcyber.ca/research/allow-hold-deny/lessons/07-accuracy-and-uncertainty/index.md)

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)