TrustCyber / machine
/research/allow-hold-deny/lessons/06-valid-responses/index.mdtext/markdown3.7 KB ~935 tokens
Open raw

What counts as a response?

Separate decision validity, distribution validity and full compliance.

Canonical HTML page

Learning objective

Separate decision validity, distribution validity and full compliance.

Completion comes before correctness

An answer can be correct only if the required decision arrives in a valid response. A response contract is the set of required fields and rules for an answer. JSON is a text format for named fields and values. Decision validity checks that the complete answer comes from the expected model, identifies the right case and contains an unambiguous lowercase decision.

Distribution validity adds three finite numeric probabilities between zero and one, an acceptable sum, and a selected decision at a reported maximum. Full compliance adds format rules, including GLM's instruction to choose the first supplied label on an exact tie. These are separate checks even though their counts happened to coincide in this experiment.

Try a small contract

The toy object below uses case_id TEACHING and decision hold. Changing Hold to Allow changes the decoded decision. Removing the closing brace makes the JSON incomplete. It supplies no valid decision, even if a human can guess the intended text. This example checks the JSON text only. The study also checks whether a complete response arrived from the expected model and whether its probabilities meet the required rules.

The study repaired no JSON, normalized no probabilities, salvaged no prefixes and made no automatic retries. GLM's 33 incomplete outputs stayed invalid in the assigned-case denominator.

Go deeper: decimal tolerance

The recorded rule is τ(d) = max(1.5 × 10⁻ᵈ, 10⁻¹⁵), and |sum(p) − 1| must not exceed τ(d). The terms are compared, not added. Precision comes from original expanded decimal spellings, with native floors of two places for JEV and four for KEV. Accepted nonexact vectors remain unchanged. The mathematics reference explains the calculation in D.12.

Interactive activities

Use the canonical HTML page for controls, live results and reset buttons.

Check your understanding (optional)

An output ends at its token cap before completing JSON. How is it counted?

Take this forward

Next, calculate scores while retaining missing responses.

Reference material

Lesson navigation

Companion navigation