What counts as a response?

Completion comes before correctness

An answer can be correct only if the required decision arrives in a valid response. A response contract is the set of required fields and rules for an answer. JSON is a text format for named fields and values. Decision validity checks that the complete answer comes from the expected model, identifies the right case and contains an unambiguous lowercase decision.

Distribution validity adds three finite numeric probabilities between zero and one, an acceptable sum, and a selected decision at a reported maximum. Full compliance adds format rules, including GLM’s instruction to choose the first supplied label on an exact tie. These are separate checks even though their counts happened to coincide in this experiment.

Try a small contract

The toy object below uses case_id TEACHING and decision hold. Changing Hold to Allow changes the decoded decision. Removing the closing brace makes the JSON incomplete. It supplies no valid decision, even if a human can guess the intended text. This example checks the JSON text only. The study also checks whether a complete response arrived from the expected model and whether its probabilities meet the required rules.

The study repaired no JSON, normalized no probabilities, salvaged no prefixes and made no automatic retries. GLM’s 33 incomplete outputs stayed invalid in the assigned-case denominator.