Lesson 16 of 18
What the evidence cannot show
Step 16 in the guided paper walkthrough.
By the end of this lesson
Explain the major limits without changing the recorded results.
Separate an observation from its cause
CLM always chose Hold in the tested MPS configuration. That observation does not isolate whether the model, task representation, implementation or their interaction caused it. Hosted and local timing mixes hardware, framework, networking and provider conditions. A larger GLM completion cap could alter several outcomes, but this run does not estimate them.
Inspect the reference limitation
All erroneous allows fell among eleven pre-run changed references. The review treats an unestablished condition as requiring Hold. Some unchanged Allow cases also leave conditions implicit: C270 mentions owner-approved purchase in its rule without separately affirming approval in the facts; C164 requires verified queue length but states that the trigger is exceeded. The review does not establish consistent treatment of every such wording.
These examples identify an unresolved reference-consistency issue. They do not supply newly adjudicated labels. The reference remains frozen, and a scored error means disagreement with that key.
The case bank was AI-drafted and single-author reviewed. One case-level bootstrap does not account for shared authorship, scenario similarity, alternative reasonable readings or repeated-run variation. No proposed action was executed, and neither correctness nor certainty certifies deployment safety.
Go deeper: what another study would need
Independent human adjudication, new cases and repeated sessions could address different sources of uncertainty. Randomizing arm order and observing provider load could improve timing interpretation. These are future design choices; none should be described as completed in this experiment.
Try it
Explore the idea
Choose a claim and inspect its limit.
Check your understanding · optional
What should we do with C164/C270 while explaining this paper?
Take this forward
Next, state the conclusion with its evidence and disclosures.
Paper and reference material
Original paper: §VI.A–C · page 7 (new tab)
The teaching explanation and interactive examples are companion additions. Recorded study values and frozen references remain unchanged.