Four configurations, one frozen case bank

What was held in common

Each arm received the same 300 scored cases, in the same saved order. The frozen key contains 89 allow, 111 hold, and 100 deny references. Eleven proposed allow labels were changed to hold during pre-run review. A rationale for C071 was corrected while its deny label stayed the same.

Each arm also received five disjoint, unscored practice cases. The resulting deposit contains 1,200 scored records and 20 practice records, along with preparation and analysis artifacts. Practice success is not scored completion.

Six permutations of the three choice labels appear 50 times each. The saved order uses seed 20260927. There was one outstanding request per arm, a 60-second timeout, and no automatic retry, output repair, or rescoring. GLM used a fresh context for each case.

What differed between arms

Arm	Tested configuration	Probability origin
CLM	v0.1-8B, local BF16 Qwen3-8B, unmerged MPS implementation	Native classifier output
JEV	Hosted TypeSafe service, requested and validated version 1.13.0	Native classifier output
KEV	Local Kev-4B, MLX BF16, Qwen3.5-4B-Base with pinned head and adapter	Native classifier output
GLM	GLM 5.3 FlashX through OpenRouter, Z.AI FP8 fixed, fallback disabled, temperature 0	Estimates generated under a JSON contract

GLM’s 640-token completion cap included billed reasoning tokens. Its effort setting was maximum. KEV used its shipped temperature and no added date-fact preprocessing. These settings belong to the September 27 experiment; they are not refreshed claims about current vendor defaults.

The arms ran sequentially as CLM, JEV, KEV, then GLM on the same M4 Pro machine with 14 CPU cores, 20 GPU cores, and 48 GB unified memory. Local runs had a 120-second cooldown. Spotlight and backup conditions changed between arms, and GLM ran hours later. Hosted hardware and cache state were not controlled. Response-time comparisons therefore describe these configurations and this session.

Three separate validity checks

Decision validity requires successful complete transport from the expected model, unambiguous JSON, the expected case identifier, and a recognized lowercase decision.

Distribution validity additionally requires three numeric, finite probabilities between zero and one, a permitted sum, and a selected decision at a maximum probability.

Full compliance includes the applicable format rules and GLM’s additional instruction to choose the first criterion in supplied order on an exact tie. Native metadata was permitted. A maximum alone does not establish transport, identity, identifier, or formatting validity.

The counts happen to coincide in this run—300 for CLM, JEV, and KEV, 267 for GLM—but the concepts remain distinct.

Decimal tolerance preserves the observations

For precision d, the paper’s rule is τ(d) = max(1.5 × 10⁻ᵈ, 10⁻¹⁵), with |sum(p) − 1| ≤ τ(d). The two terms are compared, not added. The precision rule uses expanded decimal spellings, with a floor of two places for native JEV and four for KEV.

The accepted vectors were not normalized. Nonexact accepted sums occurred in 285 CLM responses and 75 KEV responses, with none in JEV or GLM. Rounding percentages for Table IV can also make a displayed row total 100.01%. That is a display effect, not a reason to alter the deposited values or require a rounded transcription to satisfy the original parser.

Pins and rights

CLM source commit 4858da6fef7e516847ce56e5274077f32cb5b350 identifies the unmerged MPS path. CUDA parity remains unverified. KEV source commit 5920c5fe4ca8e0970ed4209ac2c9b8e18bea5109 and adapter revision 139fdd94f1b6a6ad80cc15e08fcb99cac885a101 identify distinct components. JEV’s service version was 1.13.0. GLM’s observed service identifier, z-ai/glm-5.3-flashx-20260918, is not an exposed immutable weights revision.

The source repositories declare Apache-2.0; model and hosted-service rights remain separate. Hosted access does not grant a weights redistribution license.

Original paper, section IV and Appendix B · Evidence and reproduction