# Four configurations, one frozen case bank

> Read the controls and the differences between configurations before comparing the results.

[Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/study-design/)

- Author: Junior Williams
- Type: Research companion addition

## What was held in common

Each arm received the same 300 scored cases, in the same saved order. The frozen key contains 89 allow, 111 hold, and 100 deny references. Eleven proposed allow labels were changed to hold during pre-run review. A rationale for C071 was corrected while its deny label stayed the same.

Each arm also received five disjoint, unscored practice cases. The resulting deposit contains 1,200 scored records and 20 practice records, along with preparation and analysis artifacts. Practice success is not scored completion.

Six permutations of the three choice labels appear 50 times each. The saved order uses seed 20260927. There was one outstanding request per arm, a 60-second timeout, and no automatic retry, output repair, or rescoring. GLM used a fresh context for each case.

## What differed between arms

| Arm | Tested configuration | Probability origin |
| --- | --- | --- |
| CLM | v0.1-8B, local BF16 Qwen3-8B, unmerged MPS implementation | Native classifier output |
| JEV | Hosted TypeSafe service, requested and validated version 1.13.0 | Native classifier output |
| KEV | Local Kev-4B, MLX BF16, Qwen3.5-4B-Base with pinned head and adapter | Native classifier output |
| GLM | GLM 5.3 FlashX through OpenRouter, Z.AI FP8 fixed, fallback disabled, temperature 0 | Estimates generated under a JSON contract |

GLM’s 640-token completion cap included billed reasoning tokens. Its effort setting was maximum. KEV used its shipped temperature and no added date-fact preprocessing. These settings belong to the September 27 experiment; they are not refreshed claims about current vendor defaults.

The arms ran sequentially as CLM, JEV, KEV, then GLM on the same M4 Pro machine with 14 CPU cores, 20 GPU cores, and 48 GB unified memory. Local runs had a 120-second cooldown. Spotlight and backup conditions changed between arms, and GLM ran hours later. Hosted hardware and cache state were not controlled. Response-time comparisons therefore describe these configurations and this session.

## Three separate validity checks

**Decision validity** requires successful complete transport from the expected model, unambiguous JSON, the expected case identifier, and a recognized lowercase decision.

**Distribution validity** additionally requires three numeric, finite probabilities between zero and one, a permitted sum, and a selected decision at a maximum probability.

**Full compliance** includes the applicable format rules and GLM’s additional instruction to choose the first criterion in supplied order on an exact tie. Native metadata was permitted. A maximum alone does not establish transport, identity, identifier, or formatting validity.

The counts happen to coincide in this run—300 for CLM, JEV, and KEV, 267 for GLM—but the concepts remain distinct.

## Decimal tolerance preserves the observations

For precision d, the paper’s rule is τ(d) = max(1.5 × 10⁻ᵈ, 10⁻¹⁵), with |sum(p) − 1| ≤ τ(d). The two terms are compared, not added. The precision rule uses expanded decimal spellings, with a floor of two places for native JEV and four for KEV.

The accepted vectors were not normalized. Nonexact accepted sums occurred in 285 CLM responses and 75 KEV responses, with none in JEV or GLM. Rounding percentages for Table IV can also make a displayed row total 100.01%. That is a display effect, not a reason to alter the deposited values or require a rounded transcription to satisfy the original parser.

## Pins and rights

CLM source commit `4858da6fef7e516847ce56e5274077f32cb5b350` identifies the unmerged MPS path. CUDA parity remains unverified. KEV source commit `5920c5fe4ca8e0970ed4209ac2c9b8e18bea5109` and adapter revision `139fdd94f1b6a6ad80cc15e08fcb99cac885a101` identify distinct components. JEV’s service version was 1.13.0. GLM’s observed service identifier, `z-ai/glm-5.3-flashx-20260918`, is not an exposed immutable weights revision.

The source repositories declare Apache-2.0; model and hosted-service rights remain separate. Hosted access does not grant a weights redistribution license.

[Original paper, section IV and Appendix B](/research/allow-hold-deny/original.pdf#page=3) · [Evidence and reproduction](/research/allow-hold-deny/reproduction/)

## Companion navigation

- [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md)
- [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)