# How the experiment was run > Identify shared controls and remaining configuration differences. [Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/lessons/11-experimental-design/) - Author: Junior Williams - Type: Research companion addition ## Learning objective Identify shared controls and remaining configuration differences. - [Original paper: §IV.A–D; Appendix B · page 3](https://trustcyber.ca/research/allow-hold-deny/original.pdf#page=3) Source pages: 3, 4, 10 ## Hold the task constant Each configuration received the same substantive policy and 300 cases in saved order, plus five separate unscored practice cases. Six label permutations occurred 50 times each. There was one outstanding request per arm, a 60-second timeout and no automatic retry. GLM had fresh context for each case. ## Meet the four configurations CLM and KEV ran on the study Mac. JEV used a hosted TypeSafe service. These three returned native choice probabilities. GLM used a hosted generative service through OpenRouter and generated a JSON answer with probability estimates. The same written task reached different interfaces.
Go deeper: exact historical configurations

CLM used local v0.1-8B, a BF16 Qwen3-8B encoder and an unmerged MPS implementation. JEV was hosted TypeSafe version 1.13.0 with identity validation. KEV was local Kev-4B using MLX BF16, a pinned adapter and head, and Qwen3.5-4B-Base. GLM 5.3 FlashX used OpenRouter's fixed Z.AI FP8 route, fallback disabled, temperature zero and maximum reasoning effort. These are historical September 27 settings.

Decode the configuration terms

Local means running on the study Mac; hosted means contacting a remote service. BF16 and FP8 describe numerical formats used to store or compute model values. An encoder represents input text numerically; a head maps that representation to choices; an adapter modifies a base model. MPS and MLX are software paths for Apple hardware. A CPU handles general computation, a GPU many computations in parallel, and unified memory is shared by those processors. A token is a unit of model text processing. A completion cap limits returned processing/output tokens under the specified service contract.

The local host was an M4 Pro Mac with 14 CPU cores, 20 GPU cores and 48 GB unified memory. Arm order was CLM, JEV, KEV, GLM. Local startup followed a 120-second cooldown. Background permissions changed between arms; GLM ran hours later. Hosted hardware and some cache behavior were unknown. ## A fixed budget can affect completion GLM's 640-token completion cap included reasoning tokens. Separate feasibility fixtures completed, but 33 scored outputs later reached the cap unfinished. The cap remained frozen. LAYA was excluded before scored testing because its 512-token input limit could not accommodate the shared-policy inputs.
Go deeper: evidence, stops, rights and cost

Three preserved readiness stops contained no practice or scored requests. The first complete 300-attempt session per arm supplied primary results. The deposit preserves 1,200 scored records, 20 practice records, failures, preparation, versions and analysis. Recorded confirmed study charges were approximately US$0.16; US$0.86 remained unresolved JEV reservations, for conservative exposure of US$1.01 after independent rounding. These historical figures are not current pricing or a deployment total-cost estimate. Reproducing analysis needs no new paid inference. Source repositories declare Apache-2.0, but model and service rights remain separate; hosted access grants no weights redistribution right.

## Explore the idea Inspect a protocol choice. - Five unscored practices per arm: Practice warms the client and checks readiness; its success is not part of scored accuracy. - No repair or retry: Failed outputs remain visible in protocol-level results. - GLM completion cap fixed: A larger cap would define a different configuration; this run does not estimate that counterfactual. ## Interactive activities Use the canonical HTML page for controls, live results and reset buttons. ## Check your understanding (optional) Did using the same Mac isolate the causal effect of model architecture? - No; model size, implementation and framework still differed: Correct. A common host reduces one difference without equalizing the remaining components. - Yes; every condition was identical: Hosted hardware, background activity, model sizes and runtimes differed. - Yes; practices guarantee all cases complete: Separate feasibility fixtures did not prevent scored truncation. ## Take this forward Next, read the accuracy results with these controls and limits in mind. ## Reference material - [study-design](https://trustcyber.ca/research/allow-hold-deny/study-design/index.md) - [reproduction](https://trustcyber.ca/research/allow-hold-deny/reproduction/index.md) ## Lesson navigation - [Previous: Different errors, different effects](https://trustcyber.ca/research/allow-hold-deny/lessons/10-error-consequences/index.md) - [Next: Read the accuracy results](https://trustcyber.ca/research/allow-hold-deny/lessons/12-accuracy-results/index.md) ## Companion navigation - [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md) - [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)