Lesson 11 of 18
How the experiment was run
Step 11 in the guided paper walkthrough.
By the end of this lesson
Identify shared controls and remaining configuration differences.
Hold the task constant
Each configuration received the same substantive policy and 300 cases in saved order, plus five separate unscored practice cases. Six label permutations occurred 50 times each. There was one outstanding request per arm, a 60-second timeout and no automatic retry. GLM had fresh context for each case.
Meet the four configurations
CLM and KEV ran on the study Mac. JEV used a hosted TypeSafe service. These three returned native choice probabilities. GLM used a hosted generative service through OpenRouter and generated a JSON answer with probability estimates. The same written task reached different interfaces.
Go deeper: exact historical configurations
CLM used local v0.1-8B, a BF16 Qwen3-8B encoder and an unmerged MPS implementation. JEV was hosted TypeSafe version 1.13.0 with identity validation. KEV was local Kev-4B using MLX BF16, a pinned adapter and head, and Qwen3.5-4B-Base. GLM 5.3 FlashX used OpenRouter's fixed Z.AI FP8 route, fallback disabled, temperature zero and maximum reasoning effort. These are historical September 27 settings.
Decode the configuration terms
Local means running on the study Mac; hosted means contacting a remote service. BF16 and FP8 describe numerical formats used to store or compute model values. An encoder represents input text numerically; a head maps that representation to choices; an adapter modifies a base model. MPS and MLX are software paths for Apple hardware. A CPU handles general computation, a GPU many computations in parallel, and unified memory is shared by those processors. A token is a unit of model text processing. A completion cap limits returned processing/output tokens under the specified service contract.
The local host was an M4 Pro Mac with 14 CPU cores, 20 GPU cores and 48 GB unified memory. Arm order was CLM, JEV, KEV, GLM. Local startup followed a 120-second cooldown. Background permissions changed between arms; GLM ran hours later. Hosted hardware and some cache behavior were unknown.
A fixed budget can affect completion
GLM’s 640-token completion cap included reasoning tokens. Separate feasibility fixtures completed, but 33 scored outputs later reached the cap unfinished. The cap remained frozen. LAYA was excluded before scored testing because its 512-token input limit could not accommodate the shared-policy inputs.
Go deeper: evidence, stops, rights and cost
Three preserved readiness stops contained no practice or scored requests. The first complete 300-attempt session per arm supplied primary results. The deposit preserves 1,200 scored records, 20 practice records, failures, preparation, versions and analysis. Recorded confirmed study charges were approximately US$0.16; US$0.86 remained unresolved JEV reservations, for conservative exposure of US$1.01 after independent rounding. These historical figures are not current pricing or a deployment total-cost estimate. Reproducing analysis needs no new paid inference. Source repositories declare Apache-2.0, but model and service rights remain separate; hosted access grants no weights redistribution right.
Try it
Explore the idea
Inspect a protocol choice.
Check your understanding · optional
Did using the same Mac isolate the causal effect of model architecture?
Take this forward
Next, read the accuracy results with these controls and limits in mind.
Paper and reference material
Original paper: §IV.A–D; Appendix B · page 3 (new tab)
The teaching explanation and interactive examples are companion additions. Recorded study values and frozen references remain unchanged.