Compare speed and consequences
Interpret measured response times and explore illustrative error weights.
- Author: Junior Williams
- Type: Research companion addition
Learning objective
Interpret measured response times and explore illustrative error weights.
- Original paper: §V.E · page 6 Source pages: 6, 7
Read the observed speed
Compliant-response median HTTP times were JEV 139.1 ms, KEV 659.7 ms, CLM 1113.2 ms and GLM 2966.5 ms. Their p95 values were 211.1, 678.3, 1183.1 and 5146.6 ms. KEV's median was 4.74 times JEV's in this run, comparing a local MLX system with a hosted service.
JEV's summed scored HTTP intervals were 43.80 seconds, scored loop 118.53 seconds and full session 127.21 seconds. KEV's were 199.07, 270.89 and 412.01 seconds. The session includes work beyond response transfers; KEV's also includes cooldown and startup.
Give each error its own count
JEV had nine unauthorized allows, sixteen refusals and three escalations. KEV had eleven, thirteen and three. CLM had 189 escalations. GLM had two unauthorized allows, four refusals, one escalation and 33 invalid outputs. GLM's missing answers prevent interpreting its lower unauthorized-allow count alone as greater conservatism.
Equal teaching weights give totals CLM 189, JEV 28, KEV 27, GLM 40. Increase the unauthorized-allow weight to 5: JEV becomes 64 and KEV 71. These weights express illustrative preferences; no severity or real cost was measured.
Go deeper: the weighting identity
JEV minus KEV = −2 × unauthorized-allow weight + 3 × refusal weight. Escalation and invalid counts cancel for those two models. With weights 3 and 2 they tie. This sensitivity calculation leaves the original results and labels unchanged.
Go deeper: slower failures and early-response diagnostics
GLM's median over all responses was 3172.1 ms; over its invalid responses, median was 6004.5 ms and p95 6812.6 ms. The prespecified first-ten-versus-rest compliant median ratios were CLM 1.001, JEV 0.967, KEV 1.003 and GLM 1.050. This compares early responses with the remaining responses; no early requests were removed because of it. These diagnostics do not replace the compliant-response headline metric.
Browser teaching tools
The canonical HTML page provides interactive controls. Its chapter explains their evidence, arithmetic, and limitations.
- timing (Human view): Select JEV and switch from median response time to full-session time. Observe the changed unit and measurement boundary.
- weights (Human view): Change unauthorized-allow weight from 1 to 5. Compare JEV 64 with KEV 71, then reset to equal weights.
Interactive activities
Use the canonical HTML page for controls, live results and reset buttons.
Check your understanding (optional)
Does the lower weighted total prove a system is safer to deploy?
- No; the weights are illustrative and actions were not executed: Correct. Deployment needs environment-specific testing and severity judgments absent from this study.
- Yes; any positive weight measures harm: Choosing a number does not measure a real-world consequence.
- Yes; missing GLM decisions have no effect: Invalid responses remain a separate counted category.
Take this forward
Next, identify where interpretation must stop.