Accuracy, coverage, and response time

The primary result

These are the paper’s reported results, independently checked against the deposited scored observations for this reader. Primary accuracy uses all 300 assigned cases for each model. Invalid output contributes zero correct decisions.

Arm	Correct / assigned	Primary accuracy	Valid / assigned	Balanced accuracy	Wilson 95% interval
CLM	111 / 300	37.00%	300 / 300	33.33%	31.73–42.60%
JEV	272 / 300	90.67%	300 / 300	91.37%	86.84–93.46%
KEV	273 / 300	91.00%	300 / 300	91.67%	87.22–93.74%
GLM	260 / 300	86.67%	267 / 300	87.02%	82.35–90.05%

Why GLM has two accuracy rates

GLM returned 267 valid decisions: 260 correct and seven incorrect. All 33 invalid responses ended at the 640-token cap with finish_reason: length; they were not timeouts or semantic holds. Thus primary accuracy is 260/300 = 86.67%, coverage is 267/300 = 89.00%, and valid-only accuracy is 260/267 = 97.38%.

The last rate describes completed responses. It cannot replace the primary rate. Multiplying unrounded coverage by unrounded conditional accuracy recovers primary accuracy. If no output were valid, conditional accuracy would be undefined.

Invalid cases included eight allow, 16 hold, and nine deny references. The completed subset therefore contains 81 allow, 95 hold, and 91 deny references. Omissions are uneven: eight of the eleven changed-reference cases were truncated, compared with 25 of the other 289 cases. Fourteen of 36 cases in the disagreement-or-shared-error group were truncated, compared with 19 of the other 264. These are descriptive subset patterns, not an independent measure of difficulty.

Read a confusion matrix against the key

Rows below are frozen reference classes. Columns are returned decisions, with invalid output retained separately. Balanced accuracy averages the three class recalls, each using every assigned case in that reference class. The GLM matrix is a new derivation from deposited observations; the manuscript did not print that full matrix.

CLM selected hold on all 300 cases, giving 111 correct answers and one-third balanced accuracy. Its probability vectors were not necessarily constant. Both JEV and KEV got 86 of 89 allow references and all 100 deny references correct. Their hold-reference counts explain the net one-case difference.

The JEV–KEV agreement table on the next page has different axes. Agreement between models does not establish correctness.

Timing measures a complete HTTP response

The interval runs from HTTP transfer start to complete response body on a monotonic clock. It includes applicable connection, network, service waiting, and generation. It excludes startup, payload preparation, parsing, validation, and disk writes.

Arm	Eligible responses	Median	p95
CLM	300	1,113.2 ms	1,183.1 ms
JEV	300	139.1 ms	211.1 ms
KEV	300	659.7 ms	678.3 ms
GLM	267	2,966.5 ms	5,146.6 ms

Primary timing uses fully compliant responses. GLM’s all-response median was 3,172.1 ms, and its invalid-response median was 6,004.5 ms (p95 6,812.6 ms). Those timings answer different questions from the compliant-only row.

The ratio 659.7/139.1 ≈ 4.74 compares KEV’s local configuration with JEV’s hosted configuration in this session. It does not isolate architecture, model computation, or general throughput.

Summed HTTP intervals, scored-loop elapsed time, and session elapsed time also differ. JEV’s values are 43.80, 118.53, and 127.21 seconds; KEV’s are 199.07, 270.89, and 412.01 seconds. KEV’s session includes the 120-second cooldown and 9.65-second startup. The timing explorer changes units explicitly when showing these scopes.

Original paper, Tables I and III, sections V.A–B and V.G · Original figures