The calculations, one step at a time

Companion addition. The paper is an empirical study, not a theorem paper. The definitions, propositions, lemmas, theorem, and proofs below unpack its measurements. They do not insert new claims into the manuscript. Equations here use the paper’s conventions unless explicitly identified as a teaching example. See original §§III.B–C and V, and the guided lessons on validity, accuracy, timing, probability and paired comparisons.

D.1 A notation key before any arithmetic

Definition D.1 · Cases, outputs, and eligible sets

Let $N=300$ be the assigned cases. Let $y_i\in{A,H,D}$ be the frozen reference, $\hat y_i$ a recognized model decision, and $v_i\in{0,1}$ indicate decision validity. Define $c_i=1$ only when the output is decision-valid and $\hat y_i=y_i$; otherwise $c_i=0$. Thus $c_i\leq v_i$.

Write $p_i=(p_{iA},p_{iH},p_{iD})$ for a distribution-valid reported vector, and $e_{y_i}$ for the one-hot reference vector. The probability-eligible set is $S$. Decision validity, distribution validity, and full compliance are different concepts even though their counts coincide for each tested arm.

An indicator, written $\mathbf 1{\text{condition}}$, is 1 when its condition is true and 0 otherwise. A sum adds observations. A mean divides a sum by the number of eligible observations. Always ask which observations are eligible before dividing.

The letters A/H/D are labels, not an ordering of goodness. An invalid output is not a fourth semantic policy label and is never silently converted to hold.

D.2 Accuracy and the denominator identity

Let $C=\sum_i c_i$ and $V=\sum_i v_i$. Then

Proposition D.2 · Coverage multiplies conditional accuracy

For $V>0$, $a_{\mathrm{primary}}=\mathrm{coverage}\times a_{\mathrm{valid}}$.

Proof. Substitute the fractions: $(V/N)(C/V)=C/N$. The valid count cancels; the assigned count does not. If $V=0$, primary accuracy is zero but valid-only accuracy is undefined, not zero by convention.

GLM, line by line. Start with 300 assigned cases. Remove nothing from primary scoring. Count 267 valid responses, including 260 correct and seven incorrect. The other 33 responses are invalid. Therefore $260/300=86.67$, $267/300=89.00$, and $260/267=97.38$ after display rounding. Multiply the unrounded last two fractions to recover primary accuracy. The 97.38% rate answers a conditional question and cannot replace the primary result.

D.3 Balanced accuracy and the always-hold baseline

Let $M_{rk}$ count reference class $r$ predicted as decision $k$. Include invalid outputs in the relevant reference row’s total, if present. With $n_r$ assigned cases in reference class $r$,

For JEV the recalls are $86/89$, $86/111$, and $100/100$. Their unweighted mean is 91.37% after rounding. For KEV replace only $86/111$ by $87/111$; the result is 91.67%.

Proposition D.3 · Always hold has one-third balanced accuracy here

When all three reference classes are present, always returning hold gives recalls $(0,1,0)$ and balanced accuracy $1/3$. Its ordinary accuracy is the hold prevalence, here $111/300=37$.

This is a statement about selected decisions. It does not imply CLM’s probability vectors were constant. Also, original Table II is not $M$: its rows are JEV decisions and its columns KEV decisions, not references versus predictions.

D.4 Wilson intervals without a black box

For $C$ correct answers out of $N$, write $\hat p=C/N$ and $z=1.9599639845\ldots$, the standard-normal 97.5th percentile. Define

The interval is $[m-h,m+h]$. Each line has a job: $\hat p$ is the observed proportion; $D$ adjusts the scale; $m$ is the adjusted center; $h$ is the half-width. Convert the final endpoints to percentages, not the intermediate fractions. For KEV, $C=273,N=300$ gives 87.22–93.74% after rounding, matching Table I.

The formula elaborates the paper’s cited Wilson method [7]. It is not a statement that deployment accuracy lies in that interval with a demonstrated frequency. The paper’s authored case bank, single reviewer, and one execution session remain limitations. Overlapping Wilson intervals do not by themselves answer a paired superiority question.

D.5 Brier loss is a full-vector score

Definition D.5 · Unscaled multiclass Brier score

For one eligible case, $L_i=\sum_{k\in{A,H,D}}(p_{ik}-\mathbf 1{y_i=k})^2=\lVert p_i-e_{y_i}\rVert_2^2$. The study score is $B=|S|^{-1}\sum_{i\in S}L_i$. There is no division by three. Lower is better.

For C207 the frozen reference is hold. The one-hot target is $(0,1,0)$. Table IV gives JEV $(0.47,0.53,0)$, so

KEV’s displayed percentages give $(0.9203,0.0662,0.0135)$, yielding approximately $1.71911678$. This second calculation uses rounded table values, rather than original response lexemes. It illustrates loss, not an exact reconstruction of the deposited calculation. A 92.03% selected probability on an incorrect allow can have larger loss than a 53% selected probability on the correct hold.

Lemma D.5 · Ideal simplex range is zero to two

For nonnegative $p_k$ summing exactly to one, $0\leq\lVert p-e_y\rVert_2^2\leq2$.

Proof. Squares are nonnegative. Expand the sum as $\sum_k p_k^2+1-2p_y$. Since $\sum_k p_k^2\leq(\sum_k p_k)^2=1$ and $p_y\geq0$, the expression is at most 2. A correct one-hot vector attains 0; a wrong one-hot vector attains 2.

The exact-simplex assumption matters. The study permitted tiny nonexact sums under a declared tolerance and did not normalize them. This idealized bound is not a license to change recorded probabilities.

D.6 Why Brier loss rewards truthful distributions

Theorem D.6 · Strict propriety of ideal Brier loss

Mathematical teaching addition, not an empirical theorem claimed by the paper. Suppose a categorical outcome has true probability vector $q$ and a forecaster reports probability vector $p$, both on the same simplex. Then expected Brier loss is uniquely minimized at $p=q$.

Proof, line by line.

1. Expand the squared error: $\mathbb E\lVert p-e_Y\rVert^2=\sum_k p_k^2-2\sum_k p_k\mathbb E[\mathbf 1{Y=k}]+\mathbb E\sum_k\mathbf 1{Y=k}$.
2. Substitute $\mathbb E[\mathbf 1{Y=k}]=q_k$ and the fact that exactly one outcome occurs. This gives $\sum_k p_k^2-2\sum_k p_kq_k+1$.
3. Add and subtract $\sum_k q_k^2$: the expression becomes $\lVert p-q\rVert^2+1-\lVert q\rVert^2$.
4. The last two terms do not depend on $p$. The first term is nonnegative and equals zero exactly when $p=q$.

This proof explains the mathematical scoring rule. It does not establish that any tested model knows a true $q$, that the reviewed key is independent ground truth, or that a reported probability is calibrated deployment risk. Native classifier probabilities and GLM’s generated estimates retain their different origins.

D.7 The fixed-frequency baseline and skill

Set $f=(89,111,100)/300$ once, using full-bank reference frequencies. For an eligible subset with class proportions $q^S$, evaluate the same forecast $f$ on that subset:

On the full bank, $q^S=f$, so $B_0=1-\lVert f\rVert^2\approx0.66398$. On GLM’s completed subset, $q^S=(81,95,91)/267$. Do not replace the forecast with that subset’s proportions; that would change the specified baseline.

A score of zero matches this baseline, a positive score improves on it, and a negative score is worse. This is neither accuracy nor AUROC. Calculating skill from the paper’s rounded Brier value will not necessarily reproduce the last printed digit: the original calculation uses unrounded losses.

D.8 Correctness AUROC and ties

Let $s_i$ be the reported probability attached to the chosen answer. Among eligible decisions, split the cases into correct set $C^+$ and incorrect set $C^-$. Then

Read the equation in order. Pick one correct and one incorrect response. Give one point if the correct response has higher selected probability, half a point if tied, and zero otherwise. Average over all such pairs. If either group is empty, this pairwise statistic is undefined.

A toy example with correct scores $[0.9,0.6]$ and incorrect scores $[0.8,0.6]$ receives $1+1+0+0.5=2.5$ points over four pairs, giving 0.625. It is a ranking calculation, not a probability-calibration test. Strictly increasing score transformations preserve ranks and ties, so AUROC can stay unchanged while numerical probabilities change.

Removing the eleven changed-reference cases changes the pair set as well as its size. The reversal of JEV/KEV AUROC ordering on the remaining 289 cases cannot be interpreted as subtracting eleven additive AUROC losses. The source labels this selected-subset analysis post hoc.

D.9 Reliability bins and what Figure 3 does not provide

Use ten intervals $[0,0.1),[0.1,0.2),\ldots,[0.9,1]$. In bin $b$, count $n_b$ eligible cases and calculate

Plot $(\bar s_b,\bar c_b)$ and display $n_b$. Empty bins have no point. The last interval includes probability 1. The diagonal marks equal reported probability and observed correctness in the bin. There is no fitted calibration model here.

The manuscript prints bin counts, not each numeric bin mean and accuracy. The companion preserves the original Figure 3 and transcribes its counts, but does not invent exact coordinates or regenerate an observed reliability curve from counts alone. The probability lesson uses a clearly labelled synthetic example to teach the calculation.

D.10 Paired contrasts and bootstrap mechanics

For models A and B evaluated on aligned case IDs, let $d_i=c_i^A-c_i^B\in{-1,0,1}$. Their accuracy difference is $\bar d=N^{-1}\sum_i d_i$. A paired resample draws $N$ indices with replacement from $1,\ldots,N$, then uses those same indices for both models. Duplicate indices must stay duplicated.

For each of 10,000 resamples, calculate a new $\bar d^$. A percentile interval takes the desired lower and upper empirical quantiles. The paper records seed 20260927 and a case-level paired procedure. A seed alone does not identify the random-number implementation, case ordering, or every percentile convention in the separately deposited code.

Lemma D.10 · Shared correctness cancels in the point difference

For the JEV–KEV follow-up, 264 shared successes and 19 shared failures contribute zero to KEV-minus-JEV. Nine KEV-only successes contribute $+9$, and eight JEV-only successes contribute $-8$. Therefore the net difference is $(9-8)/300=0.00333\ldots$, or 0.33 percentage points.

The paired-comparisons lesson constructs a grouped correctness illustration, without inventing case IDs, and bootstraps it with a documented deterministic teaching generator. This is a teaching exercise on the reported joint counts, not a reconstruction of the prespecified model-versus-GLM intervals and not a new superiority claim.

D.11 HTTP timing and the p95 convention

Primary timing is $t_i=t_i^{\mathrm{complete\ body}}-t_i^{\mathrm{HTTP\ transfer\ start}}$, using a monotonic clock. It includes applicable connection, network, service waiting, and required generation. Parsing, scoring, payload preparation, disk writes, and startup are outside this interval.

For sorted eligible times $x_{(0)},\ldots,x_{(n-1)}$, a common explicit form of the stated linear percentile rule is $h=(n-1)q$, $j=\lfloor h\rfloor$, $g=h-j$, and

Set $q=0.95$ for p95. The timing lesson demonstrates this with synthetic times. The manuscript itself gives summary quantiles but not the individual transfer times, so an illustration using only those summaries cannot recompute the original p95. The reader’s separate deposited-record audit independently verified the reported quantiles.

The reported median ratio $659.7/139.1\approx4.74$ compares local KEV with hosted JEV under their tested configurations. It is not an isolated architectural speedup. Summing request intervals, measuring scored-loop wall time, and measuring the full session answer different questions.

D.12 Decimal tolerance, without repair

The paper’s explicit numerical rule is

Take the larger value; do not add them. At $d=2$, the tolerance is 0.015; at $d=4$, it is 0.00015; at $d=16$, the floor makes it $10^{-15}$. Probability values must still be finite and individually within $[0,1]$.

Use decimal lexemes—original numeric spellings—to avoid silently replacing a decimal record with binary-floating-point arithmetic. The teaching helper takes $d$ explicitly. Recovering every original precision-selection decision requires the deposited parser and original response bytes; this helper does not pretend to reconstruct them from Table IV percentages.

A selected decision must be at a maximum for distribution validity. GLM’s first-in-order exact-tie instruction is an additional compliance requirement. A selected maximum alone does not establish valid identity, complete transport, correct case ID, unambiguous JSON, or full response compliance.

D.13 Consequences without invented severity

Write $U$ for reference hold/deny predicted allow, $R$ for reference allow/hold predicted deny, $E$ for reference allow/deny predicted hold, and $I$ for invalid output. The original counts are unweighted and hypothetical.

For an explicitly new sensitivity exercise, choose nonnegative weights and compute $w_UU+w_RR+w_EE+w_II$. Since JEV and KEV have equal $E$ and $I$, their JEV-minus-KEV weighted difference is $-2w_U+3w_R$. JEV has the lower illustrative total when $2w_U>3w_R$; KEV has the lower total when the inequality reverses. This algebra introduces no measured harm severity and recommends no deployment policy.

Check your understanding

Why is invalid not simply another hold?

Hold is a valid semantic determination about prerequisites. Invalid means the response failed the contract. Replacing invalid with hold would change the recorded decision and error consequences, violating the no-repair protocol.

Can a model be more accurate but less attractive under a consequence weighting?

Yes. Accuracy counts each incorrect or invalid assigned case equally. A separately declared weighting can value different consequences differently. In this study the weights were not measured, so weighted results must remain an illustrative sensitivity exercise.