# The calculations, one step at a time > Definitions, derivations, and proofs explain the measurements. They do not add empirical claims to the paper. [Canonical HTML page](https://trustcyber.ca/research/allow-hold-deny/mathematics/) - Author: Junior Williams - Type: Research companion addition **Companion addition.** The paper is an empirical study, not a theorem paper. The definitions, propositions, lemmas, theorem, and proofs below unpack its measurements. They do not insert new claims into the manuscript. Equations here use the paper’s conventions unless explicitly identified as a teaching example. See original §§III.B–C and V, and the guided lessons on validity, accuracy, timing, probability and paired comparisons. ## D.1 A notation key before any arithmetic ### Definition D.1 · Cases, outputs, and eligible sets Let N=300N=300 be the assigned cases. Let yi∈{A,H,D}y_i\in\{A,H,D\} be the **frozen reference**, ŷi\hat y_i a recognized model decision, and vi∈{0,1}v_i\in\{0,1\} indicate decision validity. Define ci=1c_i=1 only when the output is decision-valid **and** ŷi=yi\hat y_i=y_i; otherwise ci=0c_i=0. Thus ci≤vic_i\leq v_i. Write pi=(piA,piH,piD)p_i=(p_{iA},p_{iH},p_{iD}) for a distribution-valid reported vector, and eyie_{y_i} for the one-hot reference vector. The probability-eligible set is SS. Decision validity, distribution validity, and full compliance are different concepts even though their counts coincide for each tested arm. An indicator, written 𝟏{condition}\mathbf 1\{\text{condition}\}, is 1 when its condition is true and 0 otherwise. A sum adds observations. A mean divides a sum by the number of eligible observations. **Always ask which observations are eligible before dividing.** The letters A/H/D are labels, not an ordering of goodness. An invalid output is not a fourth semantic policy label and is never silently converted to hold. ## D.2 Accuracy and the denominator identity Let C=∑iciC=\sum_i c_i and V=∑iviV=\sum_i v_i. Then
aprimary=CN,coverage=VN,avalid=CV(V>0).a_{\mathrm{primary}}=\frac{C}{N},\qquad \mathrm{coverage}=\frac{V}{N},\qquad a_{\mathrm{valid}}=\frac{C}{V}\quad(V>0).
### Proposition D.2 · Coverage multiplies conditional accuracy For V>0V>0, aprimary=coverage×avalida_{\mathrm{primary}}=\mathrm{coverage}\times a_{\mathrm{valid}}. **Proof.** Substitute the fractions: (V/N)(C/V)=C/N(V/N)(C/V)=C/N. The valid count cancels; the assigned count does not. If V=0V=0, primary accuracy is zero but valid-only accuracy is undefined, not zero by convention. **GLM, line by line.** Start with 300 assigned cases. Remove nothing from primary scoring. Count 267 valid responses, including 260 correct and seven incorrect. The other 33 responses are invalid. Therefore 260/300=86.67%260/300=86.67\%, 267/300=89.00%267/300=89.00\%, and 260/267=97.38%260/267=97.38\% after display rounding. Multiply the **unrounded** last two fractions to recover primary accuracy. The 97.38% rate answers a conditional question and cannot replace the primary result. ## D.3 Balanced accuracy and the always-hold baseline Let MrkM_{rk} count reference class rr predicted as decision kk. Include invalid outputs in the relevant reference row’s total, if present. With nrn_r assigned cases in reference class rr,
recallr=Mrrnr,BA=13∑r∈{A,H,D}recallr.\mathrm{recall}_r=\frac{M_{rr}}{n_r},\qquad \mathrm{BA}=\frac{1}{3}\sum_{r\in\{A,H,D\}}\mathrm{recall}_r.
For JEV the recalls are 86/8986/89, 86/11186/111, and 100/100100/100. Their unweighted mean is 91.37% after rounding. For KEV replace only 86/11186/111 by 87/11187/111; the result is 91.67%. ### Proposition D.3 · Always hold has one-third balanced accuracy here When all three reference classes are present, always returning hold gives recalls (0,1,0)(0,1,0) and balanced accuracy 1/31/3. Its ordinary accuracy is the hold prevalence, here 111/300=37%111/300=37\%. This is a statement about selected decisions. It does **not** imply CLM’s probability vectors were constant. Also, original Table II is not MM: its rows are JEV decisions and its columns KEV decisions, not references versus predictions. ## D.4 Wilson intervals without a black box For CC correct answers out of NN, write p̂=C/N\hat p=C/N and z=1.9599639845…z=1.9599639845\ldots, the standard-normal 97.5th percentile. Define
D=1+z2N,m=p̂+z2/(2N)D,h=zDp̂(1−p̂)N+z24N2.D=1+\frac{z^2}{N},\quad m=\frac{\hat p+z^2/(2N)}{D},\quad h=\frac{z}{D}\sqrt{\frac{\hat p(1-\hat p)}{N}+\frac{z^2}{4N^2}}.
The interval is [m−h,m+h][m-h,m+h]. Each line has a job: p̂\hat p is the observed proportion; DD adjusts the scale; mm is the adjusted center; hh is the half-width. Convert the final endpoints to percentages, not the intermediate fractions. For KEV, C=273,N=300C=273,N=300 gives 87.22–93.74% after rounding, matching Table I. The formula elaborates the paper’s cited Wilson method [7]. It is not a statement that deployment accuracy lies in that interval with a demonstrated frequency. The paper’s authored case bank, single reviewer, and one execution session remain limitations. Overlapping Wilson intervals do not by themselves answer a paired superiority question. ## D.5 Brier loss is a full-vector score ### Definition D.5 · Unscaled multiclass Brier score For one eligible case, Li=∑k∈{A,H,D}(pik−𝟏{yi=k})2=‖pi−eyi‖22L_i=\sum_{k\in\{A,H,D\}}(p_{ik}-\mathbf 1\{y_i=k\})^2=\lVert p_i-e_{y_i}\rVert_2^2. The study score is B=|S|−1∑i∈SLiB=|S|^{-1}\sum_{i\in S}L_i. **There is no division by three.** Lower is better. For C207 the frozen reference is hold. The one-hot target is (0,1,0)(0,1,0). Table IV gives JEV (0.47,0.53,0)(0.47,0.53,0), so
LJEV=(0.47−0)2+(0.53−1)2+(0−0)2=0.4418.L_{\mathrm{JEV}}=(0.47-0)^2+(0.53-1)^2+(0-0)^2=0.4418.
KEV’s displayed percentages give (0.9203,0.0662,0.0135)(0.9203,0.0662,0.0135), yielding approximately 1.719116781.71911678. This second calculation uses **rounded table values**, rather than original response lexemes. It illustrates loss, not an exact reconstruction of the deposited calculation. A 92.03% selected probability on an incorrect allow can have larger loss than a 53% selected probability on the correct hold. ### Lemma D.5 · Ideal simplex range is zero to two For nonnegative pkp_k summing exactly to one, 0≤‖p−ey‖22≤20\leq\lVert p-e_y\rVert_2^2\leq2. **Proof.** Squares are nonnegative. Expand the sum as ∑kpk2+1−2py\sum_k p_k^2+1-2p_y. Since ∑kpk2≤(∑kpk)2=1\sum_k p_k^2\leq(\sum_k p_k)^2=1 and py≥0p_y\geq0, the expression is at most 2. A correct one-hot vector attains 0; a wrong one-hot vector attains 2. The exact-simplex assumption matters. The study permitted tiny nonexact sums under a declared tolerance and did not normalize them. This idealized bound is not a license to change recorded probabilities. ## D.6 Why Brier loss rewards truthful distributions ### Theorem D.6 · Strict propriety of ideal Brier loss **Mathematical teaching addition, not an empirical theorem claimed by the paper.** Suppose a categorical outcome has true probability vector qq and a forecaster reports probability vector pp, both on the same simplex. Then expected Brier loss is uniquely minimized at p=qp=q. **Proof, line by line.** 1. Expand the squared error: 𝔼‖p−eY‖2=∑kpk2−2∑kpk𝔼[𝟏{Y=k}]+𝔼∑k𝟏{Y=k}\mathbb E\lVert p-e_Y\rVert^2=\sum_k p_k^2-2\sum_k p_k\mathbb E[\mathbf 1\{Y=k\}]+\mathbb E\sum_k\mathbf 1\{Y=k\}. 2. Substitute 𝔼[𝟏{Y=k}]=qk\mathbb E[\mathbf 1\{Y=k\}]=q_k and the fact that exactly one outcome occurs. This gives ∑kpk2−2∑kpkqk+1\sum_k p_k^2-2\sum_k p_kq_k+1. 3. Add and subtract ∑kqk2\sum_k q_k^2: the expression becomes ‖p−q‖2+1−‖q‖2\lVert p-q\rVert^2+1-\lVert q\rVert^2. 4. The last two terms do not depend on pp. The first term is nonnegative and equals zero exactly when p=qp=q. This proof explains the mathematical scoring rule. It **does not** establish that any tested model knows a true qq, that the reviewed key is independent ground truth, or that a reported probability is calibrated deployment risk. Native classifier probabilities and GLM’s generated estimates retain their different origins. ## D.7 The fixed-frequency baseline and skill Set f=(89,111,100)/300f=(89,111,100)/300 once, using full-bank reference frequencies. For an eligible subset with class proportions qSq^S, evaluate the **same** forecast ff on that subset:
B0(S)=∑kqkS‖f−ek‖2=1+‖f‖2−2f⋅qS.B_0(S)=\sum_{k}q_k^S\,\lVert f-e_k\rVert^2=1+\lVert f\rVert^2-2f\cdot q^S.
On the full bank, qS=fq^S=f, so B0=1−‖f‖2≈0.66398B_0=1-\lVert f\rVert^2\approx0.66398. On GLM’s completed subset, qS=(81,95,91)/267q^S=(81,95,91)/267. Do **not** replace the forecast with that subset’s proportions; that would change the specified baseline.
Brierskill=1−BB0(S).\mathrm{Brier\ skill}=1-\frac{B}{B_0(S)}.
A score of zero matches this baseline, a positive score improves on it, and a negative score is worse. This is neither accuracy nor AUROC. Calculating skill from the paper’s rounded Brier value will not necessarily reproduce the last printed digit: the original calculation uses unrounded losses. ## D.8 Correctness AUROC and ties Let sis_i be the reported probability attached to the chosen answer. Among eligible decisions, split the cases into correct set C+C^+ and incorrect set C−C^-. Then
AUROC=1|C+||C−|∑i∈C+∑j∈C−[𝟏{si>sj}+12𝟏{si=sj}].\mathrm{AUROC}=\frac{1}{|C^+||C^-|}\sum_{i\in C^+}\sum_{j\in C^-}\left[\mathbf1\{s_i>s_j\}+\tfrac12\mathbf1\{s_i=s_j\}\right].
**Read the equation in order.** Pick one correct and one incorrect response. Give one point if the correct response has higher selected probability, half a point if tied, and zero otherwise. Average over all such pairs. If either group is empty, this pairwise statistic is undefined. A toy example with correct scores [0.9,0.6][0.9,0.6] and incorrect scores [0.8,0.6][0.8,0.6] receives 1+1+0+0.5=2.51+1+0+0.5=2.5 points over four pairs, giving 0.625. It is a ranking calculation, not a probability-calibration test. Strictly increasing score transformations preserve ranks and ties, so AUROC can stay unchanged while numerical probabilities change. Removing the eleven changed-reference cases changes the pair set as well as its size. The reversal of JEV/KEV AUROC ordering on the remaining 289 cases cannot be interpreted as subtracting eleven additive AUROC losses. The source labels this selected-subset analysis post hoc. ## D.9 Reliability bins and what Figure 3 does not provide Use ten intervals [0,0.1),[0.1,0.2),…,[0.9,1][0,0.1),[0.1,0.2),\ldots,[0.9,1]. In bin bb, count nbn_b eligible cases and calculate
s‾b=1nb∑i∈bsi,c‾b=1nb∑i∈bci.\bar s_b=\frac{1}{n_b}\sum_{i\in b}s_i,\qquad \bar c_b=\frac{1}{n_b}\sum_{i\in b}c_i.
Plot (s‾b,c‾b)(\bar s_b,\bar c_b) and display nbn_b. Empty bins have no point. The last interval includes probability 1. The diagonal marks equal reported probability and observed correctness in the bin. There is no fitted calibration model here. The manuscript prints bin counts, not each numeric bin mean and accuracy. The companion preserves the **original Figure 3** and transcribes its counts, but does not invent exact coordinates or regenerate an observed reliability curve from counts alone. The probability lesson uses a clearly labelled synthetic example to teach the calculation. ## D.10 Paired contrasts and bootstrap mechanics For models A and B evaluated on aligned case IDs, let di=ciA−ciB∈{−1,0,1}d_i=c_i^A-c_i^B\in\{-1,0,1\}. Their accuracy difference is d‾=N−1∑idi\bar d=N^{-1}\sum_i d_i. A paired resample draws NN indices **with replacement** from 1,…,N1,\ldots,N, then uses those same indices for both models. Duplicate indices must stay duplicated. For each of 10,000 resamples, calculate a new d‾*\bar d^*. A percentile interval takes the desired lower and upper empirical quantiles. The paper records seed 20260927 and a case-level paired procedure. A seed alone does not identify the random-number implementation, case ordering, or every percentile convention in the separately deposited code. ### Lemma D.10 · Shared correctness cancels in the point difference For the JEV–KEV follow-up, 264 shared successes and 19 shared failures contribute zero to KEV-minus-JEV. Nine KEV-only successes contribute +9+9, and eight JEV-only successes contribute −8-8. Therefore the net difference is (9−8)/300=0.00333…(9-8)/300=0.00333\ldots, or 0.33 percentage points. The paired-comparisons lesson constructs a **grouped correctness illustration**, without inventing case IDs, and bootstraps it with a documented deterministic teaching generator. This is a teaching exercise on the reported joint counts, not a reconstruction of the prespecified model-versus-GLM intervals and not a new superiority claim. ## D.11 HTTP timing and the p95 convention Primary timing is ti=ticompletebody−tiHTTPtransferstartt_i=t_i^{\mathrm{complete\ body}}-t_i^{\mathrm{HTTP\ transfer\ start}}, using a monotonic clock. It includes applicable connection, network, service waiting, and required generation. Parsing, scoring, payload preparation, disk writes, and startup are outside this interval. For sorted eligible times x(0),…,x(n−1)x_{(0)},\ldots,x_{(n-1)}, a common explicit form of the stated linear percentile rule is h=(n−1)qh=(n-1)q, j=⌊h⌋j=\lfloor h\rfloor, g=h−jg=h-j, and
Q(q)=(1−g)x(j)+gx(min⁡(j+1,n−1)).Q(q)=(1-g)x_{(j)}+g x_{(\min(j+1,n-1))}.
Set q=0.95q=0.95 for p95. The timing lesson demonstrates this with **synthetic** times. The manuscript itself gives summary quantiles but not the individual transfer times, so an illustration using only those summaries cannot recompute the original p95. The reader’s separate deposited-record audit independently verified the reported quantiles. The reported median ratio 659.7/139.1≈4.74659.7/139.1\approx4.74 compares local KEV with hosted JEV under their tested configurations. It is not an isolated architectural speedup. Summing request intervals, measuring scored-loop wall time, and measuring the full session answer different questions. ## D.12 Decimal tolerance, without repair The paper’s explicit numerical rule is
τ(d)=max⁡(1.5×10−d,10−15),|∑kpk−1|≤τ(d).\tau(d)=\max\!\left(1.5\times10^{-d},\ 10^{-15}\right),\qquad \left|\sum_kp_k-1\right|\leq\tau(d).
Take the larger value; do not add them. At d=2d=2, the tolerance is 0.015; at d=4d=4, it is 0.00015; at d=16d=16, the floor makes it 10−1510^{-15}. Probability values must still be finite and individually within [0,1][0,1]. Use decimal lexemes—original numeric spellings—to avoid silently replacing a decimal record with binary-floating-point arithmetic. The teaching helper takes dd explicitly. Recovering every original precision-selection decision requires the deposited parser and original response bytes; this helper does not pretend to reconstruct them from Table IV percentages. A selected decision must be at a maximum for distribution validity. GLM’s first-in-order exact-tie instruction is an **additional** compliance requirement. A selected maximum alone does not establish valid identity, complete transport, correct case ID, unambiguous JSON, or full response compliance. ## D.13 Consequences without invented severity Write UU for reference hold/deny predicted allow, RR for reference allow/hold predicted deny, EE for reference allow/deny predicted hold, and II for invalid output. The original counts are unweighted and hypothetical. For an explicitly new sensitivity exercise, choose nonnegative weights and compute wUU+wRR+wEE+wIIw_UU+w_RR+w_EE+w_II. Since JEV and KEV have equal EE and II, their JEV-minus-KEV weighted difference is −2wU+3wR-2w_U+3w_R. JEV has the lower illustrative total when 2wU>3wR2w_U>3w_R; KEV has the lower total when the inequality reverses. This algebra introduces no measured harm severity and recommends no deployment policy. ## Check your understanding
Why is invalid not simply another hold? Hold is a valid semantic determination about prerequisites. Invalid means the response failed the contract. Replacing invalid with hold would change the recorded decision and error consequences, violating the no-repair protocol.
Can a model be more accurate but less attractive under a consequence weighting? Yes. Accuracy counts each incorrect or invalid assigned case equally. A separately declared weighting can value different consequences differently. In this study the weights were not measured, so weighted results must remain an illustrative sensitivity exercise.
## Companion navigation - [Overview](https://trustcyber.ca/research/allow-hold-deny/index.md) - [Both authorization studies](https://trustcyber.ca/research/authorization/index.md)