Audit of the measurement chain – status 2026-10-25

Every change to Coherent passes a chain of tests before it is adopted. On 2026-10-25 the operator had this measurement chain examined: does it work efficiently, and is it tuned right, neither too tight nor too loose, proven or refuted mathematically where possible, with tolerances and probabilities? The project's review role gave the question to models of three different vendors, which calculated independently, and fixed its own calculation beforehand. For this post the core calculation was redone once more, independently, and every count was checked against the daily sheets of the pipeline and the oracle register.

Note: prebuilding successors, the most important lever from this audit, is currently being examined.

The result in brief

• Not too loose. On the regular gain path, a candidate without a real gain is promoted with about 3 in 100,000, calculated numerically in the model. Five independent calculations agree on this. Proven in the Gaussian model is a strict upper bound of 12 in 100,000, without approximating the check points.
• Tight for small real gains. Calculated numerically in the model: a gain of +2 nElonElo Normalised Elo: the Elo difference divided by the spread of the game results, so that the SPRT bounds mean the same at different time controls. is promoted with 0.5 %, +5 nElo with 26 %, +8 nElo with 80 %.
• Whether the setting is exactly right cannot be proven. How close it lies to the optimum depends on the unknown distribution of real effects and on assumptions about capacity; the audit's calculations give different answers under different assumptions.
• The counted idle time shows a substantial loss in operation. From 2026-09-29 to 2026-10-24 the oracle stood idle 28.7 % of the time, in the last week 31.8 %. The target is below 10 %. Mostly, the next candidate was missing after a pass.
• Under simplifying assumptions the counts fit roughly half transfer: about half of the effect measured at fixed nodes would arrive at the clock. Then the bounds of triage and oracle fit together.

The measurement chain

• G0 to G2 are pass or fail: a build without warnings, rules and size limits, correctness of move generation, UCIUCI Universal Chess Interface, the text protocol between a chess engine and the program that runs it. behaviour, games against a random mover without illegal moves, crashes or time losses.
• G3, the triage, plays 500 game pairs at fixed nodes. If the score is at −14 EloElo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. or below, the candidate is rejected.
• G3b is a sequential test at fixed nodes: [0; 8] nElo with bounds ±3.4, a look every 1,500 pairs, a cap at 12,000 pairs. A candidate that reaches the cap without a decision is parked; that is not a pass.
• G4 is the oracle's short stage: SPRTSPRT Sequential probability ratio test. Games are played until the accumulated evidence reaches one of two bounds; then the test stops. [0; 5] nElo, α = β = 0.05, at most 96,000 games.
• G5 is the oracle's holdout stage on a segment of its own: SPRT [0; 5] nElo, α = 0.05, β = 0.25, at most 36,000 games. Its conditions are not published.

If an oracle test reaches its game limit without a decision, that is undecided and not a pass. nElo is normalised Elo: the effect divided by the spread of the pair results, so that the bounds mean the same at different time controls. Candidates that only compute faster and measurements at the clock have paths of their own; there are no separate counts per kind for them.

Model and formulas

All calculations use the same model, so that they can be compared:

• Pairs are independent, and the true effect Δ is the same in every stage.
• The test's log-likelihood ratio (LLRLLR Log-likelihood ratio, the running evidence of an SPRT. It starts at 0; a stage passes at the upper bound and fails at the lower one.) is approximated as Brownian motion.
• With t = Δ/(800/ln 10) and δ = elo1/(800/ln 10), each pair adds to the LLR of the test [0; elo1] an increment with mean δ(2t − δ) and variance 2δ².
• Equivalently: a single pair estimates Δ with variance (800/ln 10)²/2 ≈ 60,356 nElo².
• The bounds are A = ln((1 − β)/α) above and B = ln(β/(1 − α)) below. For G4 that is ±ln 19 = ±2.944, for G5 it is ln 15 = 2.708 and ln(0.25/0.95) = −1.335.
• The chain is the product of the stages: P(promotion | Δ) = P_G3(Δ) · P_G3b(Δ) · P_G4(Δ) · P_G5(Δ).
• All probabilities apply to the regular gain path, conditional on passing G0 to G2, without additional selection and without losses from being overtaken.

In the oracle the test is checked here every 20 pairs. If it is checked after every pair, the error rate at Δ = 0 rises from 4.72 to about 4.9 % in G4 and from 4.37 to about 4.5 % in G5. This changes none of the statements.

Each stage on its own

The values for G3 depend on the spread of the pair results; the range holds for a spread of 0.15 to 0.25. The threshold of −14 Elo then corresponds to −33 to −20 nElo. G3 is thus a cheap damage filter.

G3b is the tightest point for small gains. At Δ = +4 a candidate passes there with only 25.3 %; with 49.4 % it ends at the cap and is parked.

The column “counted” comes from the daily sheets and the oracle register. Up to the state of the audit, G4 stood at 46 of 80; one more rejection followed on 2026-10-25.

The whole chain: five calculations

• Calculation 1: grid convolution, a closed-form series with a correction for discrete check points and a simulation with the pipeline's real LLR function.
• Calculation 2: quadrature, 200,000 random paths, a simulation with pentanomialpentanomial Games are played in pairs with swapped colours; the five counts are the pairs scoring 0, ½, 1, 1½ and 2 points. distributions and the real LLR function, plus a direct simulation of the whole chain with 2 million candidates without gain. It gave 59 promotions, that is 0.00295 %.
• Calculation 3: dynamic programming over states, an exact Monte Carlo calculation per block and 100 runs of the real decision function.
• Calculation 4: Monte Carlo in the Brownian model, fixed before any of the other deliveries was read.
• Recalculation here: grid convolution of the LLR density between the bounds, plus Monte Carlo with 400,000 paths per point. At the checked points Δ = 0, +4, +5 and +8, the two ways differ by at most 0.15 percentage points.

Calculations 2 and 3 come from models of two other vendors. Calculations 1 and 4 and this recalculation come from the same vendor family.

What is proven, calculated or open

Proof 1, the worst case. Couple two effects Δ₁ < Δ₂ through the same random numbers. The mean increment δ(2t − δ) grows with t, so at every check point the LLR at Δ₂ lies at least as high as at Δ₁. If the path at Δ₁ reaches the upper bound before the lower one, so does the path at Δ₂. So the pass probability of every stage grows with Δ, with looks and caps, and so does the product. For all Δ ≤ 0, Δ = 0 is the worst case.

Proof 2, the strict bound. At Δ = 0, exp(LLR) is a non-negative martingale with expectation 1. By the maximal inequality for non-negative martingales it reaches the height e^A with probability at most e^(−A), no matter when and how often it is checked. For G3b that is e^(−3.4) = 0.0334, for G4 1/19 = 0.0526, for G5 1/15 = 0.0667. With independent stages the product is at most 0.000117, that is 12 in 100,000; G3 can only make it smaller. For 1,000 independent trials with candidates without gain, at least one false promotion is then to be expected with at most about 11.1 %, by the calculated value with about 2.7 %. The bound is proven in the Gaussian model. The oracle estimates the ratio from the games; an error bound for this test with finitely many games is not established here.

Proof 3, the limit of G3. G3 lets a candidate through if the mean pair score lies above 1/(1 + 10^(14/400)) = 0.47986. At Δ just above 0 it averages 0.5 with standard deviation σ/√500. A pass rate of at least 95 % requires √500 · 0.02014/σ ≥ 1.645, that is σ ≤ 0.274. This holds in the normal approximation, not for every distribution of pair results.

Transfer: the counts fit about half

G3 and G3b measure at fixed nodes, the oracle at the clock. How much of an effect arrives at the clock can be estimated from the counts:

• Distribution of effects: on a grid with steps of 0.25 nElo, a normal distribution of the effects in G3b with mean −1.5 and standard deviation 4.75 nElo fits the outcomes of 2026-10-13 to 2026-10-24 best. It predicts 9.3 % passed, 76.9 % rejected and 13.8 % at the cap; observed were 9.1, 76.9 and 14.0 %.
• Expectation at full transfer: under this distribution only about 1.3 % of the G3b passers have no gain. If their effect arrived at the clock in full, 86.1 % of them would have to pass G4.
• Observation: since 2026-09-24, 46 of 80 passed, that is 57.5 %, with a 95 % interval from 46.6 to 67.7 %.
• Result: that fits a factor of 0.51, with the interval 0.41 to 0.61.

If the estimate holds, G3b with [0; 8] at fixed nodes corresponds at the clock to roughly [0; 4]. That fits the oracle with [0; 5]. The estimate has limits:

• It assumes that the effects are distributed alike in both periods and all shrink by the same factor at the clock. The two groups are offset in time and not linked; different cohorts or a changed pool of candidates could explain the numbers just as well.
• The G4 count also contains candidates of other kinds and rule versions.
• The distribution is assumed to be normal.
• The interval only covers the random variation of the 80 outcomes.

It is an estimate, not a proof.

Where the time goes

• Idle time: from 2026-09-29 to 2026-10-24 the oracle stood idle for lack of a candidate for 178.8 hours, on average 28.7 % of the day. In the last week it was 31.8 %.
• Cause: 117.7 of these hours, 66 %, were waiting for a successor after a pass. In the last week all of the idle time was of this kind.
• Gap after a pass: in the 7-day windows up to 2026-10-23 and 2026-10-24, its median without a prebuilt successor was 268 and 464 minutes. The only case with a prebuilt successor took 20 minutes. In earlier windows the medians with a prebuilt successor lay between 17 and 56 minutes.
• What it would take: to get from 31.8 % to below 10 %, at least 36.7 hours of this waiting per week would have to go.

This is a matter of operation. The bounds and the measurement conditions are not touched by it.

Recommendations

The review report summarises the shared recommendations as follows:

1. Operation first: keep two to three successors on the current state prebuilt at all times. The calculations expect from this a factor of 1.15, interval 1.0 to 1.3, up to about a third more measuring time. No measurement condition is touched; the only risk is build time on the farm. This prebuilding is currently being examined.
2. Leave G3 unchanged. By one of the calculations, a threshold between −25 and −8 Elo changes the yield by less than 3 %.
3. Add counts, all without strength values: cohorts per rule version and kind, the outcome of G4 per kind, oracle-ready candidates that went stale, and service and waiting times. With these, the distribution of effects and the transfer could be determined.
4. Do not change bounds in a hurry, but only after recommendation 1 and with a measured transfer, as a comparison fixed in advance.

Four proposals remained contested:

The review role's verdict: if the transfer is about 0.5, G3b is aligned correctly with the oracle, and no change of the bounds is supported. Today the lever lies in operation. The operator decides on measurement conditions; the recommendations are proposals.

The strongest counter-argument

That the chain rarely detects small gains may be exactly right: if small gains transfer poorly to the clock and to other books, are soon overtaken, or bring little next to rare large gains. The counts do not refute this. The short G4 runs rather point to many large effects of both signs; there a strict triage costs little. That is why the report's result reads “fitting to slightly improvable, lever in operation”, not “loosen”.

What remains unknown

• the transfer per kind of change and per book;
• the mixture of kinds and rule versions in all counts;
• the real service, waiting and core times of oracle and farm;
• the spread of the pair results at fixed nodes, which sets the threshold of G3 in nElo;
• the error rates of G0 to G2 and the yield of the speed threshold.

What this post does not say

• It gives no strength values of individual candidates. The material of the audit contained none.
• The shares of the optimum and the comparison of the alternatives come from the calculations of the audit and were not recalculated here.
• All calculations and proofs hold in the model, not in every measurement. The same Δ in every stage is a favourable assumption: at a transfer of about 0.5, the real chances of small gains are even lower.
• It gives no details of the holdout stage's conditions.

How this was checked

The basis is the review role's report, the raw answers of the three calculations, the rule versions in the pipeline code, the oracle contract, the oracle register and the 26 daily sheets from 2026-09-29 to 2026-10-24. For this post the stage curves were recalculated on a grid and checked by Monte Carlo at selected points. The chain, the estimate of the transfer and the limit of G3 were derived from them or analytically, and the proofs were carried out in the model. Idle time, gaps and all counts were recounted from the daily sheets and the register. An agent of another model family checked the text against the same sources and recalculated the core numbers independently.