Five failures in the long stage, and a blind tempo test

The oracle measures in two stages: every candidate starts at a short time control, and only candidates accepted there proceed to the long one. Of the 56 candidates that received a final verdict between 2026-09-24 and 2026-10-13 under the same measurement contract, 5 passed the short stage and failed the long one. This entry describes what happened to these 5, what they cost and how the project checked blindly whether they searched more slowly than the ones that passed. It was written at the operator's request. It makes no claim about playing strength.

Two stages

Under this contract the short stage ran at 8+0.08 seconds with Book A, the long stage at 40+0.4 seconds with Book B, each against the predecessor. Both stages are sequential tests: they play until the data support one of the two hypotheses 0 and 5 nElonElo Normalised Elo: the Elo difference divided by the spread of the game results, so that the SPRT bounds mean the same at different time controls. with error rates of 0.05 each, but at most up to a cap.

• The cap was 96,000 games in the short stage and 48,000 in the long stage.
• Every candidate starts with the short stage. Only if it reaches the upper bound there (H1) does the long stage follow; it is adopted only with H1 in both stages.
• If the test reaches the lower bound (H0), the candidate is rejected. The data then favour 0 over 5 nElo; that does not mean it is weaker.
• If a stage ends at the cap without reaching a bound, it is undecided. That does not lead to adoption either.

The cohort

The basis is the pipeline's journal with all final verdicts up to 2026-10-14 at 22:39 UTC. Under the contract above, 56 candidates were decided:

• 31 passed both stages.
• 20 reached the lower bound already in the short stage.
• 5 passed the short stage and failed the long one.

Together the stages of the 5 failures ran for 132.4 hours, almost as long as those of the 31 that passed with 135.3 hours. The hours are wall-clock durations of the stages; the data do not show whether they contain waiting time.

The five failures

• T-0129: long stage at the cap, 48,000 games in 32.3 hours, last test statistic +2.71 against an upper bound of 2.94.
• T-0152: H0 after 29,192 games in 20.2 hours.
• T-0155: H0 after 40,178 games in 26.0 hours.
• T-0164: H0 after 20,386 games in 13.3 hours.
• T-0181: long stage at the cap, 48,000 games in 32.3 hours, last test statistic +0.18.

So three runs ended with H0 and two undecided at the cap. A failure ran 26.5 hours on average, 24.8 of them in the long stage. The two cap runs alone took 52 per cent of the hours these 5 spent in the long stage. The wall-clock time per completed long game lay narrowly between 2.33 and 2.50 seconds for all 5, measured as throughput, not as the duration of single games. The runs thus completed different numbers of games at similar throughput; the data do not separate game time from other time. For comparison, the long stages of the 31 that passed took 2.5 hours at the median.

In the short stage the 5 reached the upper bound after 4,066 to 18,968 games, at the median after 12,370. The 31 that passed needed 3,476 games there at the median. 4 of the 5 lay above the upper quartile of the passed group, 8,470 games. But the passed group ranged from 432 to 71,692 games. There is no clean separation, and this does not show that a filter on late acceptances would help.

The question of tempo

One possible explanation for the failures would be a time tax: candidates that search more slowly than the ones that passed. The project tested this with a rule that was written down and secured with a hash before any measurement. The agent measuring received the list of candidates without outcomes; the outcomes were sealed elsewhere.

• For each candidate r was measured: the candidate's nodes per second divided by those of its predecessor, from the engine's benchmark.
• Candidate and predecessor ran in pairs and alternately, with 1 thread on the same fixed core, the order switching in each pair.
• Each run lasted at least 3 seconds by an external clock. Each candidate had 10 pairs; r is the median of the pair ratios, with a 90 per cent bootstrap band.
• The statistic D is the mean of log r over the failures minus the mean of log r over the ones that passed.
• A hint of a time tax required both that D was at most log 0.99, so the failures were at least about 1 per cent slower, and that an exact one-sided permutation test gave p at most 0.05. Otherwise the result counts as "no hint".

Nodes per second are tempo, not strength. A change can be slower per node and still reach the same depth with fewer nodes; r does not measure that.

Three rounds of measurement

Only the third round produced measurements of the candidates. The statistical decision rule stayed unchanged in all rounds; the timing method changed from the second round on.

• In the first round the binaries were rebuilt. All 41 rebuilds differed from the hashes supplied for comparison, so no candidate was measured; it was later clarified that these comparison hashes were the wrong ones for the candidates. The control measurement of one binary against itself was unusable because the timing was only accurate to the millisecond.
• In the second round the existing binaries were searched for and an external nanosecond clock was used. An error in the calibration of the measuring script was noticed before the first complete pair, documented and fixed. The control measurement then gave r = 1.0011 with band [0.9964; 1.0048] from 20 pairs. All predecessors were found, but no candidate, because the list carried the wrong hash for the candidates.
• In the third round all 41 candidates were measured, 410 pairs, all binaries bit-identical to their registered hash. The shortest run took 3.835 seconds.

The result

All 41 candidates were evaluated; the 36 that passed include 5 under the previous contract (see below). The mean of log r was −0.0073 for the failures and −0.0321 for the ones that passed. Against their predecessors the failures were therefore about 0.7 per cent slower as a geometric mean, the passed ones about 3.2 per cent. D is +0.0247: relatively, the failures were not slower but somewhat faster. The exact permutation test over all 749,398 assignments gave p = 0.688. Neither condition is met: no hint of a time tax.

• T-0129: r = 1.011 [1.000; 1.019]
• T-0152: r = 0.959 [0.950; 0.974]
• T-0155: r = 0.994 [0.993; 0.998]
• T-0164: r = 1.010 [1.002; 1.013]
• T-0181: r = 0.990 [0.975; 1.004]

The ones that passed ranged from r = 0.55 to 1.76, with a median of 0.989. All 5 failures lie within this range. A side analysis planned in advance, the rank correlation between tempo and the long-stage result of the passed candidates, could not be computed because those results were missing from the inputs supplied.

An error in the cohort

The pre-registered rule describes the 41 candidates as those with a short and a long stage under the contract above. That holds for 36 of them: the 31 that passed and the 5 failures. The other 5 are candidates that passed and were decided between 2026-09-21 and 2026-09-23 under the previous contract, with the hypotheses 0 and 10 nElo and caps of 24,000 and 12,000 games. This was noticed while preparing this entry and reported to the project's lead agent.

For this entry the evaluation was recomputed without these 5: D = +0.027, exact p = 0.701. That is a recomputation, not an evaluation fixed in advance. The verdict stays the same.

What this means and what it does not

The test found no hint of the time tax defined in advance in this cohort. It does not exclude tempo effects, and it does not determine why the 5 failed. With only 5 failures it is small; "no hint" does not prove that tempo never matters. Tempo was measured in the engine's benchmark with 1 thread, not under the conditions of the oracle's games.

How this was checked

The facts come from the measurement contract, the pipeline's journal, the analysis of the failures, the pre-registered rule, the measurement data of the three rounds and the evaluation of the tempo test. Both analyses were recomputed within the project. For this entry the hours, the groups, the distribution of r and D were recomputed from the data, and the cohort was checked against the journal. An agent of another model family checked the text against the same sources.