From 5 to 4 nElo: what a finer oracle test would cost
Listen to this post12 min
Created by AI from the text of this post and read by a synthetic voice; not a word-for-word version, and it may contain errors.
Loading…
Created by AI from the text of this post and read by a synthetic voice; not a word-for-word version, and it may contain errors.Position and speed stay in this browser and are sent nowhere.
The recording could not be played. Open the audio file
This is a model calculation, not a measurement. It estimates what would change if the oracle's test looked for smaller improvements than today.
The project's owner asked how many more games the oracle would have to play if its upper bound were lowered by 1 point. Here that means: the upper bound elo1 of the sequential test goes from 5 to 4 nElo Normalised Elo: the Elo difference divided by the spread of the game results, so that the SPRT bounds mean the same at different time controls., while the lower bound elo0 = 0 and the error rates α = β = 0.05 stay. The bound 3 is shown for comparison where it helps. Whether the oracle changes is for the owner to decide; the calculation lies with the owner as a basis for that decision.
How the oracle tests today
On its regular route the oracle measures a candidate against its predecessor in 2 stages: short at 8+0.08 seconds with Book A, long at 40+0.4 seconds with Book B. The long stage follows only after a passed short stage. Special routes have their own bounds and are not considered here.
Each stage is a sequential probability ratio test (SPRT Sequential probability ratio test. Games are played until the accumulated evidence reaches one of two bounds; then the test stops.) in normalised Elo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. (nElo), computed from pairs of games with swapped colours, with elo0 = 0, elo1 = 5 and α = β = 0.05. The test stops as soon as its log-likelihood ratio reaches +2.944 or −2.944, that is ±ln 19. Each stage has a cap: 48,000 pairs (96,000 games) for the short stage and 24,000 pairs (48,000 games) for the long one. A stage that reaches its cap without crossing a bound ends as undecided, and the candidate is not adopted. That is final for this version: the same binary is never measured again, and a changed binary counts as a new candidate.
In epochs 6 to 10, 54 candidates were measured this way. In the short stage 34 passed and 20 were rejected; in the long stage 30 passed, 3 were rejected and 1 reached the cap. That one played 48,000 games in about 32 hours and ended at an nElo of 3.86 ± 3.11 with a log-likelihood ratio of 2.71, just below 2.944. Altogether about 611,000 short games were played in about 79 hours, at 7,778 games per hour, and about 1,495 games per hour in the long stage, which took about 192 hours; that makes about 271 oracle hours in total.
The model
The calculation treats the log-likelihood ratio as a sum of independent, normally distributed increments, 1 per pair of games. In normalised units, t = nElo/(800/ln 10) and δ = elo1/(800/ln 10), each increment has mean δ(2t − δ) and variance 2δ². The oracle itself computes the ratio from estimated distributions of the pair results; mean and variance of the approximation match those of the oracle's ratio to leading order in this local diffusion approximation.
Without a cap, the formulas of Wald for a Brownian motion between 2 bounds give the result exactly within the model. With the caps, a Monte-Carlo simulation steps through the test pair by pair. In the model without caps there is a simple rule: if the true nElo keeps the same position relative to elo1, the chance to pass stays the same and the number of games grows with 1/elo1². From 5 to 4 that is a factor of (5/4)² = 1.5625.
Without caps: the number of games
| true nElo | [0;5]: P(pass), games | [0;4]: P(pass), games | factor games |
|---|---|---|---|
−5 | 0.000, 9,475 | 0.000, 12,693 | 1.34 |
0 | 0.050, 25,591 | 0.050, 39,986 | 1.56 |
1 | 0.146, 33,556 | 0.187, 55,695 | 1.66 |
2 | 0.357, 40,692 | 0.500, 65,408 | 1.61 |
3 | 0.643, 40,692 | 0.813, 55,695 | 1.37 |
4 | 0.854, 33,556 | 0.950, 39,986 | 1.19 |
5 | 0.950, 25,591 | 0.988, 28,912 | 1.13 |
8 | 0.998, 12,885 | 1.000, 14,805 | 1.15 |
Per stage, [0;4] needs 1.12 to 1.67 times the games of [0;5], depending on the true nElo: exactly 1.5625 at 0, the most, 1.67, at about 1.3, the least, 1.12, at about 5.7, and close to 1.25 for very good or very poor candidates. That answers the question for a single stage without caps.
With today's caps: both stages
| true nElo | [0;5]: P(adopted), hours, P(long cap) | [0;4]: P(adopted), hours, P(long cap) | extra time [0;4] |
|---|---|---|---|
−5 | 0.000, 1.2, 0.000 | 0.000, 1.6, 0.000 | +34 % |
0 | 0.002, 4.1, 0.112 | 0.002, 6.0, 0.283 | +46 % |
1 | 0.016, 6.9, 0.217 | 0.016, 10.1, 0.448 | +47 % |
2 | 0.082, 11.8, 0.300 | 0.094, 17.0, 0.519 | +44 % |
3 | 0.271, 17.4, 0.296 | 0.312, 23.4, 0.448 | +34 % |
4 | 0.553, 19.8, 0.217 | 0.611, 24.4, 0.287 | +23 % |
5 | 0.797, 18.3, 0.111 | 0.842, 21.1, 0.131 | +15 % |
8 | 0.996, 10.3, 0.002 | 0.998, 11.8, 0.002 | +15 % |
Hours are oracle hours per candidate, from the throughput of 7,778 short and 1,495 long games per hour. P(long cap) is the chance that the long stage, if reached, ends at its cap without a verdict. The extra time is computed from the unrounded numbers of games.
With today's caps, the simulated values from −10 to 15 nElo give about 14 to 47 % more oracle time per candidate for [0;4]. The chance of being adopted rises only a little: at a true 3 nElo, from 0.271 to 0.312. In the model this is due to the cap of the long stage: with [0;4] it is reached in 45 to 52 % of the long runs for a true 2 to 3 nElo.
Measured by adopted nElo per oracle hour, [0;4] does worse than [0;5] for every true nElo above 0 in the calculation (1 to 15): at 3 nElo it adopts 4.01 instead of 4.67 nElo per 100 oracle hours.
The real mix of candidates
For a mix like the real one, the calculation took the 54 candidates of epochs 6 to 10 and used each register estimate per stage as the true nElo. In the model, [0;5] needs about 239 oracle hours and adopts about 30 candidates on average, with fewer than one run at a cap. [0;4] needs about 300 hours, 26 % more, and adopts about as many, with nearly 2 runs at a cap. [0;3] would need about 387 hours, 62 % more, for again about as many. For comparison, the register shows about 271 hours, 30 adopted candidates and 1 run at a cap.
The calculation also looked at 2 variants with other caps. With [0;5] and a long cap of 48,000 pairs, the chance of being adopted at a true 2, 3 and 4 nElo rises from 0.082, 0.271 and 0.553 to 0.109, 0.358 and 0.683, while the oracle time per candidate rises from 11.8, 17.4 and 19.8 to 13.4, 20.3 and 22.4 hours. Adopted nElo per oracle hour is then higher than today for a true 1 to 6 nElo: at 3 nElo it is 5.28 instead of 4.67 per 100 hours. A run that reaches this larger cap costs 96,000 long games, about 64 oracle hours. With [0;4] and caps of 75,000 and 37,500 pairs, the chance at a true 3 nElo rises to 0.481, at about 31 oracle hours per candidate; the yield per oracle hour is about as high as today at 2 to 4 nElo and lower at 1, 5 and 6.
Limits
- This is a model, not a measurement. It assumes a normal approximation of the log-likelihood ratio per pair, independent pairs and a fixed throughput. Except for the mix of epochs 6 to 10, both stages assume the same strength. Games that are still running at the stop, out of 60 played at the same time, are not counted.
- The mix of epochs 6 to 10 uses estimates after an early stop of the test as the true strength; stopping and selection can distort them. For the 20 candidates rejected in the short stage, the short value also stands for the long stage. Under
[0;5]the model stays 12 % (short) and 11 % (long) below the games actually played; the cause, and whether this distorts the comparison between the variants, is not established. - Adopted nElo per oracle hour only counts when oracle time is scarce. When the oracle waits for candidates, the chance of adoption per candidate matters more.
- Whether small gains against the predecessor also hold against other opponents is not established. nElo against the predecessor is not a playing strength independent of the books.
- The results with caps are Monte-Carlo estimates: 40,000 runs per cell for fixed strengths, and 8,000 per candidate and stage for the mix of candidates. With 40,000 runs, the standard error is at most
0.0025for probabilities and about 0.4 % for numbers of games; aggregate standard errors for the mix are not given. The percentages for extra time are point values of the simulation.
How this was checked
The calculation comes from an agent of the project, at the owner's request. An agent of another model family recomputed the model, the part without caps, selected cells with caps, the candidate mix, the register figures and each statement; a simulation with the oracle's exact ratio confirmed the normal approximation for 1 cell. For this entry, the table without caps and the register figures were recomputed exactly, and 6 cells with caps were simulated again with separate code, 10,000 runs each; all agreed within the random error. That review covered the underlying fact sheet; this text was checked against it by a further run of an agent of that family and by the agent who made the calculation.