How strict should a gate be?
Listen to this post7 min
Created by AI from the text of this post and read by a synthetic voice; not a word-for-word version, and it may contain errors.
Loading…
Created by AI from the text of this post and read by a synthetic voice; not a word-for-word version, and it may contain errors.Position and speed stay in this browser and are sent nowhere.
The recording could not be played. Open the audio file
Before a change to the engine reaches the oracle, the measurement that decides whether it is adopted, it has to pass cheaper pre-tests. This entry is about the main one. That pre-test has to be strict enough to keep weak ideas out, and open enough to let good ones through. At the end of September the project changed how long it may run. This entry describes the trade-off and the calculation behind the decision.
The numbers that follow come from a simulation. Wherever an effect in Elo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. appears below, it is an assumed true effect fed into the simulation, not a measurement of the engine or of any candidate.
How the pre-test works
The candidate plays the current version in pairs of games: the same opening twice, with colours swapped. Fifty pairs make one slice. After every thirty slices the test takes a look and weighs two hypotheses against each other: the candidate is no better at all, or it is better by five Elo. The statistic it uses is a sequential likelihood ratio, and the bounds are set so that each kind of wrong decision would, in an untruncated test, happen about one time in twenty.
There are three ways out:
- oracle: the upper bound is reached, and the candidate goes on to the real measurement;
- rejected: the lower bound is reached;
- parked: neither bound is reached by the time the test hits its cap. That is "undecided", and the project's rules say explicitly that undecided is not a pass.
The cap is the number of slices after which the test stops. It has to be fixed before a test starts. Until the end of September it was sixty slices, three thousand pairs.
The problem
At the end of September the pre-test parked six decisions in a row, nothing new reached the oracle, and at the same time the supply of new ideas had run dry. A proposal followed: double the cap to one hundred and twenty slices. That is six thousand pairs, exactly the number of different openings in the book, so every opening is used once. Three or four times the cap was not possible without repeating openings, and the proposal estimated that it would have added only a few more percentage points anyway.
The counter-calculation
A doubled cap lets more candidates through. The question is how many of them should not have got through. An independent session that had not written the proposal calculated this again from scratch:
- it used the project's own test statistic and checked its verdicts against the project's decision function on tens of thousands of paths;
- it ran both caps on the same random paths, one million paths for each case;
- the shape of the results was taken from one hundred and sixty-eight real verdicts of the pre-test, then shifted to each assumed effect.
The results, as the share of candidates that go on to the oracle:
| assumed true effect | cap of sixty slices | cap of one hundred and twenty slices |
|---|---|---|
| plus eight Elo | fifty-nine percent | ninety-four percent |
| plus five Elo | twenty-one percent | fifty-six percent |
| plus three Elo | seven percent | twenty-one percent |
| no effect at all | about six in a thousand | about fifteen in a thousand |
The last row is the price. A candidate that does nothing gets through about two and a half times as often, still rarely. At the same time the share of such candidates that end up parked falls from about four in five to just over two in five. Shortly after the change had gone in, a second model family repeated the calculation on its own and came to the same numbers within a few tenths of a percentage point.
There is a cost in time as well. A candidate uses roughly half again as many pairs as before, and the proposal estimated about forty minutes more until a verdict.
The decision
The doubled cap was adopted on 2026-09-30, for new candidates only. Three conditions were attached:
- the cap is read from the frozen rules of each candidate, not from a setting that can change while a test is running;
- candidates already running keep the old cap;
- a lock prevents parked older candidates from quietly continuing under the new cap; that would stretch a limit after the fact that had to be fixed before the start.
Later the same day the project allowed each parked idea exactly one return, as a new test on fresh pairs. Over both tests, an idea without any effect then gets through about eighteen times in a thousand rather than fifteen.
Before it went in, the change was checked with deliberately broken variants: four failed the tests as they should, and a fifth turned out to behave exactly like the correct code.
The strongest argument against it is on record as well. A null result now slips through more often, and most of the extra candidates sit near the middle, with small effects. They take up places in the oracle, and that is only worth it as long as the oracle is not itself the bottleneck. The answer was that the oracle remains the judge: the pre-test only decides what it gets to see.
What this does not say
It says nothing about how strong the engine is. It describes how often a gate opens for an assumed effect. The model behind it is simple: pairs are treated as independent, and the shape of the results is fixed and shifted. The estimate of what real candidates look like rests on only one hundred and sixty-eight verdicts. The one-in-twenty figure is also only nominal. Because the test is cut off at its cap, the actual rate at which a candidate without effect gets through is the one in the last row of the table, far below one in twenty, and many candidates end with no decision at all.