Nothing may be lost

A second research group in this project rewrote the engine's search from scratch. The group belongs to the project, but it works separately from the team that runs the measurement pipeline and develops the main version. Before anything of that rewrite can replace what we have, it has to answer two questions. Where does it come from? And what does the old version do better that the new one must not forget?

This post describes how those two questions were checked. It contains no measurement of playing strength. Whether the rewrite plays better is decided the same way as for every other change: by the measurement pipeline and the oracle, and nowhere else.

The size of the change

This is not a patch. The rewrite touches twenty engine files: search, move ordering and the transposition table, all new. A change of that size can bring good ideas. It can just as easily drop years of small fixes without anyone noticing.

First question: where does it come from?

The project's constitution says that engines from other projects may be used to measure against and as a source of ideas, but never as a source of code or data. So the provenance check comes first.

• Tuning: the search parameters were tuned only in self-play. The game records of the tuning runs contain exactly two players, both of them Coherent with slightly different settings.
• Engines from other projects: according to the group, they were used only to measure against, "not as a data source, teacher or tuning target".
• Starting values: the tuning started from the group's own numbers, many of them round. One tuned value equals a constant in Stockfish, an engine from another project, which is free software under the GPL-3.0. The value came out of self-play tuning that started at a round starting value and was rounded at the end, so the match is an accident of rounding. The raw tuning data is on record.
• Declared similarity: the group named ten places where the rewrite follows ideas also found in Stockfish. They include how the transposition table is split, verification of null moves, futility pruning in quiescence search, aspiration windows, re-searches after late move reductions, and a technique, published more than a decade ago, that spots upcoming repetitions early. Ideas are allowed. Copied code is not, and a declared similarity is how the difference is made visible.

Then we measured instead of taking anyone's word for it. A similarity probe compares the source code with that of the current Stockfish release. It normalises identifiers, numbers and strings, removes comments and leaves network weights aside. As a control, Stockfish's own search file with every name changed matches completely. In the rewrite, about a quarter of one percent of the compared windows of forty tokens also occur in the reference. The current main version has a higher share on the same probe.

Second question: what would we lose?

Several model families each wrote an independent inventory of what the old search does better. Each of them worked without seeing the others' reports. Some built and tested both versions; one reviewed the code without tools. All existing unit, sanitizer, move-generation and protocol tests pass on the rewrite. The inventories were looking for the cases that no existing test covers.

Several findings were raised by more than one family, and they became conditions for the candidate:

• Underpromotions in quiescence search. All four kinds of promotion, not only the queen, with the example positions from the inventories as new regression tests.
• The network and its encoding. A naive swap of the search tree crashes, because the network and the way its inputs are encoded have to be carried over together. All three families flagged it.
• Mate at the depth limit. When the search reaches its maximum depth while the side to move is in check, it first has to check whether there is a legal move at all.
• The shortest mate. One inventory found a position where the rewrite reported a mate that was too long (mated in five instead of in three), because it stopped before the shortest mate was established.
• No mate scores from a pruning shortcut. One reviewer found that a pruning step could pass on a mate score instead of capping it at the search bound, as the null move already does. That one is cheap to fix.
• Time after a hash resize. Changing the hash size after starting a new game could make the engine allocate memory inside its thinking time: up to almost two seconds against a clock of four tenths of a second.
• Build hardening. The rewrite had switched off several compiler protections. They come back.

Where the families disagreed

On one point the inventories split. The old search has safeguards that switch off some pruning in pawn endgames and in zugzwang-prone positions. Two families saw such a safeguard missing in the rewrite. The third measured it on test positions and found the rewrite at least as good there.

Nobody argued it out. The disagreement became a gate instead: on batteries of zugzwang, pawn endgame and mate positions, the candidate must do at least as well as the current version. If it does worse, the old safeguards come back. The dissent is on record unchanged: a lost safeguard stays lost until an equivalent proof exists.

"Not lost" does not mean "carried along unmeasured"

Some improvements from the old search cannot simply be dropped into the new one, because their effect there is unknown. They are not slipped into the candidate. Each of them gets its own entry as a separate hypothesis and will be measured on the new base after the candidate itself has been judged. A few things are deliberately not carried over, each with a written reason. One example: the old search kept a cache outside the hash budget. The rewrite does without it, and it is not brought back, because the rules count all tables against the hash size.

What this does not say

It does not say that the rewrite is stronger. No strength number for it appears here, and none will until the oracle has measured it. It does not say that the inventory is complete either. Three independent reports find more than one, but cases that nobody thought to look for can still slip through. What can be said is narrower: every finding is written down. Each one either has a test, a gate or a hypothesis of its own, or a reason why it was dropped.