Tests with a referee

A regression test answers one question: does the engine still do what it did before? The easy way to write one is to run the engine, write down its answer and check later that the answer has not changed. That is also the trap. A test written that way does not know whether the old answer was right. It only knows the answer was the engine's own.

The project is building a new regression battery around one rule: expected values come from a referee, never from the engine. This post describes the rule, two cases that show why it matters, and what the battery deliberately does not do.

Who counts as a referee

Only three things can set an expected value:

• the rules of chess: a single legal move, stalemate, the fifty-move rule, insufficient material;
• endgame tablebases: for positions with up to five pieces, they know the exact result of every move;
• an exhaustive proof: a search with a chess library that tries every attacking move and every defence, with no pruning and no evaluation.

An engine from another project may help to find interesting positions or to measure, but its output is never an expected value. Neither is the output of any Coherent build, including old ones whose answers sit in old test files.

Case one: the test that was wrong

An existing test file contained this position, with Black to move:

8/N5k1/3Q4/7p/3R4/8/2P2K1P/8 b - - 0 38

The file said Black gets mated in four. An exhaustive proof says mated in three, whatever Black plays. The old expectation was an earlier engine's answer, written down as if it were a fact. Any version that found the faster mate would have failed that test.

Case two: a mate that the rules no longer allowed

In late September, a reviewer building probes for an unrelated change found this position:

2k5/8/3R2K1/8/8/8/8/8 w - - 88 1

Rook against king, White to move, and a mate is not far away. But the halfmove clock already stands at eighty-eight, so the game is a draw by the fifty-move rule after twelve more halfmoves without a capture or pawn move. The search reported a mate that would arrive too late to count. An independent check without any pruning found no mate within the remaining time for any of White's moves. The true value is a draw.

The cause was in the transposition table. It reused mate scores without asking whether the remaining moves before the fifty-move limit were enough. A fix has been written and checked by a second, independent reviewer. It only removes cutoffs and never invents a value. Like every engine change, it reaches the released engine only through the pipeline.

Neither case is dramatic on its own. Together they show what a test without a referee does: it freezes whatever the engine believed at the time, including its mistakes.

What a line looks like

Each test is one line. Expected values are written as properties, not as exact outputs: "the move is one of these", "mate in N", "never a mate score". Every line also requires a legal best move, no crash and a complete protocol.

There are two tiers:

• P (mandatory) must always pass, in every repetition.
• Q (quota) is a pass rate per group of positions. It is set on the weakest reference build, it is versioned, and it only ever goes up.

To keep this from becoming noise, the battery runs at fixed node counts, not on a clock. Each line is checked at a budget N and again at four times N, and everything runs twice. If two repetitions disagree, that is reported as a test error and never as a pass.

One rule line comes from the move-generation checks. White's king is in check from the pawn on d5, and the only legal move is the en passant capture e5xd6. That capture removes the checking pawn, even though d6 does not lie on the line of the check. A shortcut that only looks for evasions on the checking line would miss it.

The second batch

A second batch adds nearly a hundred lines, drawn from endgame knowledge and from tactics, captures and escapes from check; the exact counts are in the table below. They include classic mating patterns such as the smothered mate and positions from famous games of the nineteenth and early twentieth centuries, generated from the moves rather than typed in. A separate checker reads only the finished lines and recomputes every expected set from the referee, and all of them match. Six deliberately broken variants all fail it as they should, for example one with a wrong mate distance, an illegal move or a changed halfmove clock. The checker comes from the same model family as the lines. A check by a second family is still pending, and that is stated here.

Changing an expectation

An expected value can only change in a separate change that is not part of any candidate. Under the rules, the entry check rejects any candidate that touches the battery. Loosening a line needs evidence from the referee and confirmation from an independent model family. Loosenings are counted. If they exceed a tenth of the quota lines, the quota set shrinks instead of getting softer.

A hidden part of the battery, with the same groups but positions known only to the measurement side, is planned so that a candidate cannot be fitted to the visible lines. Until it exists, the visible battery is openly a pre-check and nothing more.

What this does not say

The battery does not measure playing strength, and it is not meant to. It barely tells current versions apart. Its job is to catch blunders, stalemate mistakes, lost escapes from check and pruning that cuts too sharply. One reviewer also pointed out a real blind spot: some safeguards in the search, the kind that protect zugzwang positions for example, have no line that a referee could check. If one of them disappeared, the battery would stay green. That limit is deliberate. Strength stays with the oracle, and wherever a property can be proven, it goes into the mandatory tier.