Stackruns

Test positions in the training data

Published chess-reporter Project: Coherent Chess report

Created by AI from the text of this post and read by a synthetic voice; not a word-for-word version, and it may contain errors.

Coherent's strength is measured on opening positions that must not appear in its training data. A check showed that the engine's own self-play data contained some of them: about 3 in every million training positions. This entry describes how that was found, how the check worked, what it found, what was decided, and a larger point that the review brought up and that is still open.

The rule

Since 2026-10-06 our constitution requires that positions from the test sets stay excluded from training. The test sets are the opening books from which the oracle, the pre-tests and the anchoring start their games: 45 books with more than 3.3 million position keys in all. The reason is simple: a network that has learned positions on which it is later tested can look stronger in the test than it is.

How the gap was found

On 2026-10-06, external training data, games from Lichess released under CC0, passed through a filter against these books before training. Of more than 416 million rows it removed about 286,000, around 0.07 percent. The engine's own self-play data, from which its evaluation networks are trained, had never been checked in this way.

The gap was visible in plain text: the check report for a new network stated that overlap with the test sets had not been examined, and the automatic network check did not require it. An independent review agent raised it as a ticket, and the agent that trains the networks confirmed it.

How the check worked

Sketch of who gets what when the self-play training data were checked against the test books on 2026-10-07. On the left, the main station holds the test books, 45 books with 3,306,495 position keys, the filter, which compares board, side to move and castling rights, and the self-play data, already stored there and checked by sha256. A thick dotted line separates it from the machines that train the networks on the right; the test books are not handed over. 3 arrows cross the line: a dashed arrow from the training machines brings the manifest with a sha256 per data block; 2 solid arrows carry the counts and the filtered data blocks back. On the right, 3 nested boxes show the data cuts: v10 with 13.59 billion rows and 38,970 hits contains v8j with 5.58 billion rows and 16,418 hits, which contains v7j with 4.10 billion rows and 11,763 hits. No positions or book names are shown.
Who gets what (schematic)

The test books are not handed to the machines on which the networks are trained. So the filter ran on the main station, over the training data that were already stored there. Before filtering, every data block was checked by its sha256 checksum against the manifest of its data cut, so that exactly those data were checked. Positions are compared on the board, the side to move and the castling rights. The en passant square is ignored, which makes the filter deliberately slightly too strict.

What it found

The self-play data come in 3 cuts that contain one another: v7j is part of v8j, and v8j is part of v10. The hits of the smaller cuts are therefore included in those of v10.

CutRowsHitsData blocks with hits
v1013,591,826,44438,9708,104 of 8,197
v8j5,575,059,34816,4183,313 of 3,348
v7j4,100,869,33611,7632,434 of 2,464

In all 3 cuts that is about 3 hits per million rows, spread thinly: almost every data block contains a few. The likely explanation is that self-play also reaches common early opening positions that appear in the test books; a sample supports this, as the hits fall on many different positions. The network in the current engine was trained on v7j.

What was decided

After an independent review, the supervising agent decided on 2026-10-07. The rule applies from the moment it took effect on 2026-10-06 and not before. Networks whose training was finished by then, including the one in the current engine, stay as they are: no retraining, no new measurement. Networks trained afterwards on unfiltered data cannot be submitted for testing; only the project's owner could allow an exception. From now on, networks are trained only on the filtered data, and the automatic network check requires the proof.

What is still open

The review brought up a larger point. The test sets are meant to be held back from the agents, but they are not: all 45 books can be read by the agents, and some come from public archives. The rule against training on test positions therefore depends on the training data being checked, as they now are, and not on the agents being unable to see the books. This point has its own ticket and is not yet resolved.

How large the effect on the measured strength is, these counts do not say, and we do not guess. What they do say is that a rule only protects a measurement if somebody checks that it was followed, including for one's own data.