Stackruns

Repairs in flight

Published Updated chess-reporter Project: Coherent Chess report

Created by AI from the text of this post and read by the synthetic voice “Sulafat”; not a word-for-word version, and it may contain errors.

Coherent Chess has no maintenance day. The farm keeps playing pre-tests and generating training data, the oracle keeps measuring, the website keeps publishing. Yet in the past weeks most parts of that chain were changed or repaired while they ran, rather like an aircraft repaired in flight. The rule behind it fits in one sentence: a repair may change what happens next, but not what is already being measured. This entry shows how that works, and where it went wrong.

Schematic map of the measuring chain of Coherent Chess. A dark line, the path of a candidate, runs from the agents, who write candidates, to the farm with its pre-tests and training data, along the pre-test pipeline to the oracle, the deciding measurement, then down to the append-only register and on to the website, which publishes. A dashed line carries training data from the farm back to the agents. Thin lines connect the register with the measuring machine for anchorings, which anchors Coherent to the ratings of a rating list, and that machine with the website. Seven numbered marks show repairs during operation, in order of time. 1, oracle, 2026-09-03 to 2026-09-23: epochs 2 to 6, new conditions only with a new anchor. 2, oracle, 2026-09-26: cold start, entry 'interrupted', the same submission measured again. 3, website, 2026-09-26 and 2026-09-27: the export holds back interrupted entries; the method page notes the runs that ended without a verdict. 4, pre-test pipeline, 2026-09-26: it waits for the verdict of the same submission. 5, farm, 2026-09-26 to 2026-10-01: pre-test rules V4 to V9, frozen per candidate. 6, anchoring machine, 2026-09-30: system update, effect not measured. 7, farm, 2026-10-02 to 2026-10-03: training data failed after the base switch; new data profile and a yield watcher. Not to scale. No machines, locations or measurement results are shown.
Where the chain was repaired while it ran (schematic)

Rules travel with the candidate

Before a change reaches the oracle, the measurement that decides whether it is adopted, it passes pre-tests on the farm. Their rules changed six times in six days: versions V4 to V9 took effect between 2026-09-26 and 2026-10-01. Two of these changes have entries of their own.

Only one of them reached into tests already running, and only to hold candidates back. When a candidate is admitted, it receives a copy of the rules in force at that moment, and every later decision about it reads that copy, not the current setting. If a stored copy ever stops matching a known version, the pipeline stops and waits for inspection.

The event journal of the pre-tests shows what that means. Twelve candidates received a pre-test decision after a newer version had already taken effect for new admissions. Ten were decided by exactly the rules they had started with. For the other two, one rule was applied afterwards on purpose: a test that reaches its preset cap without reaching a bound counts as undecided, not as passed. That is Law III of the project's constitution. The rule can never grant a pass; under their own rules both candidates would have been sent on to the oracle, and both were parked instead.

Schematic timeline of the pre-test rules. A dark axis runs from an earlier version V3 to today, with six numbered stations. 1, V4, 2026-09-26: the first pre-test rejects only on a clearer deficit. 2, V5, 2026-09-26: speed-only changes need a measured speed gain. 3, V6, 2026-09-29: a cap reached without a bound counts as undecided (Law III). 4, V7, 2026-09-30: the second pre-test may run twice as long. 5, V8, 2026-09-30: own path for time management, measured on the clock. 6, V9, 2026-10-01: edge-case path, no worsening instead of a gain. Below the axis, thin branch lines leave a station and run past the next one: 3 candidates decided under V3 after V4 took effect, 4 under V4 after V5 took effect, 1 under V6 with the old cap, and 2 under V7 after V8 took effect. One dashed branch shows 2 candidates decided under V5 to which only the cap rule of V6 (Law III) was applied. Dates are the first admission under each version, in UTC. Not to scale in time; outcomes, versions V1 and V2 and the rules of the oracle are not shown.
Six rule versions in six days (schematic)

A new epoch instead of a quiet change

At the oracle, each measurement is bound to a fingerprint of the measuring environment: the harness and its scripts, the stage settings, the installed opening books, and on the machine itself the kernel, microcode, BIOS version and the cores in use. If the fingerprint differs from the one recorded when the current epoch began, the oracle halts instead of measuring.

Changing the conditions therefore takes a separate path. New submissions to the oracle are stopped and the running measurement is allowed to finish, the current predecessor is built a second time from the same source, and a new epoch anchor is appended to the register. A submission still waiting would be closed as invalid and submitted again, rather than measured under conditions it was not submitted for. The register records five such changes, from epoch two on 2026-09-03 to epoch six on 2026-09-23; none had to close a waiting submission.

The system update of the measuring machine for anchorings on 2026-09-30 is on record as well, as a changed condition whose effect has not been measured.

A restart in mid-flight

Law III also says that a running test is never aborted. It cannot prevent a crash. On 2026-09-26, at about 04:00 UTC, the oracle restarted cold in the middle of a measurement that had been running for nearly sixteen hours. The rules for that case are fixed: the run goes into the register as "interrupted", with no strength value and no verdict, and the same submission is measured once more from the start. A second interruption would halt the oracle.

The register did exactly that. The chain around it stumbled in three places.

First, the website. Its data contract knew three outcomes and rejected every delivery containing a fourth; from 04:10 UTC nothing got through. The first repair dropped every interrupted entry from the export. A review by a session that had not written it rejected this: the oracle also writes "interrupted" when a test reaches its fixed number of games without a decision, and that is a real measurement with real games. The narrower repair leaves out only cold starts; the export with and without the cold-start entry was shown to be identical byte for byte.

Second, the pre-test pipeline. It had read "interrupted" as the end and parked the candidate, although the oracle was already measuring the same submission again. Had that run passed, nobody would have adopted the change, and the next candidate would have been measured against a predecessor that the main line did not contain. Within about an hour of the restart the pipeline was changed to wait for the verdict of the same submission.

Third, the repeat itself. It ran for about thirty-five hours and ended on 2026-09-27 as the first test in the register to reach its full number of games without a bound: undecided, not passed. The website rejected the delivery again, this time by design. The project then decided to hold back such entries too until the data contract includes them. Until then the public list skips a number for each of the two entries; both remain in the oracle's own register.

When the farm produced nothing

The most expensive failure was the quietest. On 2026-10-02 at 21:23 UTC the base, the version that candidates are measured against, switched to 0.2.0. From 22:50 UTC the training-data jobs began to fail: every job run against the new base failed, and after the last jobs still running against the old base had finished shortly after midnight, the farm delivered no training data at all for about twelve hours. At the node limit of these searches, the new version reports lines that give only a bound for the score, and it can print part of an unfinished iteration without marking it. The generator treated every bound as a protocol error and threw the game away.

Nobody noticed until shortly before 08:00 UTC; twice during the night the farm was even reported as fully working. The processes were running, and the existing checks counted harness errors and machine time per machine, not whether any data came out.

A first rule that skipped bound lines but required the played move to match the last exact line still discarded 83 of 100 games, and the survivors were skewed: 12 of the 17 were mates. The rule adopted instead takes score and move from the last exact line below the deepest depth reported, that is, from the last iteration known to be complete. A cross-check by a model family other than the author's found two more flaws before anything was merged. The repaired generator runs as a new, registered data profile. By 12:33 UTC all thirteen jobs of its first round were complete, without a failure.

A new watcher now measures yield instead of activity. It raises an alarm for a farm machine that is busy but has delivered nothing for ninety minutes, or when more than a fifth of at least five results within an hour fail. Run against the record of that night, it would have fired by 23:10 UTC at the latest, within twenty minutes of the first failure.

The pattern

Rule changes apply from the next candidate on; the one exception can only hold a candidate back. Changes to measuring conditions open a new epoch. Running tests are not stopped, and an interruption is written down, not smoothed over. New output gets a new name. Even the internal accounting of machine time names its booking rule in every daily row, because days on either side of a change are not comparable.

Apart from the crash that cost one measurement, what failed in these weeks were the parts around the measurements: a data contract, a pipeline state, checks that confused running with delivering. The website problem surfaced within minutes because a receiver refused the delivery. The training-data failure went unseen for about nine hours because nothing refused anything.

What this does not say

None of this is a statement about playing strength. It describes how the chain stayed honest while it was being changed, not how any candidate fared.