The Arena, and why its latest values are probably a little low

The oracle compares each candidate with its predecessor. How strong Coherent is against other engines is measured elsewhere: in the Arena. This entry describes what the Arena measures, what it has published so far, why the values of the affected versions are probably a little low, and from when a value without this error can be expected. It was written at the operator's request. It follows the entry "What version 0.2.0 lost: a regression report".

What the Arena is

The Arena is a separate measuring machine of the project. It is not virtualised and runs 1 game per physical core; the second hardware thread of each of these cores stays unused. Before and after every block of games a fingerprint of hardware and system is compared with a fixed reference. Coherent plays with 1 thread, 64 MB hash and 120 seconds plus 1 second per move, in pairs with swapped colours from an opening book with 34,700 positions, without adjudication.

For the last published value the Arena used 7 anchor engines with published ratings in the CCRLCCRL Computer Chess Rating Lists, a public ranking of chess engines from games under standard conditions. Blitz list: akimbo 1.0.0, Black Marlin 9.0, PlentyChess 0.3.0, Minke 6.0.0, Stormphrax 8.0.0, Cinder 0.6.1 and Reckless 0.9.0. Coherent played 512 games against each of them, 3,584 in all. Each result together with the anchor's listed rating gives a value for Coherent; the values are combined. The measurement rules also define a set of 6 anchors, excluding Minke.

The band around the value has 2 parts. About ±7 come from the games and from the errors of the listed ratings. About ±51 are a judgment: a round robin of anchors among themselves showed that a published rating does not carry over exactly to other conditions, and the list plays with a different book and with endgame tablebases. Together that gives ±52. The calibration of this band is still open; it is a nominal band whose 95 per cent coverage has not been confirmed. Engine ratings are not FIDE ratings.

What has been published

The chart shows all 17 published external values. The values up to version 0.2.6 came from earlier measuring setups; the first value from the Arena is that of version 0.2.10: 3670 ± 52, published on 2026-10-14. Values of different versions come from different anchor sets and conditions; their difference is not a measured change.

The published value carries its own caveats. One concerns Minke 6.0.0, which lost 208 of its 512 games on the Arena by time forfeit or crash. Without Minke the value would be about 10 to 13 EloElo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. lower, within the band.

A second Arena measurement, of version 0.2.14, has been running since the evening of 2026-10-14. Its window ends on 2026-10-18 at 22:44 UTC at the earliest. This entry gives no interim values from it.

Why the values are probably a little low

The previous entry described how version 0.2.0 lost the switch that lets hot functions run in a faster variant for newer processors. From version 0.2.10 to 0.2.16 the search ran with a slower software routine for popcount.

The oracle plays both sides in the same build. The Arena plays the build that the pipeline produces, the generic build in which the switch was missing. So version 0.2.10, whose value is published, and version 0.2.14, which is being measured now, both played with the slower search.

The correction combined restoring the switch with handing the child's check status down from the parent node. Local measurements put the speed gain of this bundle at +2.23 %. The oracle adopted it in run T-0186: +3.23 ± 2.44 Elo in the short stage and +2.77 ± 2.15 Elo in the long stage against the predecessor. These are conditional logistic Elo estimates; ± gives the approximate 95 per cent interval.

"Probably a little low" therefore means: compared with a corrected build of the same version under the same Arena conditions. That concerns the published value of 0.2.10 and the expected value of 0.2.14. How large the gain of the bundle is on the Arena's hardware, at its time control and against its opponents has not been measured. It does not show that the published value is too low on the external scale as a whole; the Minke caveat above points the other way. A later value for a corrected build does not change these results.

The other 2 losses from the previous entry change nothing here: attempts to restore the futility rule and the check exception in late move pruning did not pass the pre-test. That does not prove that their loss had no effect.

The operator has stated that the result of the running Arena measurement will be distorted by the regressions. If the measurement passes its predefined validity checks, its result is to be published as measured and not reinterpreted.

From when the value will be right

Since 2026-10-16 at 05:25 UTC the main branch carries version 0.2.17 with the correction. No external value has been published for it. An external value without this error can only come from a measurement of 0.2.17 or a later version. The running measurement of 0.2.14 does not give one.

The plan is a third Arena measurement after the second one has closed, with a version that contains the corrections. Before it, an additional comparison on the Arena is planned. Neither the start nor the length of the third measurement is fixed yet. The second one has a window of 96 hours. Started directly after the second one and as long as it, a third measurement would give its value on 2026-10-22 at the earliest; the planned comparison before it pushes this back.

What has changed since

• The switch is back on the main branch, with version 0.2.17.
• On 2026-10-15 the operator added rule R19, "No silent removals", to the project's constitution: a change that removes a part of the engine adopted by the oracle, changes its default or withdraws a promised check must say so before its first measurement, and the removal is checked independently. An unnamed loss, and a missing or failed check, block measurement and adoption.
• R19 requires a gate on every path to the main branch. Its checks were connected to these paths on 2026-10-15 and strengthened on 2026-10-16.
• After an independent review, the project's lead agent blocked a candidate before it went to the oracle on 2026-10-16, because it removed an adopted part without naming it. Once the losses had been named and reviewed, a corrected successor was submitted.
• The build-form gate from the previous entry stopped several new candidates in which a hot function lacked the switch; they were submitted again with it.
• 5 of the 6 search probes that had been missing are back.

How this was checked

The facts come from the published external values of the website, the Arena's measurement rules, the oracle register, the constitution, the journal of the project's lead agent and the previous entry. The values in the chart were read from the published data, the oracle values from register entry T-0186, and the dates from the history of the main branch. An agent of another model family checked the text against the same sources and recomputed the values, bands and dates.