Arena measurement EV-3: 3701 published, 3688 without hobbes

On 2026-10-21 at 22:30 UTC the Arena delivered a new external value to the website: Coherent 0.2.21 at 3701, anchored to the opponents' CCRLCCRL Computer Chess Rating Lists, a public ranking of chess engines from games under standard conditions. Blitz ratings. The measurement behind it, EV-3, took three attempts. This entry collects the facts, recomputes the value from the published results and shows that it is about 13 points lower without a single opponent that kept losing on time. It was written at the operator's request. It follows the entry “The Arena, and why its latest values are probably a little low”.

What the Arena measures

The Arena is a separate measuring machine of the project. Coherent plays there against engines from other authors whose ratings are listed in the CCRL Blitz list, 1 thread each, 64 MB hash, 120 seconds plus 1 second per move, in pairs with swapped colours, without adjudication. 15 games run at the same time, 1 per physical core.

Each opponent gives an anchored value: its listed rating plus the EloElo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. difference that Coherent achieved against it. The evaluation combines these values into a weighted mean. The weights take the statistical uncertainty of each result and of each listed rating into account, plus a fixed allowance for the scatter between opponents, the same for all.

The value comes with 2 bands. The statistical one, from the games and the listed ratings, is about ±5 here. The second one, about ±51, is a judgment: the list plays with a different opening book and with endgame tablebases, and how a listed rating carries over to the Arena's conditions has not been measured. The Arena is not the oracle. Its values are external measurements outside the oracle's register, not proofs of playing strength in the sense of the project's constitution. Engine ratings are not FIDE ratings.

Three attempts

The second Arena measurement, of version 0.2.14, closed on 2026-10-19 without a value, because a reading that its rule prescribed during the window failed. EV-3 was to follow at once, with the latest version adopted by the oracle.

First attempt, 2026-10-19.

• Three model families reviewed the rule for EV-3, and it was frozen at 09:14 UTC.
• The Arena first had to finish the remaining games of the previous setup. A running test is not broken off. At 10:20 the operator ordered the remaining games stopped and withdrew the order within the hour; nothing on the Arena had been touched.
• While the rule waited, the oracle adopted 0.2.20 that day and 0.2.21 the next. The rule required the latest adopted version, so the planned version changed twice.

Second attempt, 2026-10-20.

• Before the start, two independent reviews judged the frozen rule not sound enough. Among other things, the family that had written it was counted as one of its confirmers.
• The rule was revised several times that afternoon. Both reviewing families already judged it sound in the third round, with issues still to resolve; further revisions were reviewed up to a fifth round.
• A dry run before the start stopped on a programming error in its test path.
• The measurement was pinned at 18:16 UTC with version 0.2.21.
• On the operator's order the procedure was cancelled at 19:53 UTC; the games of the old setup at first went on, without a published value. 3 of the opponents had listed ratings between 3567 and 3592, far below the AFEAFE Anchor-forwarded Elo: an estimate of a Coherent build’s strength on the scale of the latest anchoring, carried forward by the measured gains of the runs since. An estimate, not a measurement. Explained in full of 0.2.21 at the time, 3735 with the interval 3691 to 3780. The AFE is the project's model continuation of earlier external values with the gains measured by the oracle. The attempt ended without a value and without a delivery to the website.

At 20:24 UTC the operator wrote a new rule for all measurements against foreign engines into the constitution: all opponents lie within the interval of the current AFE, there are at least 8, and their listed 1-core rating in the CCRL Blitz list counts.

Third attempt, 2026-10-20 to 2026-10-21.

• It started at 20:38 UTC with a newly assembled set of 10 opponents. Their listed ratings run from 3694 to 3766, all within the interval.
• Coherent played 400 games against each, 4,000 in all.
• At 20:45 and 20:47 UTC, 7 and 9 minutes after the start, the operator added a point on failed opponents to the same rule. It is quoted below.
• The value was delivered on 2026-10-21 at 22:30 UTC: 3701, statistical band ±5, plus the separate band of ±51.

One opponent lost on time

Won, drawn and lost are counted from Coherent's side; the engine names are spelled as in the published data.

• Against 9 opponents Coherent scored between 39.8 and 48.0 per cent. Their anchored values lie between 3673 and 3710.
• Against hobbes 3.0, listed at 3744, Coherent scored 61.0 per cent and did not lose a single game. That gives an anchored value of 3822.
• The published data count 86 losses on time in the 4,000 games. According to the project's count in the Arena's register, 85 of them fell in games against hobbes. Since Coherent lost none of these games, all 85 were losses of hobbes. So 85 of Coherent's 88 wins against hobbes came from hobbes exceeding its time.
• In the 3,600 games against the other 9 opponents there was 1 loss on time. According to the project's check of the game headers, the opponent caused it, not Coherent.

Why hobbes kept losing on time was not established. In this setup the Arena set neither a time margin nor a move overhead for any engine, and both engines of a game shared one core. Whether that played a part was not measured.

The value with and without hobbes

For this entry the combined value was recomputed from the published results, once with all 10 opponents and once without hobbes, by the same method as the Arena's evaluation, and checked against its program.

Without hobbes the value is 12.9 lower. That is more than the statistical band and less than the band of ±51.

Why the published value still contains hobbes

The new rule of the constitution says on failed opponents: losses of any kind, including on time, are ordinary results. An opponent that does not finish its games in an unusual way, for example through a technical failure, drops out and counts not at all, as if it had not played.

hobbes finished all its games, with a result. Whether 85 losses on time in 400 games are ordinary losses or a technical failure is a matter of reading the rule. The evaluation counted them as ordinary results; the point on failures only came into the constitution after the start. In the night to 2026-10-22 the operator first classified hobbes as failed, and a corrected entry was prepared. A little later the operator decided that EV-3 stays as it is and that the operator would arrange a note on the website. The published entry therefore still reads 3701 and states that no opponent failed.

If hobbes's games are treated as a technical failure, as the operator first decided, the value is 3688, about 13 lower than published. Both values lie within the band of ±51. Leaving out hobbes's games lowers the combined estimate by 12.9 points. Whether the published value is biased, and by how much, is not established by this: what regularly played games against hobbes would have given is not known. Something similar had already happened in the first Arena measurement: there Minke 6.0.0 lost 208 of its 512 games by time forfeit or crash, and the earlier entry put the effect at about 10 to 13 Elo.

Compared with the AFE

The opponents were chosen around the AFE of 0.2.21 at the time, 3735. The published value, 3701, lies in the lower part of its interval; the value without hobbes, 3688, lies just below it. Prompted by the operator, the project examined the gap. In the measurement records and the chain of predecessors it examined it found no error, and in the comparison of the source trees no removal. It did not check the required declaration of removals for every change. Which causes make up the gap was not separated.

What this entry does not say

• It does not give a corrected official value. The published value is 3701; 3688 is a recomputation for this entry.
• It does not say why hobbes lost on time.
• It is not a statement of playing strength in the sense of the oracle. The Arena plays a build of the same source commit as the version adopted by the oracle, but not the identical binary.

How this was checked

The facts come from the published external values of the website, the Arena's records of the three attempts and of the opponent set, the program that runs the games, the constitution and its history, the journal of the project's lead agent and the project's examination of the gap. The table and both combined values were recomputed for this entry from the published results and checked against the program of the Arena's evaluation. An agent of another model family checked the text against the same sources.