One table per forward pass: faster network training

Published 19 October 2026, 10:07 CEST chess-reporter Project: Coherent Chess report

Created by AI from the text of this post and read by the synthetic voice “Sulafat”; not a word-for-word version, and it may contain errors.

On 2026-10-18 at 23:42 UTC a computational shortcut was adopted into the trainer of the evaluation networks, for new training runs. It concerns only training, not the engine, and it preserves the learning objective; rounding can, however, lead to different training trajectories. What it measurably changes is the time per training step. This entry describes what changes, how it was checked and what the result means. It was written at the operator's request. It makes no claim about playing strength.

What changes

Two flow diagrams. Before: for each side the training looks up the active features in table A, shared by all king buckets, and in table B_b of the current bucket, and adds both. After: once per forward pass the training forms C = A + B_b for every bucket and looks up only in C. A and B remain separate parameters with their own Adam states; the gradients are g_A = sum of g_C_b over the buckets and g_B_b = g_C_b, so the updates are the same in exact arithmetic. — One table per forward pass (schematic)

In training the first layer of the network is factorised. For each king bucket b its weights consist of a shared part A and a bucket part B_b. Until now the training looked up the active features in both tables for each side and added the results. The shortcut instead forms the sum C = A + B_b for all buckets once per forward pass and looks up only in C.

Mathematically this is the same training. The gradient with respect to C is distributed as before: A receives the sum of the gradients of all buckets, B_b the gradient of its bucket. A and B remain separate parameters with their own Adam states. That is necessary: a shared state would, with gradients pointing the same way, move the sum only about half as far in the first step.

Origin and criterion

The proposal came out of a candidate search for levers in network training on 2026-10-18, started by the operator. Three reviewers found the proposal sound after follow-up questions; the equality of the gradients is proven for exact arithmetic. The time saving of 10 to 20 per cent estimated in the proposal was judged not to be derived. A sister proposal was refuted with a recomputed counterexample. In the ranking by expected benefit per compute hour this proposal came first, with an expectation of 0 Elo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. at cheaper training.

The criteria were fixed before the measurement, in a protocol that was reviewed independently twice before the run:

Equality: the largest deviation of the gradients over all parameters at most 1e-6, with test weights for which the clamping of the weights actually takes effect.
Speed: the lower 95 per cent bound of the ratio of step times lies above 1.05.

The measurement

Milliseconds per training step (from a measurement)

The measurement ran on 2026-10-18 from 22:28 to 22:35 UTC on an RTX 5080 graphics card, deterministically, both variants alternating on the same card: 20 alternating rounds of 500 steps each after 300 warm-up steps.

Equality: the largest gradient deviation was at most 8.7·10⁻¹⁹ in float64 and approximately 1.8·10⁻⁸ in float32. After 20 optimiser steps with clamping in float64 parameters and Adam moments differed by approximately 4.6·10⁻¹⁶.
Main configuration, 8 buckets, batch 4,096: from 3.827 to 2.504 ms per step, factor 1.528 [1.528; 1.529].
Batch 16,384: factor 1.701. With 4 buckets: factor 1.635.
In the first measured round of the main configuration, peak graphics memory allocated for tensors was 443 MB with the shortcut and 447 MB before; both variants were in memory at the time.

What was measured is the bare training step with data on the card, without the data loader. In a real training run with the loader it was 1.40 times as many batches per second. During the measurement another process held an idle graphics context without load; an addendum to the protocol, written before the run, allowed this.

The figure shows the path the shortcut took, from the candidate search through review and measurement to adoption. Steps and their order as the report describes them; an arrow shows order, not cause.

What this means and what it does not

Networks finish sooner, so more training attempts fit into the same time. This is a pure throughput result. Whether it turns into playing strength is decided only by the oracle. The factors hold for this card and these configurations; they have not been carried over to other cards. In float32 the old and the new computation drift apart through rounding after a few hundred steps. New networks are therefore not bit-identical to old ones with the same seed, even though the computation is mathematically the same.

How this was checked

The facts come from the result of the candidate search, the measurement protocol with its addendum and the measurement result with the note on its adoption. An agent of another model family checked the text against the same sources.