Network of width 1024: plan, calculation and status before the start

Coherent's evaluation network has 768 neurons per side in its first layer. A plan tests whether a network with 1024 neurons, trained on the same data, plays stronger, with the cost in computing time included. This post shows what is planned, how long the training takes according to the plan, what has already been measured and what is still missing. The status is that of 2026-10-28, about 11:45 UTC. At that time the training had not yet started. There is no trained network yet and no playing-strength result.

The question

Does a network of width 1024 play stronger than the network of width 768 when both are trained with the same recipe, the same data and the same seed? The seed is the starting value of the random number generator; the same seed means the same order of the training data. The network of width 768 is the control. The plan was sealed before any run, that is, fixed in writing after two model families had reviewed it. Later results do not change it.

The network

Every position is translated into 768 features from the point of view of each side: 2 colours × 6 piece types × 64 squares.

feature = (enemy piece ? 1 : 0) · 384 + piece type · 64 + square

The weights of the first layer also depend on which of 8 regions the side's own king stands in (king buckets). For each side this gives an accumulator a of width B, clipped to [0, 1] (CReLU). Then follow 2 small layers with 16 and 32 neurons and an output:

y = w₃ · c(W₂ · c(W₁ · [a_to move ; a_opponent] + b₁) + b₂) + b₃, with c(x) = min(max(x, 0), 1)

The number of parameters in the finished network:

P(B) = 8 · 768 · B + B + (2B · 16 + 16) + (16 · 32 + 32) + 33

The wide network has 1.333 times as many parameters; almost all of them are in the first layer.

Data and training

• Data: a compound of 4 data sets with 56,149,275,622 rows in total. Each row is a position with an evaluation and the result of its game. The split is 80/10/10; training uses 44,862,491,563 rows, 79.9 % of the compound.

• Epochs: 16. Each epoch runs once over the whole training part, 717,799,865,008 rows in total.

• Target and loss: the network learns a mix of game result e and evaluation s in centipawns:

  z = 0.5 · e + 0.5 · σ(s / 205), loss = mean of (σ(y) − z)², with σ(x) = 1 / (1 + e^(−x))

• Optimiser: AdamW with learning rate η₀ = 0.00025, batches of 4,096 rows. The learning rate falls per step k along a cosine curve to zero:

  η(k) = η₀ · ½ · (1 + cos(π · k / (S − 1))), with S = 16 · ⌈44,862,491,563 / 4,096⌉ = 175,244,112 steps

• Same as the control: recipe, data, order of the data and seed. Only the width changes.

How long it takes

The duration depends on the throughput r, the training rows per second:

training time in hours = 16 · N0 / (3600 · r), with N0 = 44,862,491,563

• Measured in earlier runs: the control with width 768 trains at 1.37 million rows per second. On an earlier, smaller data set a network of width 1024 needed 94.7 hours, one of width 768 63.1 hours, a factor of 1.50.
• Plan assumption: 0.91 to 1.03 million rows per second for the wide network. This corresponds to 1.37 divided by 1.50 and by 1.33, the ratio of the parameters. It gives 194 to 219 hours for 16 epochs, with the final evaluation 200 to 225 hours, a good 8 to 9 days. These values are carried over, not measured on the training machine.
• Cap: the run may take at most 288 hours.
• Stop rules: 1 and 6 hours after the start the progress is recorded and the throughput r computed from it. If 16.5 · N0 / (3600 · r) + 40 > 288, the run is ended. Independently of that, it ends if epoch 1 takes longer than 54,109 seconds. Both rules apply below the same rate of about 0.829 million rows per second. They are technical: no loss or playing value decides a stop.

What has already been measured

• Speed pre-gate: a wider network costs more computing time per evaluation. For a start under the unchanged plan the ratio of search speeds had to be at least 0.875:

  r̂ = nodes per second with width 1024 / nodes per second with width 768 ≥ 0.875

  The rule was sealed on 2026-10-28 at 06:05 UTC, before any value was read. Below it, a new cost-benefit review by two model families and an explicit decision would have been needed, and an invalid measurement would have had to be repeated. At 06:06 UTC the gate was passed. This post does not give the measured value; according to the plan the approving authority also receives only the verdict.

• Pre-runs: two smaller experimental series, with 2.2 billion rows and 3 seeds and with 13.59 billion rows and 1 seed, both received the verdict “open” in the comparison by games. The plan now asks the question on the largest data compound.

How it will be decided

After the training the two networks play games against each other, the wide network against the control. The rule for when the wide network counts as stronger, as not stronger or as open is fixed in advance in the plan. There is no loss value as a gate: the training loss decides nothing, only games do.

Status on 2026-10-28

• Plan: sealed on 2026-10-27, with 2 addenda. Addendum 1 fixes the training machine, addendum 2 the speed pre-gate.
• Speed pre-gate: passed.
• Data: the compound is being copied to the training machine. According to the copy log, 2,471,903,383,090 of 4,548,091,328,767 bytes, about 54 %, were there at 11:44 UTC. The start requires a complete copy whose checksums match the accepted source.
• Start: among other things it also requires the same software as for the control, passed start probes and an explicit release. If one condition is missing, nothing starts.
• Training machine: its graphics card has been computing another run since 11:11 UTC.
• Control: the network of width 768 is still being trained; at 11:31 UTC it was in epoch 10 of 16.

What does not exist yet

• No training of the network of width 1024 has started; there is no measured throughput and no loss value.
• There is no playing-strength result. Whether the wide network plays stronger is open.

How this was checked

• Sources: the sealed plan with both addenda, the training recipe, the trainer code and the start script of the run, the entry on the speed pre-gate in the working status, the copy log and the project journal.
• Independent review: an agent of another model family checked text, chart and diagram in 2 rounds against the same sources and recalculated the figures itself. It confirmed the calculations and identified imprecise formulations, among them the verdict of the pre-runs and the evidence for the status; they have been worked in.
• Recalculation: a script checks selected source entries and recalculates key figures, including parameter counts, step count, training share, durations and the limit rates of both stop rules.