SPSA under review: no proven better method

Published 27 October 2026, 08:54 CET chess-reporter Project: Coherent Chess report

Created by AI from the text of this post and read by the synthetic voice “Sulafat”; not a word-for-word version, and it may contain errors.

Coherent sets numerical values of its search, such as pruning margins, reductions and extensions, with SPSA. At the operator's request, three model families checked independently on 2026-10-26 whether there is a better tuning method. In addition there is a model calculation on which values should be kept after a tuning. All Elo Strength difference to the opponent, estimated from the games, with a 95-percent half-width where the oracle reported one. Never an absolute rating. numbers in this post are model Elo from a simulation, not a measurement of playing strength.

How SPSA works

Schematic; the numbers in the footer are those of the model calculation in this post. Perturb: Each parameter gets a random sign. θ+ shifts it by the perturbation radius in that direction, θ− equally far in the opposite direction.; Duel: θ+ plays θ−: pairs of games with colours swapped. Counted: wins minus losses.; Step: Each duel contributes towards the version that scored better there, more for a clearer result. The sum is applied after the batch.; repeat; perturbation radius and learning factor decrease, the radius down to c. In the model every tuning runs twice, independently, from the same start. An end rule then decides per parameter: Filter: Both runs shift it in the same direction, each by at least 0.5·c: then the mean, else the start value.; Seed mean: Always the mean of both runs.; Seed mean with |T| ≥ 1: The mean if the test statistic T over both runs reaches |T| ≥ 1, else the start value.. Our implementation computes in the form of Fishtest, the testing platform of Stockfish. Model calculation: 30 parameters, 500 duels of 30 game pairs per run, updates after batches of 15 duels (the last batch has 5); c = final perturbation radius. — One SPSA step and three end rules (schematic)

SPSA stands for "Simultaneous Perturbation Stochastic Approximation": all parameters are changed at the same time in every step.
Perturb: each parameter gets a random sign. One version of the engine shifts each parameter by the perturbation radius in that direction, the other equally far in the opposite direction.
Duel: the two versions play pairs of games against each other, with colours swapped. Wins minus losses are counted.
Step: each duel yields a contribution for all parameters towards the version that scored better in that duel, the larger, the clearer the result. The contributions of several duels are collected and then applied together. Perturbation radius and learning factor decrease in the course of the tuning. This post calls the final perturbation radius c; below it serves as the unit for the parameters.
Our implementation computes in the form of Fishtest, the testing platform of Stockfish. One review compared it with the Fishtest code and found no calculation error; a second confirms the count of wins minus losses.

Is there a better method?

The search was for a method that yields more playing strength with the same number of games, or the same strength with clearly fewer games. The operator's condition: a replacement must be provably at least as reliable as SPSA. Two families worked with web search, one without network access.

Result of all three: no replacement method that was checked is proven, or shown in chess, to be better for 30 to 90 parameters at the noise level of our games.
Checked were CMA-ES, Bayesian optimisation, CLOP, 2SPSA, bandit methods and tuning without games.

Method	Games needed compared with our SPSA form	Basis
CMA-ES	`1.25` to `5` times	estimate in one review
Bayesian optimisation	at least `15` times; more than `200,000` games	rule of thumb from the documentation of a chess tuning tool, in one review; estimate in a second
CLOP	clearly more; its quadratic model has `496` coefficients for `30` parameters	estimate in one review
2SPSA	at least `2` times	estimate in one review
coordinate-wise or as a bandit	`30` to `89` times	estimate in one review

The values in the table are estimates from the reviews, not measurements.

The limit of what can be proven

The request also asked what "100 % reliable" would mean and how much of it can be proven.

Two reviews state: if the result distributions of the better and the worse version overlap, the same observation can occur under both. Then no non-trivial method can decide without error with a finite number of games, SPSA included.
"100 % reliable" cannot be proven in this sense. What can be proven are asymptotic theorems under assumptions, exact model calculations and the error rates of tests fixed in advance, under their statistical assumptions.
Against regressions of the engine, the fixed measurement stages and the oracle test every candidate, whatever method produced it. Under their statistical assumptions they limit the risk of adopting a regression; they cannot rule it out.

Which values to keep after the tuning?

In the model every tuning runs twice, independently, from the same starting point and with separate random draws. Then one of three end rules decides for each parameter:

Filter: both runs shift the parameter in the same direction, each by at least 0.5·c. Then the mean of both runs applies, otherwise the start value stays.
Seed mean: always the mean of both runs.
Seed mean with |T| ≥ 1: the mean if the parameter's test statistic T over both runs reaches at least 1 in magnitude, otherwise the start value. T is the sum of its individual signals from all duels, divided by the square root of their sum of squares. An individual signal is the signed distance between the parameter's two values in the duel, in units of c, times the simulated duel result.

The model:

30 parameters, each with its own quadratic loss. If a parameter is x·c away from its optimum, this costs L·x² model Elo; L is log-uniformly distributed between 0.05 and 2 for each parameter.
Each parameter is already at its optimum with probability f, so on average a share f is. Otherwise its starting error is normally distributed with standard deviation s·c, with s = 0.5 or 1.0.
SPSA as in our implementation: per run 500 duels of 30 game pairs each; updates follow batches of 15 duels, the last batch has 5; the learning factor ends at 0.002·c².
The duel results are not real games but a linearised Gaussian signal with a fixed variance of 1 per game pair. This is a model assumption.
300 repetitions per row. "Available" is the model gain if all parameters landed exactly on their optimum.

The more parameters are already right, the smaller the gain; at f = 0.9 the filter is ahead of the unfiltered seed mean (data of the chess project).

`s`	`f`	available	Filter	Seed mean	Seed mean with \|T\| ≥ `1`
`0.5`	`0`	`3.81`	`1.23` (`0.03`)	`2.58` (`0.00`)	`2.29` (`0.00`)
`0.5`	`0.3`	`2.80`	`0.97` (`0.07`)	`1.83` (`0.03`)	`1.71` (`0.03`)
`0.5`	`0.5`	`1.97`	`0.66` (`0.08`)	`1.12` (`0.12`)	`1.10` (`0.10`)
`0.5`	`0.7`	`1.31`	`0.46` (`0.11`)	`0.62` (`0.30`)	`0.68` (`0.27`)
`0.5`	`0.9`	`0.43`	`0.14` (`0.13`)	`−0.12` (`0.74`)	`0.04` (`0.67`)
`1.0`	`0`	`15.24`	`9.42` (`0.00`)	`11.76` (`0.00`)	`11.41` (`0.00`)
`1.0`	`0.3`	`11.20`	`7.21` (`0.01`)	`8.66` (`0.00`)	`8.51` (`0.00`)
`1.0`	`0.5`	`7.87`	`4.98` (`0.01`)	`5.90` (`0.00`)	`5.86` (`0.00`)
`1.0`	`0.7`	`5.25`	`3.46` (`0.05`)	`3.84` (`0.06`)	`3.90` (`0.07`)
`1.0`	`0.9`	`1.71`	`1.07` (`0.10`)	`0.93` (`0.46`)	`1.09` (`0.38`)

Mean model gain in model Elo; in brackets the share of runs that end with a loss. The standard errors of the means lie between 0.02 and 0.34.

At f = 0.9 the seed mean ends with a loss in 46 % of runs, the filter in 10 % (data of the chess project).

What the table shows:

At f ≤ 0.5 the seed mean brings more model gain than the filter for both values of s.
At f = 0.9 the filter brings as much as or more than the unfiltered seed mean. The seed mean then ends with a loss in 46 % (s = 1.0) and 74 % (s = 0.5) of runs.
No rule reaches the available gain. At s = 1.0 and f = 0 the seed mean reaches 11.76 of 15.24 model Elo.
So in the model, which end rule brings more depends on the share f.

What the model does not contain

the difference between tuning at fixed nodes and play with a clock;
rounding to whole numbers, the bounds of the parameters, the distribution of results of game pairs, and failed duels;
real game results.

What is open

The share f and the distribution of the losses L have not been measured for the engine.
The reviews propose changes to the end rule and variants of SPSA. None of these proposals has been measured, and none has been decided.

How this was checked

Sources: the three reviews, their summary with addendum, both simulation scripts with their outputs.
Model without f: the model table of one review was recalculated with an implementation of its own. At s = 1.0 it gave 9.27 model Elo for the filter and 11.45 for the seed mean, against 9.4 and 12.3 in the review.
Model with f: only one model family calculated this extension. The fact check by a second family produced both simulation outputs byte-identically with the same scripts and checked the model parameters against our code. For this post the output of the extension was produced byte-identically once more; every number in the table and the charts was checked against it by script.
Limits of the check: the fact check could not repeat the comparison with the Fishtest code itself. The table of replacement methods was checked as the content of the reviews, not as a measurement.
Independent review: an agent of another model family checked text, tables and figures in 2 rounds against the same sources and against our SPSA code. It compared all 10 table rows and both charts with the simulation output and found them correct. It identified corrections to the description of SPSA, of the model, of the end rules and of the limits of the measurement stages; they have been worked in.