---
title: 'PolyStep: Gradient-Free Forward Optimizer'
url: https://www.emergentmind.com/topics/polystep
type: topic
---

# PolyStep: Gradient-Free Forward Optimizer

PolyStep is a forward-only, gradient-free optimizer for training models whose forward pass is genuinely non-differentiable. It targets settings such as hard leaky integrate-and-fire spiking neurons, integer rounding, argmax routing, staircase activations, hard mixture-of-experts gates, black-box simulators, reinforcement-learning policy search, and piecewise-constant discrete objectives. Rather than estimating gradients, PolyStep evaluates the loss at vertices of a small polytope around many local parameter blocks, computes soft cost-dependent weights over those vertices, and moves each block to a barycentric average of low-cost vertices. The practical update uses only batched forward passes and a row-wise softmax, while the paper shows that this update is the one-sided limit of a regularized optimal-transport problem; the name “PolyStep” is explicitly tied to a polytope-based step with OT geometry [2605.01928].

## 1. Problem setting and conceptual basis

PolyStep is designed for models in which non-differentiability is part of the forward computation itself, not merely an optimization inconvenience. The paper lists hard operators such as $\mathrm{sign}(\cdot)$, $\mathrm{round}(\cdot)$, $\mathrm{argmax}(\cdot)$, and $\mathbb{1}[u \ge u_{\mathrm{th}}]$ as representative examples. In such systems, exact backpropagation is either undefined or uninformative, and common replacements such as surrogate gradients or straight-through estimators optimize a smoothed or substituted backward model rather than the actual hard forward computation [2605.01928].

The method is also positioned against finite-difference and evolution-strategy baselines. The paper argues that SPSA, DeepZero-style perturbation zeroth-order methods, and MeZO-style estimators fail on piecewise-constant losses because small perturbations usually remain in the same flat region, so the estimate is zero with high probability. Evolution strategies such as OpenAI-ES, CMA-ES, PEPG, and ARS are treated as perturbation-based search-distribution methods rather than structured local optimizers. PolyStep differs in that it does not form a perturb-and-regress gradient estimate. Instead, it evaluates a structured set of candidate directions, organizes the resulting losses into a cost matrix, computes soft assignments, and performs a barycentric displacement toward low-cost vertices [2605.01928].

This yields a specific niche. PolyStep is presented as strongest when exact gradients are unavailable or unusable, and when surrogate-gradient fidelity to the true hard forward pass is unacceptable. It is not presented as a general replacement for first-order optimization on differentiable models; the paper explicitly notes that Adam remains faster, more accurate, and more scalable when gradients exist [2605.01928].

## 2. Mathematical formulation

Let the model parameters be $\theta \in \mathbb{R}^d$. PolyStep either works in full space or in a compressed subspace of dimension $d_{\text{sub}}$. Parameters in the optimization space are reshaped into particles,
$$
X = [x_1,\dots,x_P]^\top \in \mathbb{R}^{P \times d_p},
\qquad
P = \left\lceil \frac{d_{\text{sub}}}{d_p}\right\rceil,
$$
where each row $x_i \in \mathbb{R}^{d_p}$ is a particle. The default polytope is the orthoplex
$$
\mathcal V = \{\pm e_j\}_{j=1}^{d_p},
\qquad
V = 2d_p.
$$
For each particle, PolyStep samples a rotation
$$
R_i \sim \mathrm{Uniform}(\mathrm{SO}(d_p)),
$$
and defines candidate vertices
$$
v_{i,j} = x_i + r_s \varepsilon\, R_i v_j,
\qquad
v_j \in \mathcal V.
$$
Probe points are then placed along each vertex direction:
$$
p_{i,v,k} = x_i + r_p (1+\eta_t)\varepsilon\, R_i v_j \lambda_k,
\qquad
\lambda_k = \frac{k}{K+1},
\quad
k=1,\dots,K,
$$
with jitter
$$
\eta_t \sim \mathrm{Uniform}[-\eta_{\max},\eta_{\max}].
$$
The cost matrix averages probe losses,
$$
C_{iv} = \frac{1}{K}\sum_{k=1}^K L\!\left(f_{\theta(p_{i,v,k})}\right),
$$
so $C \in \mathbb{R}^{P\times V}$ stores particle-by-vertex costs [2605.01928].

The default source marginal is uniform,
$$
a_i = \frac{1}{P},
$$
and the practical update is a row-wise softmax,
$$
T^*_{iv} = a_i \cdot \mathrm{softmax}\!\left(-\frac{C_{i:}}{\varepsilon}\right)_v.
$$
Each particle is updated by barycentric projection,
$$
x_i^{(t+1)} = \frac{1}{a_i}\sum_{v=1}^V T^*_{iv}\, v_{i,v}.
$$
This is the practical PolyStep algorithm used for the headline experiments [2605.01928].

The OT interpretation is given by the balanced entropic transport problem
$$
\min_{T\ge 0} \; \langle T, C\rangle + \varepsilon\, \mathrm{KL}(T \,\|\, a\otimes b)
\quad \text{s.t. } T\mathbf 1 = a,\; T^\top \mathbf 1 = b,
$$
with uniform marginals $a_i=1/P$ and $b_v=1/V$, together with a one-sided KL-penalized interpolation
$$
\min_{T\ge 0} \; \langle T, C\rangle + \varepsilon H(T) + \lambda\, \mathrm{KL}(T^\top \mathbf 1 \,\|\, b)
\quad \text{s.t. } T\mathbf 1 = a.
$$
At $\lambda \downarrow 0$, the solution reduces to the row-wise softmax plan; at $\lambda \to \infty$, it converges to the full entropic OT plan. This is the formal basis for describing the practical rule as the one-sided limit of a regularized optimal-transport problem [2605.01928].

## 3. Algorithmic procedure and implementation variants

A PolyStep iteration consists of reshaping parameters into particles, constructing a local rotated polytope around each particle, evaluating all probe losses in parallel, assembling the cost matrix, computing either row-wise softmax weights or a full OT plan, and updating particles by barycentric projection. The per-iteration forward-query cost is $P\times V\times K$. For the orthoplex with $V=2d_p$ and $K=1$, this is $2d_{\text{sub}}$ forward evaluations; with $K=3$, it becomes $6d_{\text{sub}}$ [2605.01928].

The paper describes several subspace variants. In full-space mode, $d_{\text{sub}}=d$. In HybridSubspace mode, each layer $l$ uses a fixed projection
$$
P_l \in \mathbb{R}^{d_l \times n_l},
\qquad
n_l = (d_{\mathrm{out},l}+d_{\mathrm{in},l})\,r_l,
\qquad
r_l = \min(r,d_{\mathrm{out},l},d_{\mathrm{in},l}),
$$
with reconstruction $\delta \theta_l = P_l z_l$. LinearSubspace and AdaptiveSubspace are also reported. Among these, HybridSubspace is described as best and most stable in the ablations [2605.01928].

The default polytope is the orthoplex, but simplex and cube variants are also described. Particle dimensions $d_p \in \{2,4,8\}$ are studied explicitly. The update rule can use either the softmax solver or full entropic OT with Sinkhorn iterations; the paper states that softmax and OT are identical in the common subspace regime, while OT becomes useful in the high-particle regime $P \gg V$ where the column marginal matters [2605.01928].

Several “turbo” features are listed: biased rotation, warm-started duals for OT, Anderson acceleration, adaptive overrelaxation $\omega$, amortized OT or plan reuse, and dual momentum warm-start. These are presented as engineering accelerations rather than defining components of the method. The paper further notes practical details relevant to reproduction: theoretical convergence requires $\eta_{\max}>0$, but reported experiments default to $\eta_{\max}=0$ for deterministic reproducibility; new applications are recommended to use $\eta_{\max}=0.05$; attention and recurrent layers require VmapSafe replacements; and SNN logits were multiplied by $10$ before cross-entropy to increase probe cost contrast [2605.01928].

## 4. Convergence theory and stationary notions

The theoretical analysis begins with a piecewise-smooth class of losses. A bounded measurable $L:\mathbb{R}^d\to\mathbb{R}$ is piecewise smooth if it is $C^1$ on $\mathbb{R}^d\setminus \mathcal D$, locally Lipschitz on connected components of $\mathbb{R}^d\setminus \mathcal D$, and $\mathcal D$ is a closed semialgebraic or definable set of codimension at least one. The paper states that this covers hard LIF, rounding, argmax routing, and floor or staircase activations [2605.01928].

Under orthoplex probing, $d_p\ge 2$, i.i.d. random rotations, the schedule
$$
\varepsilon_t = \frac{\varepsilon_0}{\sqrt{t+1}},
$$
and jitter $\eta_t \sim \mathrm{Uniform}[-\eta_{\max},\eta_{\max}]$ with $\eta_{\max}\in(0,1)$, the smoothed surrogate
$$
L_\varepsilon(\theta) := \mathbb{E}_{R,\eta,v}\big[ L(\theta + r_p(1+\eta)\varepsilon Rv) \big]
$$
satisfies
$$
\min_{1\le t\le T} \mathbb{E}\!\left[ \|\nabla L_{\varepsilon_t}(\theta_t)\|^2 \right]
= O\!\left(\frac{\log T}{\sqrt{T}}\right).
$$
Every limit point is conservative-stationary. For the headline architectures—hard-LIF, INT8 rounding, hard-MoE argmax routing, and staircase activations—the paper states that conservative-stationarity upgrades to Clarke-stationarity [2605.01928].

The paper also studies the piecewise-constant regime. For $L:\mathbb{R}^d\to\{0,1\}$ with a positive-measure success set and a non-symmetric polytope such as the simplex under a uniform reachability condition, there exists $p_0>0$ such that the first hitting time
$$
\tau_{\mathcal S_1} = \inf\{t:\theta_t\in \mathcal S_1\}
$$
obeys
$$
\Pr[\tau_{\mathcal S_1}\le T] \ge 1-(1-p_0)^T.
$$
This gives a finite-time hitting guarantee for a piecewise-constant success region. The paper notes that this theorem does not directly apply to the orthoplex because its vertex sum is zero [2605.01928].

A separate proposition analyzes schedule fragility. If a row of the cost matrix has a unique best vertex with margin $\Delta$, then for sufficiently small $\varepsilon$ the softmax becomes nearly one-hot and the step magnitude is pinned near $r_s\varepsilon$. The paper concludes that $r_s\varepsilon \to 0$ jointly is necessary for asymptotic stability, and that decaying $\varepsilon$ while holding $r_s$ fixed is fragile. This theoretical point is reflected in the empirical schedule ablations [2605.01928].

## 5. Empirical performance

The empirical evaluation spans supervised non-differentiable learning, combinatorial optimization, reinforcement learning, smooth benchmarks, and scaling or memory studies. The headline result is hard-LIF spiking MNIST, where PolyStep reaches $93.4\pm0.3\%$ test accuracy, outperforming all gradient-free baselines by over $60$ percentage points and closing to within $4.4$ percentage points of a surrogate-gradient Adam ceiling [2605.01928].

| Setting | PolyStep | Comparator context |
|---|---:|---|
| Hard-LIF spiking MNIST | $93.4\pm0.3\%$ | CMA-ES $16.2\pm8.9\%$, OpenAI-ES $33.1\pm5.5\%$, SPSA $29.4\pm5.9\%$, surrogate Adam $97.8\pm0.0\%$ |
| INT8 quantization | $97.2\pm0.1$ | Adam surrogate $98.1\pm0.03$ |
| Argmax attention | $86.8\pm0.4$ | Adam $89.1\pm0.2$ |
| Staircase activations | $93.2\pm0.3$ | Adam $97.6\pm0.1$ |
| Hard MoE routing | $90.7\pm0.2$ | Leads all gradient-free competitors |
| Binary weights | $86.2\pm1.4$ | STE $87.3\pm1.9$ |
| Ternary weights | $87.8\pm0.3$ | STE $92.3\pm0.3$ |
| MAX-SAT, $10^6$ variables | $92.6\%$ clause satisfaction | OpenAI-ES $87.8\%$ |
| MNIST, smooth baseline | $96.0\pm0.1$ | Adam $97.9\pm0.04$ |
| ETTh1 forecasting | $0.121\pm0.004$ MSE | Adam $0.187\pm0.026$ |

Across MAX-SAT instances from $100$ to $1\,000\,000$ variables, PolyStep stays above $92\%$ clause satisfaction, whereas the paper states that ES methods drop by $8$–$12$ percentage points. On the $1$M-variable case, the run took $1216$ s on a single RTX 5090 with about $3$ GB peak memory, enabled by delta evaluation over affected clauses [2605.01928].

In reinforcement learning, PolyStep matches OpenAI-ES on CartPole-v1 and Acrobot-v1. On CartPole it attains $500.0\pm0.0$ in Float32, INT8, and Binary, and on Acrobot it reports $-73.4\pm2.3$ in Float32, $-74.2\pm1.9$ in INT8, and $-71.6\pm3.1$ in Binary. The paper emphasizes that PolyStep retains performance under integer and binary quantization that collapses PPO and DQN; on CartPole, PPO falls to $39.8\pm52.9$ in INT8 and $15.7\pm5.8$ in Binary [2605.01928].

The ablations are structurally important. Entropic OT and softmax-weighted PolyStep both achieve $96.4\pm0.2$ on subspace MNIST, while hard selection rules collapse: min-cost greedy reaches $12.8\pm0.8$, and top-$k$ mean reaches $11.7\pm0.5$. In full-space MNIST with $P=50{,}885$ and $V=4$, OT yields $57.6\pm11.0$ versus $34.6\pm13.9$ for softmax, illustrating when column-marginal enforcement matters. Smaller particle dimension performs better in subspace mode: $d_p=2$ gives $95.7\pm0.2$, compared with $94.9\pm0.2$ for $d_p=4$ and $93.6\pm0.1$ for $d_p=8$. On SNNVGG11Small at $T=400$ timesteps, PolyStep uses $51.6$ MB, versus $1538.2$ MB for BPTT, a $29.8\times$ memory reduction [2605.01928].

## 6. Scope, limitations, and disambiguation

PolyStep’s main practical limitation is the standard zeroth-order one: high query cost. Each step requires $PVK$ forward evaluations, and the paper explicitly ties this to known zeroth-order query-complexity lower bounds. It also states that PolyStep does not beat backpropagation on differentiable models, is sensitive to schedules and hyperparameters such as $\varepsilon$, $r_s$, amortization, and subspace rank, and fails to solve large from-scratch smooth NLP models: on a $4.2$M-parameter SST-2 transformer trained from scratch, all gradient-free methods are near-random. In full-precision smooth continuous control, PPO far outperforms PolyStep on the Unitree G1 locomotion frontier experiment [2605.01928].

The name should also be distinguished from several adjacent “step”-based methods. “Topological Optimization with Big Steps” introduces a topology-aware large-step update strategy for persistence-based optimization, but it is explicitly not a method named PolyStep [2203.16748]. “Step-TP” is a grounded, step-level dataset for LLM-guided tensor program optimization built around LEIR, atomic optimization strategies, and structured chain-of-thought supervision; it is a stepwise optimization dataset rather than the forward-only optimizer described here [2605.25954]. In structural mechanics, the STEP and M-STEP literature concerns non-intrusive identification of polynomial reduced-order models for geometrically nonlinear structures, including the STiffness Evaluation Procedure and a modified STEP for 3D finite elements; this is a distinct STEP-family usage of “step” terminology rather than the OT-based PolyStep optimizer [2009.11377].

Within its intended scope, PolyStep is therefore best understood as a forward-only optimizer for genuinely non-differentiable forward passes, with a practical softmax implementation, an OT-derived geometric interpretation, convergence guarantees on piecewise-smooth and piecewise-constant regimes, and strong empirical performance precisely where backpropagation and standard perturbation-based zeroth-order methods are least reliable [2605.01928].

Source: https://www.emergentmind.com/topics/polystep