Swift-Sarsa: Fast Robust Linear Control

• Swift-Sarsa is an on-policy control algorithm that integrates true online Sarsa(λ) with adaptive step-size techniques for efficient linear control.
• It employs per-feature eligibility traces and step-size decay to bound the effective learning rate, ensuring stability amidst noisy and non-stationary inputs.
• Empirical evaluations on an operant conditioning benchmark demonstrate near-optimal lifetime rewards even in high-dimensional, distractor-rich environments.

Swift-Sarsa is an on-policy control algorithm for reinforcement learning with linear function approximation. It is built by taking True Online Sarsa($\lambda$) and augmenting it with three key ideas from SwiftTD: online step-size optimization, a bound on the effective learning rate, and step-size decay. In the published formulation, the method is presented as a fast and robust algorithm for linear control, particularly in settings where only a very small subset of input signals is relevant and the majority of the signals are noise sampled from a non-stationary distribution (Javed et al., 22 Jul 2025).

1. Origins and scope

The named algorithm appears in the extended abstract "Swift-Sarsa: Fast and Robust Linear Control" (Javed et al., 22 Jul 2025). Its immediate antecedent is SwiftTD, a TD-learning method for prediction introduced by Javed, Sharifnassab, and Sutton in 2024; Swift-Sarsa extends those ideas from prediction to control by combining the SwiftTD machinery with True Online Sarsa($\lambda$).

The method is explicitly on-policy and uses linear function approximation. It addresses discrete-action problems by learning a separate linear value function for each action, rather than a single shared scalar predictor. The paper also introduces a simple benchmark for linear on-policy control called the operant conditioning benchmark, and uses that environment to study whether the algorithm can assign credit to the relevant signals without any prior knowledge of the structure of the problem (Javed et al., 22 Jul 2025).

In that sense, Swift-Sarsa is both an algorithm and a methodological claim about linear control: step-size adaptation, explicit control of the effective learning rate, and true-online eligibility traces can be combined to preserve on-policy temporal-difference structure while improving robustness to hyper-parameter choice and to large numbers of noisy features.

2. Algorithmic structure

For each action $j \in \{1,\dots,m\}$, Swift-Sarsa maintains a separate linear value function with weights $\mathbf{w}^j_t \in \mathbb{R}^n$. The estimated value of action $j$ at time $t$ using weights from time $t-1$ is

$$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$

Collecting these predictions yields an action-value vector $\mathbf{v}_{t-1,t}\in\mathbb{R}^m$, and the policy $\pi$ maps that vector to a discrete action. The paper allows either $\lambda$0-greedy or softmax action selection.

The temporal-difference error is written in Sarsa form, but with the paper’s indexing convention:

$\lambda$1

Eligibility traces are maintained per action. At each step, all traces are decayed by $\lambda$2; the trace for the chosen action receives the True Online TD($\lambda$3)-style increment, whereas the traces for non-chosen actions are only decayed. This preserves the forward-view motivation of true-online methods while restricting active credit assignment to the chosen action.

Swift-Sarsa augments this control backbone with per-parameter step-size adaptation. For each action $\lambda$4 and feature index $\lambda$5, it maintains a weight $\lambda$6, a log step-size $\lambda$7, an eligibility trace $\lambda$8, a “delta-eligibility” trace $\lambda$9, auxiliary traces $j \in \{1,\dots,m\}$0 and $j \in \{1,\dots,m\}$1, and auxiliary variables $j \in \{1,\dots,m\}$2, $j \in \{1,\dots,m\}$3, and $j \in \{1,\dots,m\}$4. Step-sizes are parameterized as $j \in \{1,\dots,m\}$5, clipped to the interval $j \in \{1,\dots,m\}$6, where $j \in \{1,\dots,m\}$7.

A central quantity is the effective-learning-rate proxy for the chosen action $j \in \{1,\dots,m\}$8:

$j \in \{1,\dots,m\}$9

The algorithm bounds the effective learning rate through the update

$\mathbf{w}^j_t \in \mathbb{R}^n$0

If $\mathbf{w}^j_t \in \mathbb{R}^n$1, Swift-Sarsa applies step-size decay,

$\mathbf{w}^j_t \in \mathbb{R}^n$2

and resets $\mathbf{w}^j_t \in \mathbb{R}^n$3, $\mathbf{w}^j_t \in \mathbb{R}^n$4, and $\mathbf{w}^j_t \in \mathbb{R}^n$5 to zero. The paper’s interpretation is that these mechanisms bound the net effect of all active features on each update, thereby limiting instability even when individual per-feature step-sizes become large (Javed et al., 22 Jul 2025).

3. Operant conditioning benchmark

The operant conditioning benchmark is designed so that a very small subset of input signals is relevant for decision making, while the majority of the signals are noise sampled from a non-stationary distribution (Javed et al., 22 Jul 2025). The observation vector is $\mathbf{w}^j_t \in \mathbb{R}^n$6, the action vector is $\mathbf{w}^j_t \in \mathbb{R}^n$7, and $\mathbf{w}^j_t \in \mathbb{R}^n$8. The first $\mathbf{w}^j_t \in \mathbb{R}^n$9 observation components are rare, task-relevant signals; the remaining $j$0 components are noisy distractors.

At certain special time steps, exactly one of the first $j$1 observation components is $j$2. If the agent chooses an action vector whose corresponding component is $j$3 and all other action components are $j$4, the agent receives a delayed reward. The delay is sampled each time from

$j$5

and special stimulus events occur every

$j$6

For the noisy components, each index $j$7 is Bernoulli with probability $j$8, where $j$9 performs a Gaussian random walk clipped to the interval $t$0. The benchmark therefore combines delayed reward, sparse relevance, and feature non-stationarity in a form that remains linearly representable.

The action space used in the reported experiments is based on $t$1 binary action components, which are mapped to four discrete actions: $t$2, $t$3, $t$4, and $t$5. Performance is summarized by lifetime average reward,

$t$6

4. Empirical behavior

The reported experiments vary the number of observation features $t$7, the initial step-size $t$8, the meta-step-size $t$9, and the presence or absence of step-size decay (Javed et al., 22 Jul 2025). The paper uses $t-1$0, $t-1$1, an agent lifetime $t-1$2, and a softmax policy with temperature $t-1$3. It states that the optimal lifetime reward is approximately $t-1$4.

Quantity	Setting
Relevant action components	$t-1$5
Feature counts examined	$t-1$6, $t-1$7
Effective-learning-rate bound	$t-1$8
Decay parameter	$t-1$9 or $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$0
Relevant observation components in reported plots	$$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$1

Across a wide range of $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$2 and $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$3, Swift-Sarsa achieves average rewards close to the optimal value $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$4. As $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$5 increases, performance improves, which the paper interprets as evidence that the step-size optimization is beneficial. Increasing $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$6 from $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$7 to $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$8 makes the problem harder and degrades performance somewhat, but the algorithm still performs well and robustly.

The ablation on step-size decay isolates one of the method’s defining components. When the initial step-size is too large, step-size decay significantly improves performance. With decay enabled, the algorithm remains robust even when starting from overly aggressive step-sizes, because the decay mechanism gradually reduces them to a stable regime. The reported empirical picture is therefore one of near-optimal lifetime average reward over a wide range of step-size hyper-parameters, together with robustness in the presence of tens of thousands of distractor features (Javed et al., 22 Jul 2025).

5. Relation to SARSA theory

The extended abstract introducing Swift-Sarsa does not provide formal convergence proofs (Javed et al., 22 Jul 2025). Its significance is better understood against the established theory of SARSA with linear function approximation. Finite-sample analysis for linear SARSA under a single non-i.i.d. trajectory gives a mean-square error rate of order $$v^j_{t-1,t} = \sum_{i=1}^n w^j_{t-1}[i]\,\phi_t[i].$$9 under assumptions including a Lipschitz continuous policy improvement operator and uniform ergodicity (Zou et al., 2019). A later convergence analysis shows that projected SARSA with linear function approximation can chatter in a bounded region rather than converge to a single limit point, and that the region is much smaller than the region that is projected into when the magnitude of the reward is not too large (Zhang et al., 2022).

Step-size sensitivity is a separate axis of difficulty. Implicit variants of Q-learning and SARSA reformulate their updates as fixed-point equations, yielding an adaptive effective step-size

$\mathbf{v}_{t-1,t}\in\mathbb{R}^m$0

and demonstrating substantially reduced sensitivity to step-size selection, with stable performance at step-sizes that would cause standard methods to fail (Kim et al., 26 Jan 2026). Off-policy multi-step descendants of the Sarsa family motivate yet another line of work: Gradient Expected Sarsa($\mathbf{v}_{t-1,t}\in\mathbb{R}^m$1) uses a convex-concave saddle-point framework and converges to the optimal solution at a linear convergence rate (Yang et al., 2020).

This suggests that Swift-Sarsa is best read as a particular engineering synthesis inside the linear, on-policy branch of the literature. It does not alter the basic Sarsa control target; instead, it combines true-online traces with per-feature step-size optimization, a hard cap on the effective learning rate, and decay-based recovery from overly aggressive updates.

6. Related uses and broader interpretations

Before the 2025 extended abstract fixed the name for a specific control algorithm, the expression “Swift-SARSA” also appeared informally as a label for SARSA variants designed to move aggressively while retaining stability. One 2026 paper states that “Swift-SARSA” is a good informal name for implicit SARSA, because the implicit fixed-point reformulation permits large nominal step-sizes while staying numerically stable and requiring much less tuning (Kim et al., 26 Jan 2026).

A 2024 radar-scheduling study provides a different kind of precursor. Its method, called ISQ, is an index policy based on Sarsa and Q-learning for a restless multi-armed bandit formulation of heterogeneous smart target tracking. The authors describe the combination of forward Sarsa and backward Q-learning as a concrete blueprint for what a “Swift-Sarsa” might look like in practice: Sarsa used in a way that accelerates convergence, improves sample efficiency, and scales to a large combinatorial control problem (Hao et al., 2024).

A network slicing study offers a domain-specific example of the same broader intuition. It uses standard tabular SARSA for communication-resource sharing among queues, with two states per queue, actions that change bandwidth by $\mathbf{v}_{t-1,t}\in\mathbb{R}^m$2, and event-driven invocation when a threshold is crossed. The paper characterizes this design as practically “swift” because state aggregation, coarse actions, and event-triggered learning reduce the number of state-action pairs and support quick adaptation in overload conditions (Xavier et al., 2023).

This broader usage suggests a dual meaning for the term. In the strict sense, Swift-Sarsa denotes the 2025 algorithm that extends True Online Sarsa($\mathbf{v}_{t-1,t}\in\mathbb{R}^m$3) with the SwiftTD machinery. In a looser research sense, it denotes a design orientation within SARSA research toward aggressive but controlled learning: per-feature adaptation, bounded effective updates, and mechanisms that preserve robustness in the presence of instability, non-stationarity, or large numbers of irrelevant features.