S0 Tuning: Zero-Overhead State Adaptation

• S0 tuning is a parameter-efficient method for hybrid recurrent-attention models that learns only the initial recurrent state to steer early decoding dynamics.
• The method injects a learned state at the start of the sequence, achieving zero inference overhead and significant performance gains on benchmarks like HumanEval.
• Empirical results highlight its data efficiency and robustness compared to alternatives like LoRA and prefix tuning, with minimal additional parameters.

Searching arXiv for the primary paper on “S0 Tuning” and closely related PEFT/hybrid-model references. S0 tuning is a parameter-efficient fine-tuning method for hybrid recurrent-attention LLMs in which adaptation is performed by learning only the initial recurrent state for each recurrent layer while freezing all model weights. In the formulation introduced by Young and colleagues, the method targets an adaptation surface available in architectures such as GatedDeltaNet and Mamba-2 hybrids but absent from pure Transformers: the recurrent hidden state. The learned state is injected only at the start of the sequence, after which the model follows its native recurrence, yielding what the authors describe as zero inference overhead. The method was introduced in “S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models” (Young, 1 Apr 2026).

1. Definition and scope

S0 tuning learns, for a hybrid model with $L$ recurrent layers, the set

$\{S_0^{(\ell)}\}_{\ell=1}^L$

and nothing else. Each $S_0^{(\ell)}$ has the same shape as the native recurrent state of layer $\ell$. Instead of starting from the default zero state, the model starts each recurrent layer from $\alpha S_0^{(\ell)}$, where $\alpha$ is an architecture-specific scaling hyperparameter (Young, 1 Apr 2026).

The optimization objective is completion-only cross-entropy over verified correct completions, with prompt tokens masked: $$\mathcal{L}(S_0) = \frac{1}{N}\sum_{i=1}^N \mathrm{CE}\!\left(y_i^{\mathrm{comp}}, f_\theta(x_i; \alpha S_0)\right) + \lambda \sum_{\ell=1}^L \|S_0^{(\ell)}\|_2^2 .$$ Operationally, the procedure is minimal: initialize each $S_0^{(\ell)} = 0$, inject it as the recurrent layer’s initial hidden state before the first token, optimize only $\{S_0^{(\ell)}\}$ using completion loss, and then use the learned state at inference only at $t=1$ (Young, 1 Apr 2026).

This design places S0 tuning within PEFT, but it differs from weight-space methods because it adapts state rather than parameters. A plausible implication is that the method is best understood not as a low-rank approximation to weight updates, but as a learned perturbation of the model’s dynamical trajectory.

2. Architectural setting in hybrid recurrent-attention models

The method is defined for hybrid recurrent-attention backbones rather than for pure Transformers. The paper studies two such backbones. Qwen3.5-4B interleaves 24 GatedDeltaNet layers with 8 attention layers. FalconH1-7B uses a Mamba-2 + attention hybrid, with recurrent and attention heads operating in parallel within each layer (Young, 1 Apr 2026).

For GatedDeltaNet, the recurrence is written as

$\{S_0^{(\ell)}\}_{\ell=1}^L$0

where $\{S_0^{(\ell)}\}_{\ell=1}^L$1 is a decay gate, $\{S_0^{(\ell)}\}_{\ell=1}^L$2 controls write strength, and $\{S_0^{(\ell)}\}_{\ell=1}^L$3 are key/value vectors. For Mamba-2, the state update is

$\{S_0^{(\ell)}\}_{\ell=1}^L$4

In both cases, the default initialization is $\{S_0^{(\ell)}\}_{\ell=1}^L$5; S0 tuning changes only that initial condition (Young, 1 Apr 2026).

The paper emphasizes that these models have matrix-valued recurrent states, and it links the success of S0 tuning to that expressiveness. It contrasts this setting with diagonal-state architectures such as Mamba-1, where prior work found LoRA to be better. This suggests a state-expressiveness threshold: when the recurrent state is sufficiently expressive, initial-state tuning becomes a competitive PEFT surface (Young, 1 Apr 2026).

3. Zero-overhead adaptation and relation to other PEFT methods

The principal systems claim is that S0 tuning has zero inference overhead because the learned state is injected only once, at the beginning of the sequence. After that, there is no extra branch at each decoding step, no per-token prefix storage, no runtime adapter, and no weight merging step before use (Young, 1 Apr 2026).

The paper contrasts S0 tuning with three alternative adaptation paradigms.

First, versus LoRA, S0 tuning adapts state rather than weights, has zero inference overhead, and exploits a recurrent-specific surface. The paper includes a parameter-matched control on Qwen3.5: LoRA at rank 64 has 12.6M parameters, exactly matching S0 on that model, but performs poorly relative to S0 (Young, 1 Apr 2026).

Second, versus prefix tuning, the paper uses a pure-Transformer negative control. On Qwen2.5-3B, prefix tuning degrades performance by $\{S_0^{(\ell)}\}_{\ell=1}^L$6 pp under all nine configurations tested (Young, 1 Apr 2026). The stated interpretation is that prefix-style tuning does not exploit the recurrent state mechanism that appears essential to S0 tuning.

Third, versus weight merging, S0 tuning requires neither merging nor model reload. The tuned artifact is just a state file. On Qwen3.5-4B, the tuned state corresponds to 12.6M parameters, about 0.3% of the model, and is about 48 MB; for FalconH1-7B, the tuned state is 34.6M parameters (Young, 1 Apr 2026). Task switching consists of swapping the initial state tensor(s), not modifying the backbone.

4. Training regime and data efficiency

The main HumanEval study is explicitly small-data. HumanEval has 164 problems; problems 0–79 are used for training data collection and problems 80–163 are held out for evaluation. Training uses execution-verified correct solutions sampled from the frozen base model, and the main pipeline keeps at most one passing completion per training problem, yielding about 48 verified solutions total (Young, 1 Apr 2026).

The reported training setup is deliberately lightweight: Adam, batch size 1, 20 steps for S0 tuning, learning rate $\{S_0^{(\ell)}\}_{\ell=1}^L$7, $\{S_0^{(\ell)}\}_{\ell=1}^L$8 regularization $\{S_0^{(\ell)}\}_{\ell=1}^L$9, bf16, and a single NVIDIA A10G 24 GB, with about 3 minutes of training time (Young, 1 Apr 2026).

The paper further reports that as few as 25 verified solutions suffice for stable gains, and even 10 solutions already produce large but noisier improvements (Young, 1 Apr 2026). This suggests that the optimization surface induced by recurrent-state initialization is unusually data-efficient under verified supervision. A plausible implication is that the method is especially suited to settings where curated, correctness-filtered exemplars are scarce but high precision is required.

5. Empirical results

The strongest evidence is on Qwen3.5-4B for HumanEval. The baseline is 48.8%, S0 tuning reaches $S_0^{(\ell)}$0, and the gain is $S_0^{(\ell)}$1 pp over 10 seeds. Compared to the strongest LoRA baseline, LoRA reaches $S_0^{(\ell)}$2, so S0 exceeds LoRA by $S_0^{(\ell)}$3 pp with significance $S_0^{(\ell)}$4 (Young, 1 Apr 2026).

On FalconH1-7B, the baseline is 40.5%, S0 reaches $S_0^{(\ell)}$5, and LoRA reaches $S_0^{(\ell)}$6. The paper characterizes these as statistically indistinguishable at the 3-seed sample size, while noting that S0 reaches parity without weight merging and with lower variance (Young, 1 Apr 2026).

Sampled decoding results on Qwen3.5 also favor S0: baseline pass@1 is 41.9%, pass@5 is 64.2%, and pass@10 is 69.4%; LoRA r24 gives 56.3%, 64.1%, and 66.7%; S0 gives 70.3%, 84.5%, and 88.5% (Young, 1 Apr 2026). The particularly strong gains at larger $S_0^{(\ell)}$7 indicate that S0 shifts the solution distribution rather than merely sharpening a small subset of trajectories.

The parameter-matched comparison is central. LoRA r48 with 9.4M parameters gives $S_0^{(\ell)}$8 pp, LoRA r64 with 12.6M parameters gives $S_0^{(\ell)}$9 pp, while S0 with the same 12.6M parameters gives $\ell$0 pp (Young, 1 Apr 2026). This directly supports the claim that the improvement is not attributable to parameter count alone.

The paper also reports scale dependence across Qwen3.5 sizes: $\ell$1 pp at 0.8B, $\ell$2 pp at 2B, $\ell$3 pp at 4B, and $\ell$4 pp at 9B (Young, 1 Apr 2026). This suggests that larger hybrid models expose more latent capability to this form of state-space adaptation.

6. Mechanistic interpretation: trajectory steering

The paper’s mechanistic account is that S0 tuning acts primarily through trajectory steering rather than through a persistent direct logit bias. The recurrent contribution of the initial state is written as

$\ell$5

This formulation shows that the influence of $\ell$6 is propagated through the prompt via the sequence of gating matrices (Young, 1 Apr 2026).

Empirically, the paper reports that by the end of the prompt, the direct KL influence of $\ell$7 on output logits falls to 0.03%. Yet among 27 FAIL-to-PASS flips, 23 diverge from baseline at the very first generated character; this is 85% first-character divergence, all 27 diverge within the first 10% of the completion, and a sign test gives $\ell$8 (Young, 1 Apr 2026).

The interpretation is that the learned initial state does not need to maintain a large explicit effect throughout generation. Instead, it nudges the hidden dynamics into an alternative early decoding branch, and autoregressive rollout amplifies that change. This suggests that S0 tuning is especially effective in tasks where early-token branching strongly conditions the downstream completion.

The paper reinforces this claim with a stronger but more expensive variant: a per-step state-offset method that adds a learned tensor $\ell$9 at every timestep. It achieves $\alpha S_0^{(\ell)}$0 pp on Qwen3.5, above both S0 and LoRA, but incurs per-step inference cost. In a dedicated rerun, S0 captures 87% of the offset variant’s gain at zero runtime cost (Young, 1 Apr 2026). This establishes S0 as a specific operating point on the performance–latency frontier.

7. Transfer, limits, and broader significance

Cross-domain transfer is selective. The paper reports gains on MATH-500 of $\alpha S_0^{(\ell)}$1 pp with $\alpha S_0^{(\ell)}$2 over 8 seeds and on GSM8K of $\alpha S_0^{(\ell)}$3 pp with $\alpha S_0^{(\ell)}$4 over 10 seeds (Young, 1 Apr 2026). By contrast, on Spider text-to-SQL there is no transfer: $\alpha S_0^{(\ell)}$5 pp across five alpha values in a single-seed sweep (Young, 1 Apr 2026).

The paper treats this failure case as mechanistically informative rather than anomalous. Its explanation is that Spider has low early-token diversity and limited trajectory branching, so changing the initial recurrent state leaves little to steer (Young, 1 Apr 2026). This suggests a task criterion for applicability: S0 tuning should be expected to work best when tasks exhibit early branching and the frozen model already contains latent competence that can be unlocked by altered recurrent initialization.

Architecture-specific scaling of the injection factor also matters. The best $\alpha S_0^{(\ell)}$6 is 0.07 for Qwen3.5 / GatedDeltaNet and 0.65 for FalconH1 / Mamba-2, nearly a 10× difference (Young, 1 Apr 2026). This indicates that raw initial-state magnitude is not portable across recurrence families, presumably because the gating dynamics differ substantially.

In sum, S0 tuning defines a recurrent-state analogue of PEFT in which adaptation is moved from weight space into state space. Its principal significance lies in showing that, for matrix-state hybrid LLMs, recurrent initialization itself is a strong zero-inference-overhead adaptation surface under scarce verified supervision (Young, 1 Apr 2026). This suggests a broader research direction in which model personalization, task specialization, and multi-task deployment for hybrid architectures may be organized around swappable state files rather than merged adapters or edited weights.