- The paper demonstrates that RMS-normalized hybrid decoders significantly preserve intermediate semantic retention during code curriculum specialization.
- It employs an alternating design of Differential-Attention and DeltaNet-inspired layers with calibrated scaling to mitigate catastrophic forgetting.
- Empirical results show enhanced adaptability on specialized Python tasks compared to parameter-matched Transformers, despite challenges in long-horizon syntactic retention.
SamatNext v0.2-B: RMS-Normalized Hybrid Decoders for Curriculum Retention in Small-Sized Code Models
Overview
SamatNext v0.2-B introduces a 356M-parameter hybrid causal sequence decoder architecture aimed at improving curriculum retention in small code models. The model alternates Differential-Attention-style layers with DeltaNet-inspired linear-state mixers, both regularized using RMS normalization and output scale calibration. The study empirically evaluates retention/plasticity tradeoffs under a staged Python curriculum compared to a parameter-matched Transformer baseline. The results illustrate that hybrid architectures can almost completely preserve intermediate semantics during specialization, though catastrophic forgetting persists for long-horizon syntactic skills.
Architectural Details
SamatNext v0.2-B is structured as a stack of 16 decoder layers, each alternating between:
- Differential-Attention-style attention layers (odd-indexed): These apply dual attention pathways with learnable mixture weights, incorporating RoPE for position encodings. Unlike the original Differential Transformer, SamatNext employs an additive combination of base and differential paths.
- DeltaNet-inspired simplified linear-state mixer layers (even-indexed): These utilize element-wise recurrent accumulations of context features using causal cumulative sums, with output RMS normalization and explicit scale calibration (s=0.25).
- SwiGLU MLP blocks follow each mixer with the standard two-layer design, where activations are gated and up-projected, then down-projected back to the model dimensionality.
- RMS normalization (learnable scaling) is applied throughout the architecture for improved scale management and stability.
- Verifier head: An auxiliary linear head over the final sequence state, enabling additional evaluation mechanisms.
Both SamatNext and the Transformer baseline are precisely parameter-matched (356M), using context length 8192 and untied LM head and embeddings. For fairness, the Transformer’s feedforward size is adjusted to account for the extra parameters required by SamatNext's mixers.
Experimental Setup
The staged curriculum consists of:
- Stage 2A (Syntax Foundation): Focused primarily on basic Python syntax.
- Stage 2E (Adversarial Syntax Holdout): Early-stage adversarial prompts to probe long-horizon retention.
- Stage 3 (Instruction Semantics): Higher-level instruction-following and semantic Python tasks.
- Stage 5 (Specialized Coding): Final teacher-generated tasks testing domain coding proficiency.
All code is evaluated using deterministic, execution-based pass@1 metrics. Training employs AdamW, context length 512 during curriculum learning, batch size 1 with gradient accumulation to 16, and a cosine-decay LR schedule.
Models are compared by sequentially training through the curriculum and measuring loss in earlier-stage capabilities after specialization; direct-from-scratch specialization establishes an upper bound.
Empirical Findings
Retention/Plasticity Tradeoff
SamatNext v0.2-B demonstrates a pronounced retention advantage under curriculum learning. After full specialization (Stage 5):
- SamatNext v0.2-B: 100.0% Stage 5 pass, 98.8% Stage 3 retention, 12.0% Stage 2E.
- Transformer baseline, LR=3×10−6: 49.4% Stage 5, 3.8% Stage 3, 0.0% Stage 2E.
- Transformer, LR=1×10−5: 97.6% Stage 5, 6.0% Stage 3, 3.0% Stage 2E.
SamatNext retains nearly all adjacent semantic knowledge while fully adapting to new distributions. The Transformer must be rescued with a higher learning rate to even approach full adaptability but suffers massive intermediate semantic forgetting.
Syntax and Long-Horizon Forgetting
Both models show poor retention on the earliest adversarial syntax tests (Stage 2E) post-specialization. SamatNext improves Stage 2E retention to 12.0%, while the baseline scores ≤3.0%. However, the absolute rates indicate that neither hybridization nor RMS normalization alone address catastrophic forgetting over long-horizon sequential curricula.
Out-of-Distribution and Generalization
On a controlled HumanEval subset (code tasks out-of-distribution from curriculum), SamatNext achieves a pass@1 of 12.0% versus 8.0% for the Transformer. This signals limited—but present—generalization at small scale and constrained curriculum depth.
Theoretical and Practical Implications
The results suggest that RMS-normalized sequence mixers with explicit scale calibration and alternating hybridization impose an architectural bias favoring retention in regimes requiring sequential adaptation. The empirical success in preserving intermediate semantics indicates that hybridization constrains parameter drift, possibly via the recurrent, compressed state representation in the linear mixer layers. However, the enduring susceptibility to long-horizon forgetting demonstrates that structural changes alone are insufficient; active continual learning strategies are still required for robust, lifelong code curriculum retention.
Furthermore, model scaling, the breadth of curriculum domains, and ablation between hybridization pattern and mixer type are likely to modulate the observed tradeoffs. For practical code model deployment in continual learning scenarios, hybrid architectures may improve adaptability without immediately incurring the high overhead of explicit continual learning algorithms, but further validation is required.
Limitations
- The curriculum is synthetic and limited to Python; findings may not translate to unrestricted code or natural language tasks.
- No architectural ablations isolate the effect of mixer alternation or remove Differential-Attention entirely.
- No explicit continual learning or rehearsal baselines are compared.
- Evaluations are at small scale (356M) and confined context length during curriculum training.
- Stage 5's task distribution is teacher-generated via templates, and peak Stage 5 performance may not reflect real-world generalization.
Future Directions
Further research should systematically ablate mixer patterns, investigate scaling effects, and directly compare hybrid sequence mixers to continual learning methods such as replay buffers, EWC, and lightweight adapters. Broader curriculum domains and longer contexts are essential for ecological validity. Quantifying the effect of verifier heads and alternate positional encoding strategies also remains open. Additional SamatNext runs at varied learning rates are necessary to decouple the effects of architecture versus hyperparameters.
Conclusion
SamatNext v0.2-B offers evidence that RMS-normalized hybrid sequence mixers substantially improve intermediate semantic retention over standard Transformers in curriculum-based code learning, enabling near-complete preservation during specialization. Nevertheless, the architecture does not remedy long-horizon catastrophic forgetting, and specialized continual learning remains essential for robust code modeling across staged tasks. Future ablations and domain expansion are required to generalize these findings beyond the present synthetic benchmarks.
This study provides reproducible resources and source code for independent validation of the reported phenomena (2606.22248).