Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

Abstract: Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identify the primary bottleneck in the attention mechanism's linear projections, which strictly confine feature extraction to fixed-dimensional subspaces, limiting both expressivity and incremental capacity. To address this, we introduce Nexusformer, which replaces linear $Q/K/V$ projections with a Nexus-Rank layer, a three-stage nonlinear mapping driven by dual activations in progressively higher dimensional spaces. This design overcomes the linearity constraint and enables lossless structured growth: new capacity can be injected along two axes via zero-initialized blocks that preserve pretrained knowledge. Experiments on language modeling and reasoning benchmarks demonstrate that Nexusformer matches Tokenformer's perplexity using up to 41.5\% less training compute during progressive scaling (240M to 440M). Furthermore, our analysis of growth dynamics reveals that zero initialization induces a stable convergence trajectory, allowing us to derive a geometric scaling law that accurately predicts performance across expansion scales.

• The paper presents a novel dual-axis nonlinear Nexus-Rank layer replacing standard linear attention projections.
• It achieves efficient scaling with reduced compute cost, matching or surpassing baseline performance on perplexity and downstream tasks.
• A new geometric framework and empirical scaling law are proposed to analyze model growth and predict performance during progressive expansion.

Nonlinear Attention Expansion with Nexusformer for Inheritable Transformer Scaling

Introduction and Motivation

Transformer scaling has traditionally required training larger models from scratch due to architectural rigidity, particularly in the attention mechanism's linear $Q/K/V$ projections. This leads to computational inefficiency and prevents the reuse of pretrained representations. Nexusformer introduces a nonlinear Nexus-Rank layer in place of standard linear projections, enabling progressive, inheritable model scaling via dual-axis nonlinear expansions. By enabling zero-initialized growth along both intermediate ($M$) and expansion ($A$) dimensions, Nexusformer addresses the limitations of rigid capacity and disruptive expansion in classical Transformers.

Figure 1: Nexusformer architecture: (a) Nonlinear Nexus-Rank mapping replaces linear projections; (b) three-stage expansion with dual nonlinear activations; (c) dual-axis growth over $(M, A)$ with zero-initialized new blocks for non-destructive scaling.

Nexusformer Architecture

Nexusformer restructures the attention projection pipeline into a three-stage nonlinear transformation: $D\rightarrow M\rightarrow A\rightarrow D$. Instead of linear projections directly outputting $Q, K, V$ representations, Nexusformer performs progressive expansion into higher-dimensional, nonlinear feature spaces:

• Stage 1: Linear transformation to an intermediate space ($D\to M$) with nonlinearity (GeLU), enabling primitive feature separation.
• Stage 2: Further expansion to an over-capacity manifold ($M \to A$) with an additional nonlinearity, capturing higher-order interactions.
• Stage 3: Projection back to the original space ($A \to D$), maintaining compatibility with standard Transformer outputs.

The Nexusformer supports non-disruptive scaling by zero-initializing new parameter blocks along both $M$ and $M$0 axes, preserving output invariance at the moment of expansion and facilitating seamless continued training. This dual-axis growth—contrasting with linear, single-axis scaling—enables the architecture to inherit and augment pretrained knowledge robustly.

Experimental Results and Efficiency

Experiments on language modeling and commonsense reasoning tasks across parameter regimes (170M–640M) establish that Nexusformer achieves competitive or superior accuracy relative to both the Tokenformer and Qwen3 baselines. Notably, in progressive expansion from a 240M model, Nexusformer achieves equivalent perplexity to Tokenformer with 41.5% less training compute, affirming both parameter and computational efficiency.

Figure 2: Model training efficiency and perplexity comparison, demonstrating Nexusformer’s lower compute requirements for equivalent performance.

Another strong result is Nexusformer’s ability to surpass the Qwen3-440M baseline with only half the parameter count (240M), and to maintain superior downstream accuracy and perplexity during incremental growth—particularly peaking at 380M, where marginal gains are best observed before diminishing returns.

Nexusformer’s three-stage architecture incurs only marginally higher FLOPs per token, often outperforming or matching standard Transformer variants on computational cost across scales, due to efficient utilization of smaller base hidden dimensions and redistribution of computation.

Figure 3: Per-token FLOPs comparison between Nexusformer and Qwen series across parameter scales.

Growth Dynamics and Theoretical Insights

The evolution of Nexusformer during expansion and continued training is analyzed using nonparametric overlap metrics and a novel geometric indicator: the radial energy $M$1, constructed from parameter utilization and overlap coefficients. The model’s growth trajectory in this space is characterized by a centrifugal–centripetal pattern: initial divergence upon parameter addition (centrifugal), followed by a structured convergence (centripetal) back toward an aligned, high-performance state.

Zero initialization of new blocks is critical for maintaining this geometric structure. Ablation with nonzero Gaussian perturbations disrupts this periodic trajectory, degrading both stability and convergence.

Figure 4: Evolution of Grassmann and Euclidean distances as a function of continued-training tokens under diverse expansion paths.

Figure 5: Radial energy trajectory $M$2 for Nexusformer on the 240M→380M zero-initialized expansion, showing structured, quasi-periodic evolution (R² and p-value reported).

A scaling law is empirically derived relating the log-perplexity to the log-energy radius: $M$3. This fit—with $M$4 across scales—provides a performance predictor that is decoupled from architectural scale. As $M$5 increases (suboptimal alignment), perplexity systematically worsens; as $M$6 contracts (centric movement), model quality improves.

Figure 6: Log-log relationship between energy radius and model perplexity under the empirical scaling law.

Computational Cost Analysis

Despite the three-stage projection, Nexusformer maintains competitive FLOPs profiles. At higher parameter scales, FLOPs per token are consistently below or equivalent to standard models, driven by compressed core hidden sizes and distributed projection. This effect persists even as expressivity and alignment gains are realized through dual nonlinearity and progressive expansion.

Implications and Future Directions

Theoretical observations underscore Nexusformer's ability to break the “linear bottleneck” of standard attention projections. Dual-nonlinear expansion allows deeper compositional modeling and better semantic disentanglement, as formalized by depth separation theorems. Unlike linear-only adaptation (e.g., LoRA or single-axis expansion), Nexusformer’s function class is strictly larger and supports inheritable, non-destructive model growth.

Practically, this architectural paradigm opens several avenues:

• Efficient scaling: Large models can extend capacity without destructive retraining, with implications for foundation models maintained over time or across tasks.
• Parameter-efficient adaptation: New capacity can be incorporated via nonlinear axes ($M$7, $M$8) rather than linear increments, allowing more flexible and robust transfer learning.
• Interpretability: The dual-nonlinear transformation offers new handles for probing feature dynamics across expansion and adaptation.

The current limitation is the maximum tested scale (640M parameters); larger scale and ultra-long context settings, as well as automated strategies for choosing $M$9 and $A$0, are promising directions for further research.

Conclusion

Nexusformer introduces a robust, nonlinear approach for attention expansion in Transformers, leveraging dual-axis, dual-nonlinearity projections. It achieves stable and inheritable scaling, strong efficiency gains, and superior downstream performance over classical and recent strong baselines. Its geometric framework for analyzing expansion dynamics yields deeper theoretical insights and supports empirical scaling laws, positioning Nexusformer as a practical architecture for next-generation, continually evolvable foundation models.

---

Reference:

"Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling" (2604.19147)