Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V

Abstract: We propose two complementary modifications to transformer attention blocks. First, a non-linear pre-projection MLP is inserted between layer norm and Q/K/V projections, constructing richer features in a position-agnostic manner before any positional encoding is applied. Second, a content skip connection routes the pre-projection's features around the attention mechanism, allowing content information to bypass position-aware attention where beneficial. In frozen-probe experiments on Pythia-160M and 410M, the combined approach achieves the strongest results across methods: +40.6% LAMBADA accuracy and -39% perplexity at 160M scale. Learned skip connection weights reveal a consistent pattern across model sizes: later transformer layers activate the content bypass more strongly than earlier layers, suggesting that deeper layers benefit from content information that does not pass through positional attention. All modifications add no K/V cache overhead.

• The paper introduces a nonlinear pre-projection MLP and content skip that bypasses positional encoding, addressing linearity bottlenecks in Q/K/V projections.
• The methodology integrates a residual MLP after normalization and a near-zero initialized skip connection, yielding significant improvements in LAMBADA accuracy and perplexity on Pythia models.
• The approach is parameter-efficient and modular, enabling flexible application in multimodal contexts by routing content features independently of position.

Position-Agnostic Pre-Projection and Content Skip in Transformer Attention

Architectural Contributions

The paper "Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V" (2604.10791) presents two independent but complementary modifications to the canonical transformer attention block. The first is a nonlinear pre-projection MLP, inserted between layer normalization and Q/K/V projections, which constructs richer features in a position-agnostic manner. The second is a content skip connection, a learned linear projection that routes these pre-projection features directly to the layer output, thus bypassing the position-aware attention mechanism.

The pre-projection addresses the linearity bottleneck inherent in standard Q/K/V projections, enabling single-layer nonlinear feature construction. The content skip mitigates the constraint of mandatory positional filtering by allowing content-based features (semantic type, syntactic category, entity identity) to circumvent positional attention. Both modifications are parameter-efficient, incur no K/V cache overhead, and are compatible with multimodal settings where positional encoding may be modality-specific.

Methodological Overview

The nonlinear pre-projection is realized as a residual MLP (with SiLU activation and expansion factor $e=1.25$) directly after RMS normalization and prior to Q/K/V linear projections. This enriches the features used for attention with higher-order interactions, independent of token position. Simultaneously, the same enriched features are projected via a near-zero initialized linear skip connection, $W_{\text{skip}}$, and added post-attention to the layer's output.

All injected parameters are trained in a frozen-probe regime; base transformer weights are kept fixed. The study benchmarks the Pythia-160M and 410M models on WikiText-103, using LAMBADA, HellaSwag, ARC-Easy, and perplexity metrics. LoRA baselines are included for comparison, with rank chosen to match parameter counts.

Empirical Results

Quantitative results demonstrate that the combined pre-projection and content skip method yields the strongest overall performance:

• Pythia-160M: LAMBADA accuracy achieves 0.180, a +40.6% improvement over baseline; HellaSwag accuracy improves +3.7%; perplexity is reduced by 38.9% to 47.9.
• Pythia-410M: Perplexity is reduced to 17.0, outperforming standalone LoRA (17.7) despite fewer parameters; LAMBADA accuracy of 0.484 closely matches LoRA's 0.488.
• The content skip is the only method that consistently improves both comprehension (LAMBADA) and prediction (perplexity) at 160M scale.

Layer-wise analysis reveals a robust pattern: the magnitude of the skip projection is minimal in early layers and grows significantly in later layers (2.3$\times$ for 160M, 1.5$\times$ for 410M), indicating that deep transformer layers preferentially utilize content information that is position-agnostic.

Theoretical and Practical Implications

The separation of nonlinear feature construction from positional encoding in the attention block enables increased flexibility and modularity. This architectural paradigm is particularly salient for multimodal transformer designs; modal-independent pre-projection and skip paths facilitate shared content processing, while modal-dependent positional encoding can be applied as required (e.g., RoPE for text, spatial encoding for images). As demonstrated, position-agnostic content pathways are valuable in modalities with undefined ordering, such as point clouds.

The content skip's preferential activation in late layers aligns with prior observations about transformer layer specialization: earlier layers are focused on syntax and positional routing, whereas later layers consolidate semantic information antagonistic to positional dependence. The skip connection thus provides an adaptive mechanism for content routing strategies as learned by the model.

Relation to Prior Work

While adapters [houlsby2019parameter], LoRA [hu2021lora], and prefix tuning [li2021prefix] extend transformer adaptation via linear or bottleneck modules, this paper distinguishes itself by positioning nonlinear feature construction specifically before Q/K/V, outside the scope of positional encoding, and by demonstrating empirically the utility of learned content bypass—especially in deeper transformer layers. The approach avoids cache overhead seen with some prefix-based methods and is orthogonal to attention logit mixing (e.g., talking-heads [shazeer2020talking]).

Limitations and Future Directions

Current results are limited to frozen probe adaptation on medium-scale models (Pythia-160M/410M) and small sample sizes. Scaling studies for billion-parameter models and full-model training are pending. Benchmark differences for HellaSwag and ARC-Easy may be confounded by evaluation variance. The architectural insights regarding layer-specific skip activation require broader validation on additional model families and larger scales.

Future research may investigate unified pre-projection and skip mechanisms across modalities, direct integration within full training protocols, and the potential for further architectural simplification or parameter sharing. Systematic analyses of skip activation frequency and its correlation with layer role or representation complexity in diverse transformer architectures (including vision, audio, and point cloud modalities) are warranted.

Conclusion

The proposed position-agnostic pre-projection and content skip design enhances transformer attention's expressivity and flexibility, achieving strong gains in natural language comprehension and prediction benchmarks at fixed parameter budgets. The learned skip weights uncover a principled architectural insight: deep transformer layers benefit from direct content routing independent of positional attention. This modular, cache-efficient approach is theoretically extensible to multimodal contexts and warrants systematic exploration for large-scale, unified transformer architectures.