Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

Abstract: Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures.

• The paper demonstrates that activation-free polynomial substitutions in MetaFormer-style architectures achieve competitive performance in classification, segmentation, and robustness tasks.
• It introduces PolyMLP, PolyConv, and PolyAttn modules that use Hadamard products and second-degree polynomial interactions, stabilized by Sigmoid-Scale and multi-input skip connections.
• Empirical evaluations on ImageNet-1K and out-of-distribution benchmarks highlight efficient performance and suggest potential for FHE-friendly and privacy-preserving inference.

Activation-Free Backbones for Vision: Polynomial Substitutions within MetaFormer Models

Motivation and Scope

This paper investigates the necessity of pointwise activations (e.g., ReLU, GELU) and exponential softmax as sources of nonlinearity in modern vision backbones. The authors demonstrate that polynomial interactions, realized through activation-free Hadamard products, can replace all standard nonlinearities within MetaFormer-style architectures while retaining competitive performance across classification, segmentation, and robustness metrics. The study systematically designs and analyzes polynomial modules for channel mixing (PolyMLP), convolutional spatial mixing (PolyConv), and attention-based spatial mixing (PolyAttn), enabling controlled drop-in substitution for standard vision components.

Polynomial Module Design

Three core primitives are newly formulated without any activation function:

• PolyMLP: Standard MLPs apply a pointwise activation between two linear projections. PolyMLP removes the activation: it computes the Hadamard product of two learned linear projections followed by an output projection. This yields second-degree polynomial terms in the input, with the expressive power growing exponentially with depth.
• PolyConv: Instead of standard separable convolution with pointwise activation, PolyConv fuses two depthwise convolution branches (coarse-dilated and fine-channel-flipped) through elementwise multiplication, then consolidates and projects. Heterogeneous receptive fields maximize feature diversity before fusion, critical for the efficacy of Hadamard interactions.
• PolyAttn: Replacing exponential softmax, PolyAttn employs a polynomial kernel for attention weights: $(s \cdot QK^\top + 1)^p$, where $s$ is a learnable per-head scale (sigmoid-bounded) and $p$ is polynomial degree (set to 4). Depthwise convolutions inject local context, and an $\ell_1$ normalization replaces softmax, maintaining compatibility with efficient attention schemes.

Figure 1: Visualization of PolyMLP, PolyConv, and PolyAttn, detailing activation-free mixing via Hadamard product and polynomial kernel.

Stabilization and Optimization

Polynomial networks face unique optimization challenges due to multiplicative amplification. The paper introduces key stabilization mechanisms:

• Sigmoid-Scale: Residual updates in each stack are bounded by a sigmoid parameter, ensuring residual magnitudes shrink with depth and preventing numerical instabilities.
• Multi-input skip connections: Each cell receives inputs from two preceding cells (inspired by NASNet), improving gradient flow and trainability in ultra-deep architectures.
• Depth-over-width paradigm: Deeper, narrower networks are shown to be superior in capacity and performance for polynomial backbones; wider, shallow designs (as in prior polynomial models) limit attainable polynomial degree.

Figure 2: PolyNeXt cell with multi-input skip connections and Sigmoid-Scale residual stabilization enabling training of networks up to 200 layers.

PolyNeXt Architecture and Empirical Evaluation

The polynomial modules are instantiated within MetaFormer, yielding two main strains:

• CPolyNeXt: Employs PolyConv throughout, constituting a pure polynomial convolutional backbone.
• APolyNeXt: Combines PolyConv (high-resolution stages) and PolyAttn (low-resolution stages), mirroring the hybrid structure of CAFormer.

Both models follow a four-stage hierarchy with progressive spatial reduction and channel increase.

Figure 3: PolyNeXt backbone design: four-stage hierarchical structure, with polynomial mixers and depth-over-width cell configurations.

Comprehensive evaluation is conducted on ImageNet-1K, ADE20K segmentation, robustness benchmarks (ImageNet-C, -A, -R, -Sketch), and smaller datasets. The main findings:

• PolyNeXt variants consistently match or exceed equivalent MetaFormer instantiations (ConvFormer, CAFormer) on clean accuracy, often at reduced parameter and FLOP count.
• Strong robustness: CPolyNeXt and APolyNeXt outperform baselines on out-of-distribution benchmarks, demonstrating improved efficiency-robustness tradeoffs.
• Fully polynomial variants (with polynomial-compatible BatchNorm) attain $82.7\%$ on ImageNet, surpassing both prior polynomial models and activation-based ConvNeXt-T at fewer parameters.

Figure 4: ImageNet-1K accuracy scaling: CPolyNeXt and APolyNeXt consistently outperform baseline MetaFormer and prior polynomial networks across model scales.

Ablation Analyses

Extensive ablations isolate critical factors:

• Activations are counterproductive: Reintroducing activations consistently degrades accuracy ($-$0.4 to $-$1.0 points for GELU insertion), and replacing Hadamard product with addition causes catastrophic failure ($-$22.3 points).
• Stabilization is essential: Sigmoid-Scale initialization geometry, not the nonlinearity, is vital; removing stabilization or skip connections incurs significant performance drops.
• Depth is paramount: Deeper configurations outperform wider ones at matched parameter counts (up to $+$1.5 points), highlighting the exponential expressivity benefit of stacked multiplications.

Figure 5: Layer-by-layer comparison: PolyMLP, PolyConv, and PolyAttn drop activations in favor of multiplicative fusion, outperforming their activation-based counterparts.

Practical and Theoretical Implications

This work provides evidence that polynomials, when correctly stabilized, are sufficient as sources of nonlinearity in state-of-the-art vision backbones. The compatibility of polynomial modules with modular backbone design (MetaFormer) enables broad transferability. Full polynomial construction unlocks avenues for FHE-friendly inference, facilitating privacy-preserving computation in settings where activations or normalization (e.g., LayerNorm) are prohibitive.

The findings also challenge the necessity of commonly used activation functions and exponential attention, suggesting their role is primarily optimization facilitation rather than representational capacity.

Future Directions

Notable considerations for ongoing and future research include:

• Characterization of optimization dynamics in deep polynomial networks and the interplay between stabilization and expressivity.
• Extension to language modeling domains, where attention mechanisms are central but exponential softmax could be replaced with polynomial kernels.
• Hardware specialization and homomorphic encryption deployment, leveraging the purely additive and multiplicative nature of inference for secure and efficient computation.

Conclusion

The paper rigorously demonstrates that activation-free polynomial substitutes for MLP, convolution, and attention modules within MetaFormer are sufficient for competitive vision recognition and robust transfer tasks. The depth-over-width paradigm, in tandem with lightweight stabilization, addresses training and optimization hurdles, enabling networks exceeding 200 layers. Polynomial backbones not only match but often exceed activation-based methods in accuracy, robustness, and efficiency. The research reframes understanding of nonlinearity requirements in deep vision models and opens a pathway for FHE-compatible architectures.

(2605.20839)