Quasi-Equivariant Metanetworks

Abstract: Metanetworks are neural architectures designed to operate directly on pretrained weights to perform downstream tasks. However, the parameter space serves only as a proxy for the underlying function class, and the parameter-function mapping is inherently non-injective: distinct parameter configurations may yield identical input-output behaviors. As a result, metanetworks that rely solely on raw parameters risk overlooking the intrinsic symmetries of the architecture. Reasoning about functional identity is therefore essential for effective metanetwork design, motivating the development of equivariant metanetworks, which incorporate equivariance principles to respect architectural symmetries. Existing approaches, however, typically enforce strict equivariance, which imposes rigid constraints and often leads to sparse and less expressive models. To address this limitation, we introduce the novel concept of quasi-equivariance, which allows metanetworks to move beyond the rigidity of strict equivariance while still preserving functional identity. We lay down a principled basis for this framework and demonstrate its broad applicability across diverse neural architectures, including feedforward, convolutional, and transformer networks. Through empirical evaluation, we show that quasi-equivariant metanetworks achieve good trade-offs between symmetry preservation and representational expressivity. These findings advance the theoretical understanding of weight-space learning and provide a principled foundation for the design of more expressive and functionally robust metanetworks.

• The paper proposes quasi-equivariant metanetworks that relax strict symmetry, preserving functional equivalence while enhancing model expressivity.
• The methodology employs a learnable quasi-action on continuous symmetries, parameterized via dedicated scale networks applied to weight statistics.
• Empirical evaluations demonstrate consistent performance improvements over strict equivariant methods with a marginal (sub-6%) parameter increase.

Quasi-Equivariant Metanetworks: A Theoretical and Empirical Analysis

Introduction

The paper "Quasi-Equivariant Metanetworks" (2604.23720) systematically addresses a core limitation in weight-space metanetworks: the mismatch between strict symmetry-based equivariant architectures and the broader class of functionally equivalent neural networks. The authors present a general framework for quasi-equivariance, which extends beyond strict group-equivariant mappings in weight space while properly respecting the underlying functional equivalence classes. This work delivers both a rigorous theoretical formalization and empirical evidence, demonstrating improved trade-offs between symmetry preservation and expressivity, particularly in high-dimensional, overparameterized regimes.

Functional Equivalence and Symmetry in Neural Network Weight Spaces

The non-injective relationship between parameter space and function space in neural networks leads to the phenomenon where distinct parameter sets implement the same function. This functional equivalence is characterized by symmetry groups acting on parameter spaces. Existing literature—spanning both MLPs and CNNs—has established that maximal symmetry groups capture almost all equivalence classes up to negligible algebraic varieties, such as permutations or scalings.

However, conventional equivariant metanetworks enforce strict equivariance at the parameter level—requiring output transforms to match input group actions exactly. While this guarantees invariance of functional content, it induces sparsity and reduces model capacity, as the permissible transformations are too restrictive for practical, expressive metanetwork design.

Relaxing Equivariance: The Quasi-Equivariant Paradigm

To balance symmetry constraints with expressive power, this paper generalizes strict equivariance to quasi-equivariance. A map $F$ is G-quasi-equivariant if, for any $g$ in the symmetry group $G$ and any parameter $\theta$, there exists another (possibly input-dependent) group element $g'$ such that $F(g\theta) = g'F(\theta)$. The essential innovation is that $g'$ can depend on both $g$ and $\theta$, unlike strict equivariance where $g' = g$, allowing a much richer class of mappings that preserve functional equivalence without sacrificing model flexibility.

This quasi-equivariant property is necessary and sufficient for functionally consistent mappings under the action of the maximal symmetry group, while strict equivariance is merely sufficient and often too strong. The paper presents a cohomological analysis of the conditions under which quasi-equivariant maps are well-defined, connecting the framework’s generality to established results in group theory and geometric deep learning.

Framework Construction and Implementation

The practical instantiation of quasi-equivariant metanetworks leverages group actions decomposed into continuous and discrete components, focusing the learnable quasi-action on the continuous symmetries (primarily scaling for MLPs/CNNs and invertible linear transforms in Transformers). The layerwise quasi-action is parameterized through a small, dedicated neural network (a scale network or MLP), which consumes statistical summaries (mean, variance, quantiles) of the weights and biases. The resultant group element—such as a scaling vector (for monomial matrix groups) or an invertible matrix (for GL groups in attention)—acts on the parameters, relaxing but not eliminating the underlying functional symmetry.

For MLPs/CNNs, the group comprises permutations and positive diagonal scalings, but only the latter is targeted by quasi-equivariant learning due to the discreteness of the permutation group. In transformers, the GL symmetries are exploited, enabling invertible transformations per attention head.

Careful design including structured, identity-centered perturbations ensures that the learned group actions oscillate near the identity (controlled via a small parameter), which mitigates optimization instability and preserves the dominant symmetry structure while permitting functionally meaningful deformations.

Empirical Evaluation

The proposed quasi-equivariant layers are integrated into state-of-the-art metanetworks (Monomial-NFN, Transformer-NFN) and evaluated on canonical benchmarks:

• CNN generalization prediction (Small CNN Zoo): Quasi-equivariant extension yields consistent gains over strict equivariant and statistical feature-based baselines, especially under strong group-action augmentations or when only limited parameter increases are permissible.
• Classification of implicit neural representations (Image INRs): Monomial-NFN Quasi surpasses both tuned strict equivariant and permutation-equivariant metanetworks across MNIST, FashionMNIST, and CIFAR-10, with improvements most pronounced in high-parameter, overfitting-prone regimes.
• Transformer generalization prediction (MNIST-Transformer, AGNews-Transformer): On both datasets and across multiple performance thresholds, Transformer-NFN Quasi demonstrates robust improvements in Kendall's $g$0 correlation, outperforming even heavily upscaled baselines with minimal parameter overhead. The relative gains increase under weight-space symmetry augmentations, confirming the method's functional robustness.

All improvements are achieved with a marginal (sub-6%) increase in parameter count, demonstrating the efficiency of the quasi-equivariant construction over simply scaling model width or stacking additional layers.

Theoretical and Practical Implications

Theoretical Impact

This work advances the understanding of weight-space learning by establishing that:

• Strict equivariance is unnecessary; quasi-equivariance suffices and is sharp for preservation of functional equivalence, reconciling symmetry-aware learning with practical expressivity constraints.
• The quasi-equivariant construction formalizes and subsumes various relaxed equivariance notions (including approximate and soft-constrained symmetry enforcement), and admits a classification in terms of 1-cocycles modulo coboundaries (gauge transformations).

Practical Impact

• Increased expressivity and predictive performance in metanetworks operating over pretrained weights, crucial for tasks such as model evaluation, editing, and hypernetwork-based optimization.
• Enhanced robustness to weight-space symmetries and augmentations commonly observed in overparameterized, highly reparameterized neural models.
• Immediate applicability to architectures dominated by continuous symmetries, especially monomial groups and general linear groups (MLPs, CNNs, transformers).

Future Directions

The authors identify several promising avenues:

• Extension to architectures with composite, less-structured symmetries (e.g., graph neural network metanetworks), where the variety and complexity of functional equivalence classes is not yet fully tractable.
• Application of quasi-equivariant metanetworks in domains such as computational chemistry and physics, where only approximate symmetries exist and maximal symmetry group identification is infeasible or ill-defined.
• Deeper exploration of the trade-offs between relaxation strength (degree of quasi-equivariance) and stability/robustness in large-scale, high-dimensional weight spaces.

Conclusion

"Quasi-Equivariant Metanetworks" provides a rigorous and practically trained approach to the longstanding challenge of symmetry in parameter space learning. The formalization of quasi-equivariance bridges the gap between rigid symmetry requirements and the need for expressive, robust metanetworks. Both theoretical justification and empirical validation underscore that quasi-equivariant architectures deliver superior trade-offs, with broad implications for future metanetwork and weight-space analysis, especially as model zoos and hypernetworks grow in scale and diversity.