ScaleGMNs: Equivariant Graph Metanetworks

• The topic defines ScaleGMNs as graph metanetworks that represent neural networks as graphs, encoding hidden-unit permutations and activation-induced scaling symmetries.
• They employ scale-equivariant message passing and invariant readouts to enable meta-optimization, single-shot fine-tuning, and parameter canonicalization for diverse applications.
• Empirical results demonstrate improved performance in INR classification and CNN generalization prediction, highlighting practical gains in weight-space learning.

Scale Equivariant Graph Metanetworks (ScaleGMNs) are graph metanetworks for processing neural networks in weight space while respecting the function-preserving symmetries of neural-network parameterizations, especially hidden-unit permutations and activation-induced scaling symmetries. In the formulation introduced by Kalogeropoulos et al., a neural network is treated as a graph whose vertices are neurons or channels and whose edges are parameters, and message passing is redesigned so that hidden-node and edge representations transform equivariantly under valid reparameterizations, while graph-level readout is invariant to those reparameterizations (Kalogeropoulos et al., 2024). Subsequent work positioned ScaleGMNs as a backbone for symmetry-aware fully-amortized optimization and for autoencoder-based parameter canonicalization, extending the framework from network classification and prediction to single-shot fine-tuning and model merging without explicit combinatorial alignment (Kuipers et al., 9 Oct 2025, Boufalis et al., 16 Nov 2025).

1. Emergence from weight-space learning over neural networks

ScaleGMNs arose within the broader problem of learning higher-order functions whose inputs are themselves neural networks. In this setting, the target may be a scalar predictor, a classifier, a parameter-editing operator, or another functional over the represented network, but the available input is the parameterized network $(G,\theta)$ rather than oracle access to its function values. The original ScaleGMN paper formulates this as learning an approximation $\hat f:\mathcal G\times \Uptheta \to Y$ to a functional $F:\hat X^X\to Y$, and argues that such a metanetwork should respect all function-preserving reparameterizations of the input network, not only permutations of hidden units (Kalogeropoulos et al., 2024).

Earlier graph metanetworks already exploited permutation symmetry by representing a feedforward network as a graph and applying message passing to neuron and parameter features. ScaleGMNs extend that design principle by incorporating scaling symmetries induced by common activations such as ReLU, $\tanh$, and sine. This extension is central to the framework’s claim that permutation-equivariant processing alone is incomplete for weight-space learning. Later papers retained the same core thesis while shifting the application domain: one used ScaleGMNs as a fully-amortized meta-optimizer that directly predicts improved weights in one forward pass, and another used them as invariant encoders in an autoencoder whose goal is parameter canonicalization for model merging (Kuipers et al., 9 Oct 2025, Boufalis et al., 16 Nov 2025).

A related but distinct line of work in geometric graph learning is the Similarity-Equivariant Graph Neural Network for homogenization of metamaterials, which extends $E(n)$-equivariance to the similarities group by incorporating uniform scaling. That model operates on finite-element geometries rather than on neural-network weight space, but it provides an adjacent example of scale symmetry being enforced as an inductive bias in graph learning (Hendriks et al., 2024).

2. Symmetry structure in neural-network parameterizations

The mathematical premise of ScaleGMNs is that neural-network weight space contains nontrivial symmetry orbits: multiple parameter vectors implement the same function. For feedforward networks, the classical symmetry is hidden-layer permutation. If $P_\ell$ permutes hidden units in layer $\ell$, then

$$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$

with $P_0=P_L=I$, leaves the represented function unchanged. Standard graph metanetworks are designed around this structure (Kalogeropoulos et al., 2024).

ScaleGMNs additionally incorporate scaling symmetries. For activations satisfying a relation of the form $\sigma(ax)=b\,\sigma(x)$, hidden units may be rescaled without changing the realized function, provided adjacent layers are transformed consistently. The original formulation states that, for many activations, the function-preserving linear symmetries are monomial matrices $\hat f:\mathcal G\times \Uptheta \to Y$0, where $\hat f:\mathcal G\times \Uptheta \to Y$1 is a permutation and $\hat f:\mathcal G\times \Uptheta \to Y$2 is diagonal, with diagonal entries restricted by the activation-specific set $\hat f:\mathcal G\times \Uptheta \to Y$3. For ReLU, positive scalings are allowed; for $\hat f:\mathcal G\times \Uptheta \to Y$4 and sine, the admissible diagonal entries are restricted to $\hat f:\mathcal G\times \Uptheta \to Y$5 (Kalogeropoulos et al., 2024, Boufalis et al., 16 Nov 2025).

The layerwise action used in the canonicalization work is

$\hat f:\mathcal G\times \Uptheta \to Y$6

This expresses the fact that hidden units may be permuted and rescaled consistently across adjacent layers without altering the network function. The same paper emphasizes that permutation-only alignment is incomplete: two models can be identical up to scaling transformations yet remain far apart in raw parameter space, and sequentially addressing one symmetry and then another can be harmful (Boufalis et al., 16 Nov 2025).

In the optimization paper, this redundancy is described as gauge freedom. A symmetry transformation $\hat f:\mathcal G\times \Uptheta \to Y$7 satisfies

$\hat f:\mathcal G\times \Uptheta \to Y$8

and a ScaleGMN should be invariant when it is used as a functional and equivariant when it is used as an operator:

$\hat f:\mathcal G\times \Uptheta \to Y$9

That paper further proves that the scale gauge freedom of a CNN layer is strictly smaller than that of an MLP layer with the same input/output dimensions, because convolution imposes a doubly-block Toeplitz weight-sharing structure that restricts admissible diagonal scalings. For MLPs, $F:\hat X^X\to Y$0, whereas for CNNs the admissible scalings collapse to essentially uniform factors $F:\hat X^X\to Y$1, $F:\hat X^X\to Y$2, yielding

$F:\hat X^X\to Y$3

This result is used to explain why scale-aware designs often show larger gains on MLPs than on CNNs (Kuipers et al., 9 Oct 2025).

3. Graph representation and scale-equivariant message passing

ScaleGMNs represent a neural network as a graph $F:\hat X^X\to Y$4 in which weights are edge features and biases are vertex features. For feedforward networks, neurons are vertices and directed connections are edges. For CNNs, channels become vertices, convolution kernels are flattened into edge features, and edge features are padded to a common dimension so that different kernel shapes can be handled uniformly (Kalogeropoulos et al., 2024, Kuipers et al., 9 Oct 2025).

The message-passing backbone follows the graph metanetwork template

$F:\hat X^X\to Y$5

$F:\hat X^X\to Y$6

$F:\hat X^X\to Y$7

$F:\hat X^X\to Y$8

The difference from a standard GMN is that initialization, message construction, feature updating, and readout are engineered to respect the transformation laws of weights and biases under valid scalings (Kalogeropoulos et al., 2024, Boufalis et al., 16 Nov 2025).

The central building blocks are scale-invariant, scale-equivariant, and rescale-equivariant modules. In the original construction,

$F:\hat X^X\to Y$9

$\tanh$0

and a rescale-equivariant operator satisfies

$\tanh$1

A practical construction is

$\tanh$2

The canonicalization work writes the same idea as

$\tanh$3

$\tanh$4

with $\tanh$5 removing the scale component. For sign symmetry,

$\tanh$6

and for positive scaling,

$\tanh$7

The point is that $\tanh$8 acts on scale-normalized canonical features while the $\tanh$9 maps reintroduce scale in an equivariant manner (Kalogeropoulos et al., 2024, Boufalis et al., 16 Nov 2025).

Message passing is designed so that the receiving node scales correctly under the joint transformation of receiver, sender, and edge. The original paper states

$E(n)$0

and implements this as

$E(n)$1

The optimization paper expresses the same construction as

$E(n)$2

with

$E(n)$3

and uses a scale-invariant readout obtained by canonicalization followed by Deep Sets (Kuipers et al., 9 Oct 2025).

Input and output neurons do not obey the same symmetry laws as hidden neurons, so the framework uses special handling for boundary nodes. The original paper also introduces a bidirectional ScaleGMN and initializes reverse edges with inverted weights,

$E(n)$4

so that backward edges transform correctly. For sign symmetry, inversion is unnecessary because $E(n)$5 for $E(n)$6 (Kalogeropoulos et al., 2024).

4. Equivariance, invariance, and expressivity

A central theorem of the original ScaleGMN work states that ScaleGMN is permutation/scale equivariant and, with a readout possessing the same symmetries, permutation/scale invariant. The proof proceeds by induction over message-passing depth: initialization respects the symmetry laws of biases and weights, message functions preserve the required transformation relations, update functions preserve node and edge transformation laws, and readout becomes invariant through canonicalization or symmetrization combined with DeepSets (Kalogeropoulos et al., 2024).

The same paper also gives an expressivity theorem: a bidirectional ScaleGMN with sufficiently expressive message and vertex-update functions can simulate both the forward and backward pass of any input feedforward neural network whose activations satisfy the stated symmetry characterization. Forward simulation requires $E(n)$7 message-passing steps; backward simulation requires $E(n)$8. The constructive proof stores biases, pre-activations, post-activations, and gradient-related quantities in vertex states, stores weights in edge states, and uses message passing to implement the recursions

$E(n)$9

together with backward propagation

$P_\ell$0

$P_\ell$1

A key technical observation is that if $P_\ell$2, then $P_\ell$3, which makes the backward simulation compatible with the same scale-aware design (Kalogeropoulos et al., 2024).

The formal results are accompanied by explicit limitations. The completeness of the scale/rescale-equivariant building blocks is not fully characterized, the full expressive power of ScaleGMN remains open, and the method is designed for feedforward neural networks and CNNs rather than arbitrary graphs with skip connections or normalization/softmax layers. For sine activations, additional periodic symmetries are treated separately by constraining biases to $P_\ell$4 or by a bias-shifting procedure described in the appendix (Kalogeropoulos et al., 2024).

5. Functional forms and principal applications

ScaleGMNs support two distinct modes. As an invariant functional, a ScaleGMN outputs a scalar score or prediction that is invariant to symmetry-related reparameterizations. As an equivariant operator, it outputs transformed parameters that obey the same symmetry action as the input parameters. The optimization paper formalizes this distinction as an operator $P_\ell$5 and an invariant functional, and uses the operator form as a fully-amortized meta-optimizer that maps $P_\ell$6 directly to improved weights $P_\ell$7 in one forward pass (Kuipers et al., 9 Oct 2025).

In the original evaluation, ScaleGMNs were applied to INR classification and to generalization prediction for trained CNNs. The tasks included MNIST INR, FashionMNIST INR, CIFAR-10 INR, Augmented CIFAR-10 INR, and Small CNN Zoo subsets for CIFAR-10-GS and SVHN-GS. The paper positions these tasks as cases where the object of interest is the function represented by a neural network, while only the parameterization is directly observed (Kalogeropoulos et al., 2024).

The fully-amortized optimization setting is different. Instead of generating weights from architecture alone, the method takes a pretrained model and predicts a better parameter vector in one shot. The training objective combines a sparsity term and a task loss,

$P_\ell$8

with gradients taken with respect to the metanetwork parameters $P_\ell$9, not the target-network weights. The reported use case is single-shot fine-tuning of pretrained CNNs and MLPs, including $\ell$0-regularized settings (Kuipers et al., 9 Oct 2025).

The parameter-canonicalization work uses ScaleGMNs in an autoencoder. The encoder $\ell$1 is symmetry-aware and the decoder $\ell$2 is a plain MLP:

$\ell$3

The goal is not exact parameter reconstruction but recovery of a preferred representative from the symmetry orbit,

$\ell$4

so that functionally equivalent parameter vectors collapse to a shared latent code. In this formulation, Neural Graphs provide permutation invariance only, whereas ScaleGMN provides permutation plus scaling invariance. The decoder is trained with functional losses rather than parameter-space losses: for INRs, the decoded network is compared with the original by regressing pixel activations using mean squared error; for CNNs, the loss is a KL divergence between temperature-softmax output distributions,

$\ell$5

with the intended effect that $\ell$6 (Boufalis et al., 16 Nov 2025).

Within model merging, the motivation is linear mode connectivity. Ainsworth et al. (2023) aligned models explicitly through combinatorial assignment, with interpolation

$\ell$7

The canonicalization paper contrasts this with learned amortized alignment: no assignment problem is solved at inference time, and the method instead learns a map into a canonical latent representation from which a canonical parameterization is decoded. The paper’s stated aim is to ensure that similar networks naturally converge within the same basin, facilitating smooth linear interpolation while avoiding regions of high loss (Boufalis et al., 16 Nov 2025).

6. Empirical record, comparative findings, and stated limitations

The original ScaleGMN paper reports state-of-the-art or near-state-of-the-art results on INR classification and CNN generalization prediction. For INR classification, the reported accuracies are MNIST $\ell$8 for ScaleGMN and $\ell$9 for ScaleGMN-B, FashionMNIST $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$0 and $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$1, CIFAR-10 $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$2 and $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$3, and Augmented CIFAR-10 $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$4 and $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$5. For CNN generalization prediction, the reported Kendall’s $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$6 values are approximately $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$7 on CIFAR-10-GS ReLU, $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$8 on SVHN-GS ReLU, $$W'_\ell = P_\ell W_\ell P_{\ell-1}^{-1}, \qquad b'_\ell = P_\ell b_\ell,$$9 on CIFAR-10-GS Tanh, and $P_0=P_L=I$0 on SVHN-GS Tanh. The same paper argues that these gains support scaling symmetries as an inductive bias, and notes that scaling coefficients are nontrivial in the datasets, so the symmetry is not vacuous (Kalogeropoulos et al., 2024).

In fully-amortized optimization, the evaluation used a CNN dataset based on Small CNN Zoo and an MLP zoo constructed by the authors. The CNN data comprised 30,000 CNNs trained on CIFAR-10-GS, filtered to 10,931 ReLU CNNs and 12,001 Tanh CNNs. The MLP data used $P_0=P_L=I$1 resized inputs flattened to 64 dimensions, architecture $P_0=P_L=I$2, Tanh hidden activations, and 5,386 trainable parameters, producing 6,135 retained networks after filtering above test accuracy 0.3. The baseline was SGD fine-tuning. For CNNs with Tanh, initial accuracy was $P_0=P_L=I$3, ScaleGMN-B reached $P_0=P_L=I$4, and SGD for 150 epochs reached $P_0=P_L=I$5. In the $P_0=P_L=I$6-regularized setting, ScaleGMN-B had test loss $P_0=P_L=I$7 versus $P_0=P_L=I$8 for SGD and achieved $P_0=P_L=I$9 sparsity versus $\sigma(ax)=b\,\sigma(x)$0. For MLPs with Tanh, initial accuracy was $\sigma(ax)=b\,\sigma(x)$1, ScaleGMN-B $\sigma(ax)=b\,\sigma(x)$2, and SGD 150 epochs $\sigma(ax)=b\,\sigma(x)$3; in the $\sigma(ax)=b\,\sigma(x)$4-regularized setting, test losses were $\sigma(ax)=b\,\sigma(x)$5 and $\sigma(ax)=b\,\sigma(x)$6, with the ScaleGMN result obtained in approximately $\sigma(ax)=b\,\sigma(x)$7 seconds rather than hundreds of seconds. Appendix results for ReLU CNNs reported initial accuracy $\sigma(ax)=b\,\sigma(x)$8, ScaleGMN-B $\sigma(ax)=b\,\sigma(x)$9, SGD 150 epochs $\hat f:\mathcal G\times \Uptheta \to Y$00, and $\hat f:\mathcal G\times \Uptheta \to Y$01-regularized test losses $\hat f:\mathcal G\times \Uptheta \to Y$02 and $\hat f:\mathcal G\times \Uptheta \to Y$03. Symmetry-broken variants labeled GMN-B (sym. broken) degraded more clearly on MLPs than on CNNs, matching the theory that CNNs offer less scaling gauge freedom (Kuipers et al., 9 Oct 2025).

In parameter canonicalization and model merging, the INR experiments used SIREN-like MLP INRs trained on MNIST digits, with symmetries injected as permutations only, scaling only, and permutations plus scaling. The reported qualitative findings are that naive interpolation exhibits a large loss barrier, Git Re-Basin reduces the barrier, both autoencoder variants yield near-linear mode connectivity, and ScaleGMN is slightly more stable or slightly better than permutation-only Neural Graphs, including when scaling transformations are present. Reported INR test reconstruction errors were $\hat f:\mathcal G\times \Uptheta \to Y$04 for ScaleGMN and $\hat f:\mathcal G\times \Uptheta \to Y$05 for Neural Graphs. Without added noise, ScaleGMN collapses all orbit members to a single latent point, producing a flat interpolation path. The CNN experiments used SmallCNN Zoo models on CIFAR-10 with $\hat f:\mathcal G\times \Uptheta \to Y$06 hidden channels, $\hat f:\mathcal G\times \Uptheta \to Y$07 convolutions, and both ReLU and Tanh variants. Averaged interpolation curves over 20 pairs showed that naive interpolation fails badly, Git Re-Basin helps but still leaves a substantial barrier, and the autoencoder framework yields nearly monotonic, much flatter interpolation. ScaleGMN and Neural Graphs were both effective, while ScaleGMN was marginally better on reconstruction metrics: for ReLU, $\hat f:\mathcal G\times \Uptheta \to Y$08 error $\hat f:\mathcal G\times \Uptheta \to Y$09 and Kendall’s $\hat f:\mathcal G\times \Uptheta \to Y$10; for Tanh, $\hat f:\mathcal G\times \Uptheta \to Y$11 error $\hat f:\mathcal G\times \Uptheta \to Y$12 and Kendall’s $\hat f:\mathcal G\times \Uptheta \to Y$13. The paper interprets these results as indicating that learned canonical space often merges models into a shared basin and that scaling awareness helps less for these CNNs than for INRs, though it still provides a principled orbit collapse (Boufalis et al., 16 Nov 2025).

The limitations stated across the ScaleGMN literature are consistent. The canonicalization framework requires training data for the autoencoder and has been tested on INRs and small CNNs rather than on large Transformers or LLMs. The same paper notes that ScaleGMN support is not universal, that generalized Git Re-Basin remains iterative and approximate even when extended to scaling or sign flips, that scaling matters differently depending on the downstream use case, and that perfect canonicalization is not guaranteed because the representative is learned through a functional loss. The original ScaleGMN paper additionally restricts its theoretical scope to feedforward networks and CNNs and leaves the full expressive characterization open (Boufalis et al., 16 Nov 2025, Kalogeropoulos et al., 2024).

7. Conceptual significance and relation to adjacent symmetry-aware graph learning

The main conceptual move introduced by ScaleGMNs is to treat function-preserving parameter symmetries as first-class structure in weight-space learning. Rather than viewing permutations as the only nuisance reparameterization, the framework enlarges the relevant symmetry group to include activation-specific scalings and designs the metanetwork so that internal representations transform according to the same gauge laws as the parameters of the input network. A plausible implication is that ScaleGMNs are best understood as metanetworks over equivalence classes of parameterizations rather than over raw weight vectors (Kalogeropoulos et al., 2024).

This perspective becomes especially explicit in the canonicalization work, which reframes model merging from explicit per-pair alignment to learning an invariant canonicalization map. The paper suggests a general route for handling network identifiability issues: symmetry orbits are treated as equivalence classes, invariant latent codes are learned for those classes, and canonical representatives are reconstructed from those codes. It further hints that the same machinery could support model merging, performance prediction, network classification, INR generative modeling, and possibly future extension to Transformers (Boufalis et al., 16 Nov 2025).

The broader graph-learning context shows that scale equivariance is not unique to weight-space neural-network processing. The Similarity-Equivariant Graph Neural Network for metamaterial homogenization extends an $\hat f:\mathcal G\times \Uptheta \to Y$14-equivariant GNN to the similarities group so that energy is invariant, deformed positions are equivariant, and stress and stiffness transform tensorially under coordinate changes. Its use of strain-like scale-free measures and normalized edge vectors is conceptually parallel to ScaleGMN’s use of canonicalization and rescale-equivariant message functions, but the two frameworks target different symmetry domains: SimEGNN encodes geometric and physical symmetries of finite-element discretizations, whereas ScaleGMN encodes the reparameterization symmetries of neural-network weight space (Hendriks et al., 2024).

Taken together, the ScaleGMN papers define a symmetry-aware program for neural-network metaprocessing. The original framework established permutation and scaling equivariance in graph metanetworks, later work used the same machinery for single-shot fine-tuning in weight space, and the most recent extension used it as the encoder backbone of an autoencoder for parameter canonicalization and basin-aware model merging (Kalogeropoulos et al., 2024, Kuipers et al., 9 Oct 2025, Boufalis et al., 16 Nov 2025).