Soft Equivariant Models

• Soft equivariant models are systems that treat symmetry as a graded, learnable bias, balancing exact equivariance with flexibility for real-world data.
• They employ methods such as regularizers, constrained optimization, and Bayesian selection to interpolate between rigid symmetry and unconstrained models.
• These models improve generalization in noisy, mixed, or latent symmetry scenarios and find applications in vision, dynamics, and reinforcement learning.

Searching arXiv for papers on soft equivariance and symmetry discovery to ground the article. Soft equivariant models are learning systems that treat symmetry as a graded, learnable, or partially enforced inductive bias rather than as an inflexible architectural constraint. In the exact setting, equivariance is the condition

$f(\rho_{\mathcal X}(g)x)=\rho_{\mathcal Y}(g)f(x),$

but much recent work asks how to exploit symmetries that are approximate, mixed, latent, or unknown, and how to interpolate between fully equivariant and fully unconstrained models. Across this literature, symmetry enters through regularizers, constrained optimization, learnable augmentations, projected parameter subspaces, layer-wise symmetry selection, or direct symmetry discovery, with the common aim of retaining symmetry-aware generalization while avoiding the rigidity of hard-coded equivariant architectures (Kim et al., 2023, Manolache et al., 19 May 2025, Santos-Escriche et al., 4 Jun 2025).

1. Conceptual scope and problem setting

The classical equivariant-network paradigm assumes that the relevant group action is known in advance and valid exactly across the dataset. Soft equivariant models arise when this assumption is weakened. One major motivation is approximate symmetry: real datasets may violate symmetry because of noise, gravity, sensors, wind, coordinate normalization, structural variation, measurement bias, phase transitions, or other symmetry-breaking effects. A second motivation is mixed symmetries, where several candidate groups are relevant but not to the same degree. In such settings, a hard-equivariant model can be too rigid and may underfit because it cannot represent the symmetry-breaking effects present in the data (Kim et al., 2023).

A related motivation comes from latent or partial symmetry. In many domains, the true symmetry is not cleanly expressible as a simple input-space transformation. The literature distinguishes correct equivariance, incorrect equivariance, and extrinsic equivariance. Correct equivariance matches the task symmetry on the data support; incorrect equivariance ties together samples whose labels should differ and can impose a hard performance ceiling; extrinsic equivariance applies a proxy transformation that maps observations outside the data distribution and can nonetheless improve sample efficiency and generalization. This distinction is central in domains where the true symmetry is obscured by viewpoint, occlusion, projection, or other observation-level distortions (Wang et al., 2022).

Within this framing, equivariance becomes a negotiable inductive bias rather than a binary architectural property. Some methods seek exact recovery of equivariance when the data supports it, while others deliberately learn a data-driven equilibrium between equivariance and non-equivariance. This suggests that “softness” is not a single mechanism but a family of strategies for controlling how strongly symmetry constrains the model (Manolache et al., 19 May 2025).

2. Principal methodological families

Several technical families dominate the current landscape.

Family	Core mechanism	Representative papers
Regularizer-based soft equivariance	Penalize distance of layer parameters from equivariant and invariant subspaces, possibly for multiple candidate groups	(Kim et al., 2023)
Constrained homotopy relaxation	Use equivariant and non-equivariant layer components with learned deviation variables and primal-dual updates	(Manolache et al., 19 May 2025)
Bayesian layer-wise symmetry selection	Interpolate between fully connected and convolutional structure and learn layer-wise symmetry strength by marginal likelihood	(Ouderaa et al., 2023)
Tunable subspace projection	Project pretrained weights into a symmetry-designed subspace using a cutoff $b$	(Rahman et al., 27 Mar 2026)

The regularizer-based line is exemplified by the projection-based equivariance regularizer (PER). For a symmetry group $G_k$, it uses orthonormal bases $Q_k$ and $R_k$ for equivariant linear maps and invariant biases, and penalizes the Euclidean distance of the layer parameters from those subspaces:

$${}^\text{PER}_k(W,b) = \frac{\lambda_k}{2}\left\|\mathrm{vec}(W)-Q_kQ_k^{\intercal}\mathrm{vec}(W)\right\|^2 + \frac{\lambda_k}{2}\left\|b-R_kR_k^{\intercal} b\right\|^2.$$

The total penalty is the sum over candidate symmetries, and the coefficients $\lambda_k$ are automatically reweighted during training to reflect the apparent approximation level of each symmetry type. This makes PER especially suited to mixed approximate symmetries, while leaving the architecture itself as an unconstrained MLP (Kim et al., 2023).

A second family replaces penalties by explicit constrained dynamics. Adaptive Constrained Equivariance (ACE) writes each layer as

$$f_{\theta,\gamma}^i = f_{\theta}^{\text{eq},i} + \gamma_i f_{\theta}^{\text{neq},i},$$

so that $\gamma_i$ controls the deviation from equivariance. For fully symmetric data, ACE enforces $\gamma_i=0$ as equality constraints; for imperfectly symmetric data it introduces slack variables $b$0 and resilient inequality constraints $b$1. The resulting primal-dual updates begin from a flexible non-equivariant model, typically with $b$2, and gradually tighten symmetry during training. In this sense ACE operationalizes soft equivariance as a homotopy from easy optimization and high flexibility toward a more structured solution (Manolache et al., 19 May 2025).

A third family treats symmetry as a layer-wise model-selection problem. In “Learning Layer-wise Equivariances Automatically using Gradients,” a relaxed layer is a sum of a fully connected path and an equivariant convolutional path, later compressed by factored fully connected layers and spatial sparsification. The key hyperparameters are the prior variances controlling the non-equivariant and equivariant components, with the limit $b$3 corresponding to strict equivariance. Rather than adding an ad hoc regularizer, the method uses marginal likelihood, estimated by a differentiable Laplace approximation with generalized Gauss-Newton and KFAC, to balance data fit against an Occam factor. This yields layer-wise symmetry discovery rather than a single global symmetry choice (Ouderaa et al., 2023).

A fourth family performs projection into designed symmetry subspaces. “Tunable Soft Equivariance with Guarantees” constructs soft invariant and soft equivariant layers by projecting pretrained weights into a subspace chosen from the SVD or Schur decomposition of the Lie algebra representation. The cutoff $b$4 is an explicit softness parameter: smaller $b$5 keeps more symmetry-respecting directions, while larger $b$6 admits more symmetry-breaking directions. This framework is notable for being architecture-agnostic and applicable to pretrained backbones such as ViT, DINOv2, ResNet-50, and SegFormer (Rahman et al., 27 Mar 2026).

3. Symmetry discovery rather than symmetry prescription

A major development within soft equivariant modeling is the shift from merely relaxing a known symmetry to discovering the symmetry from data. One route is algebraic. “Learning Equivariant Functions via Quadratic Forms” studies orthogonal groups of the form

$b$7

and reframes symmetry discovery as learning the quadratic form $b$8. Because $b$9 may be taken symmetric and diagonalized as $G_k$0, learning equivariance becomes learning a symmetry-adapted basis $G_k$1 and signature $G_k$2. The resulting equivariant model factors as

$G_k$3

where the “product” is the group action, $G_k$4 is norm-invariant, and $G_k$5 is scale-invariant. The same principle extends to tuples under diagonal action, where the invariant component depends on the full $G_k$6-Gram matrix and the angular component is extracted from the normalized first vector. This places symmetry discovery directly inside the predictor rather than treating it as a preprocessing stage (Karjol et al., 26 Sep 2025).

A second route is augmentation-based discovery. SEMoLA parameterizes a continuous symmetry through a learnable Lie algebra basis $G_k$7, generates group elements by

$G_k$8

and trains an arbitrary backbone with a combined task loss and equivariance loss over these learned transformations. Additional regularizers encourage augmentation non-triviality, basis orthogonality, and basis sparsity. The central point is that the symmetry is discovered jointly with the downstream objective, rather than learned in a separate unsupervised stage. This gives the learned symmetry an explicit and inspectable representation in the Lie algebra basis (Santos-Escriche et al., 4 Jun 2025).

These discovery-based approaches differ in their admissible symmetry classes. Quadratic-form discovery targets orthogonal and generalized orthogonal symmetries; SEMoLA is formulated for continuous Lie groups through learnable augmentations. A plausible implication is that symmetry discovery in soft equivariant models is currently strongest when the candidate family has a tractable algebraic parameterization.

4. Measurement, guarantees, and approximation-theoretic limits

Soft equivariance requires both a way to measure learned symmetry and a way to reason about the consequences of relaxing exact equivariance. The measurement problem is addressed by the Lie derivative framework. For a continuous symmetry generated by a vector field $G_k$9, the Lie derivative

$Q_k$0

measures infinitesimal symmetry violation, and the associated Local Equivariance Error (LEE)

$Q_k$1

gives a local, continuous, and layer-decomposable scalar summary. This framework was used to study hundreds of pretrained models and showed that many violations of equivariance are linked to spatial aliasing, including aliasing introduced by pointwise nonlinearities. It also showed that larger and more accurate models tend to display more equivariance regardless of architecture, and that transformers can be more equivariant than convolutional neural networks after training (Gruver et al., 2022).

The guarantee side appears in several forms. PER proves that the equivariance error of an MLP is bounded by the sum of projection distances of its layers from equivariant subspaces. ACE derives explicit bounds showing that small $Q_k$2 implies both small deviation from the strictly equivariant model and small equivariance error. The projection framework of (Rahman et al., 27 Mar 2026) introduces a relative notion of $Q_k$3-soft equivariance,

$Q_k$4

and proves bounds of the form $Q_k$5 or $Q_k$6, depending on the layer type (Kim et al., 2023, Manolache et al., 19 May 2025, Rahman et al., 27 Mar 2026).

Approximation theory imposes an additional caution. “On Universality Classes of Equivariant Networks” shows that separation power is not enough to characterize expressivity: model classes can distinguish the same inputs modulo symmetry and nonetheless approximate different sets of continuous functions. The paper characterizes shallow invariant universality classes using constant-coefficient differential operators, derives non-universality criteria, and shows that positive shallow universality results depend critically on structural group properties such as the existence of suitable normal subgroups. This is especially restrictive for permutation symmetry, where the relevant normal subgroups are scarce (Pacini et al., 2 Jun 2025). A common misconception is therefore that any soft symmetry mechanism preserving the same orbit separation automatically preserves the same approximation power; the theory rejects that conclusion.

5. Empirical domains and recurrent performance patterns

Soft equivariant models have been tested across regression, classification, scientific ML, self-supervised representation learning, reinforcement learning, diffusion steering, and dynamics modeling. In supervised settings, quadratic-form discovery achieved lower validation MSE than EMLP and a simple MLP on invariant polynomial regression, recovered the Lorentz metric in top quark tagging while performing symmetry discovery and classification in a single network, and achieved much lower validation loss than EMLP on moment of inertia matrix prediction (Karjol et al., 26 Sep 2025). SEMoLA recovered the correct $Q_k$7 generator on RotatedMNIST, learned the correct $Q_k$8 Lie algebra basis up to permutation on QM9, and remained robust under out-of-distribution symmetry shifts where symmetry-discovery-only methods degraded more strongly (Santos-Escriche et al., 4 Jun 2025). Tunable subspace projection improved performance while reducing equivariance error across image classification, semantic segmentation, human-trajectory prediction, and a synthetic $Q_k$9 benchmark, including competitive ImageNet results (Rahman et al., 27 Mar 2026).

In representation learning, STAR reformulates the standard invariant/equivariant dual-head design as a soft routing-of-experts mechanism. Its shared and task-specific experts reduce redundant feature learning, as evidenced by lower canonical correlations between invariant and equivariant embeddings and lower cosine similarity between backbone gradients from different experts. This yields consistent improvements in out-of-domain linear transfer, object detection, and few-shot classification (Jeon et al., 31 Oct 2025).

In control and dynamics, approximate symmetry is often indispensable. ACE improved performance metrics, sample efficiency, and robustness to input perturbations across SEGNN, EGNO, VN-DGCNN, and SchNet on N-Body simulations, QM9, ModelNet40, and CMU MoCap, with especially clear benefits when symmetry was only partial (Manolache et al., 19 May 2025). “Symmetry-Aware Steering of Equivariant Diffusion Policies” showed that steering in latent-noise space can benefit from strict or approximate equivariant RL, with approximate equivariance outperforming strict equivariance in symmetry-breaking settings such as Stack D1 (Park et al., 12 Dec 2025). In object-centric world modeling, a geometric algebra architecture with a soft geometric inductive bias outperformed non-equivariant baselines and often strict equivariant variants on 2D rigid-body dynamics with static obstacles, particularly in long-horizon rollouts where boundary effects broke exact $R_k$0-equivariance (Linander et al., 17 Dec 2025).

Across these studies, a repeated empirical pattern emerges. When symmetry is strong and faithfully realized, stricter equivariance is often best. When symmetry is mixed, approximate, latent, or broken by the observation process or environment, soft mechanisms tend to offer a better trade-off between geometric structure and task fit.

6. Relation to hard equivariance, misconceptions, and open questions

Soft equivariant models do not displace hard-equivariant architectures; rather, they occupy the regime where exact symmetry is unavailable, uncertain, or too restrictive. This distinction matters because hard-equivariant methods remain extremely strong when the symmetry is known and correct. In robotic manipulation, $R_k$1-equivariant DQN and SAC were significantly more sample efficient than augmentation-based alternatives, and in on-robot learning, Equivariant SAC with $R_k$2 equivariance and buffer augmentation learned several manipulation tasks directly on hardware in about 45 minutes to 2 hours 40 minutes (Wang et al., 2022, Wang et al., 2022). These results are consistent with the view that if the architectural symmetry matches the task, hard equivariance can dominate softer alternatives.

At the same time, several misconceptions recur. Soft equivariance is not synonymous with simple data augmentation, since the literature includes parameter-space regularization, constrained optimization, Bayesian evidence maximization, learnable augmentations, algebraic symmetry discovery, and tunable projection of pretrained weights. Nor is soft equivariance equivalent to abandoning structure: in nearly all cases the model is guided toward explicitly defined equivariant, invariant, or approximately equivariant subspaces. Finally, proxy symmetry is not automatically harmful. The distinction between extrinsic equivariance and incorrect equivariance shows that a proxy transformation can help if it does not impose false label equalities on the data support, while genuinely incorrect equivariance can cap or degrade performance (Wang et al., 2022).

The open problems are correspondingly specific. Many methods still require a prescribed candidate family of groups rather than arbitrary symmetry discovery from scratch, as in PER or SEMoLA (Kim et al., 2023, Santos-Escriche et al., 4 Jun 2025). Some discovery mechanisms are restricted to specific algebraic classes, such as orthogonal groups defined by quadratic forms (Karjol et al., 26 Sep 2025). Several adaptive procedures remain heuristic, including automatic coefficient tuning in PER and validation-based selection of the projection cutoff $R_k$3 (Kim et al., 2023, Rahman et al., 27 Mar 2026). Theoretical guarantees are often sharpest for linear layers or shallow networks, while deeper nonlinear expressivity remains subtler (Pacini et al., 2 Jun 2025, Rahman et al., 27 Mar 2026). Measurement studies also indicate an equivariance gap between training, test, and out-of-distribution data, implying that learned soft equivariance may not extrapolate as robustly as exact architectural symmetry (Gruver et al., 2022).

Soft equivariant models therefore define not a single architecture class but a research program: how to exploit symmetry when symmetry is only partly valid, partly known, or itself a target of inference. The field’s central insight is that symmetry can be enforced, tuned, routed, measured, or discovered in degrees, and that this graded view is often more faithful to real data than the older dichotomy between perfectly equivariant and entirely unconstrained models.