Equivariant Foundation Models

• Foundation and equivariant models are architectures that embed group symmetry to enforce consistent transformations, enhancing model generalization and data efficiency.
• They integrate transformation laws (e.g., SO(3), E(3)) within layers to maintain structure under input changes, leading to improved robustness across various domains.
• Leveraging vast, diverse datasets and scaling principles, these models achieve superior performance in vision, language, and scientific applications through universal approximation.

Foundation and Equivariant Models comprise a rapidly evolving class of machine learning architectures that encode inductive biases reflecting natural or task-specific symmetries, and are trained or pre-trained on large, diverse datasets for universal downstream transfer. The fusion of equivariance principles and foundation-model scaling fundamentally changes how models generalize, calibrate, and infer on a variety of modalities, including vision, language, multimodal, time-series, graph, and molecular domains. Equivariant models enforce precise transformation laws—often with respect to groups such as SO(3), E(3), O(n), Sk, or product groups—throughout their architecture, yielding improved data efficiency, robustness, and compositionality. Foundation models leverage vast data and model-scale to realize universal approximators across problem domains; the intersection of the two (“equivariant foundation models”) has revolutionized modeling in scientific, vision-language, and graph domains.

1. Mathematical and Architectural Foundations of Equivariance

At the core of equivariant model design is a formal specification of group symmetry. Let $G$ be a group acting on input space $\mathcal{X}$ and output space $\mathcal{Y}$ via representations $\rho_{in}$, $\rho_{out}$. A mapping $f:\mathcal{X}\to\mathcal{Y}$ is $G$-equivariant if

$$f(\rho_{in}(g)\cdot x) = \rho_{out}(g) \cdot f(x) \quad \forall g\in G,\,\, x\in\mathcal{X}.$$

This ensures that if the input is transformed by a symmetry operation, the output co-transforms, preserving structure prescribed by $G$. Architecturally, equivariance is enforced by constraining layerwise operations (e.g., convolutions, attentions, message passing) to intertwine the group actions. For geometric data, this includes $\mathrm{E}(3)$-equivariant MPNNs for atomic systems, permutation-equivariant layers for sets/graphs, and group-convolution layers parameterized in harmonic or irreducible-representation basis for images and manifolds (Gerken et al., 2021, Chen et al., 2022). In foundation models, these structures are scaled to billions of parameters and trained across distributions of tasks and domains (Nomura et al., 9 Feb 2025, Yan et al., 25 Feb 2025).

2. Equivariant Losses and Similarity Objectives in Vision-Language Foundation Models

Conventional vision-LLMs such as CLIP, ALIGN, and METER optimize objectives that push the similarity $\mathcal{X}$0 of image-text pairs to be high for matched and low for unmatched pairs. This enforces binary invariance but not granularity of semantic change. Equivariant objectives, as formalized in EqSim (Wang et al., 2023), require predictability of similarity changes under minimal semantic perturbations in image or text—interpolating between invariance and pure matching. The EqSim loss employs regularizers that enforce, for paired samples, equideviant responses under image or text perturbations:

$\mathcal{X}$1

with practical terms optimizing mean-square deviation from these ratios over “close” and “distant” sample pairs. This framework is architecture-agnostic and pluggable into contrastive or retrieval fine-tuning, enabling graded, semantics-aware responses crucial for compositionality and downstream transfer (Wang et al., 2023).

3. Equivariance in Computational Imaging and Inverse Problems

Equivariant models in imaging leverage group symmetries—translation, rotation, scaling—directly in the architecture via group convolutions or representation-constrained neural operators. For a forward model $\mathcal{X}$2, non-equivariance of $\mathcal{X}$3 (due to acquisition physics) motivates two strategies: (1) equivariance by design—embedding $\mathcal{X}$4-equivariant layers in unrolled solvers (e.g., proximal gradient descent with equivariant proximal operators); and (2) equivariance by learning—using data augmentations or self-supervised losses that penalize deviation from equivariance. Equivariant imaging delivers both theoretical improvements (e.g., $\mathcal{X}$5-fold sample complexity reduction, enhanced identifiability for unsupervised learning) and empirical boosts (e.g., MRI PSNR improvements of $\mathcal{X}$6–$\mathcal{X}$7 dB at half the data, near-supervised performance in unsupervised settings) (Chen et al., 2022). The architecture exploits group convolutional layers and equivariant nonlinearities, with precise mathematical formulations governing forward models and reconstruction losses.

4. Foundations and Universality: Theory and Scaling

A universal property of equivariant models is their completeness within the constrained function class specified by the symmetry. For example, triple-equivariant graph models for node-level tasks (respecting node-permutation, label-permutation, and feature-permutation symmetries) are universal in the space of continuous functions invariant and equivariant in the prescribed arguments, and guarantee zero-shot transfer across datasets with arbitrary node, label, and feature ordering (Finkelshtein et al., 17 Jun 2025). In time-series, 2D permutation-equivariant state space models decompose all permissible coupling matrices into canonical forms (local plus global pooled interactions), simplifying dependence and stability analysis, and permitting scaling to high-dimensional multivariate data (Jeong et al., 7 Mar 2026).

Foundation models inject this universal expressivity into pre-trained, large-capacity architectures by training across massive, diverse datasets (e.g., over 80 elements in materials, multi-million chemical structures, vision-language pairs) and aligning with physical or semantic invariances (e.g., $\mathcal{X}$8 in molecular dynamics; O(2) in histopathology imaging) (Nomura et al., 9 Feb 2025, Chen et al., 14 Jan 2026, Yan et al., 25 Feb 2025). Hybrid architectures, such as HIENet, integrate rapid invariant layers and expressive equivariant modules, enforcing all physical constraints (energy conservation, force and stress equivariance) by construction and driving forward both in-domain and out-of-distribution generalization (Yan et al., 25 Feb 2025).

5. Transfer, Symmetry Breaking, and Non-Euclidean Foundations

Equivariant transfer learning extends non-equivariant foundation backbones to symmetry-aware outputs via group-averaged (Equitune) or data-driven weighted (λ-Equitune) aggregation over transformed features—provably producing G-equivariant predictors and attaining universality (Basu et al., 2023). Symmetry-breaking inputs, as formalized in Any-Subgroup Equivariant Networks (ASEN), let a shared base model accommodate multiple or partial symmetry constraints by modulating auxiliary tensors whose automorphism group matches the target subgroup, with efficient computation via relaxed 2-closure orbits (Goel et al., 19 Mar 2026).

Beyond fixed symmetries, geometric foundation models advocate adapting model geometry to intrinsic data structure, leveraging non-Euclidean manifolds (hyperbolic, spherical, product spaces) to reduce distortion, align with physics or hierarchy, and increase expressivity per dimension. Attention, residuals, normalization, and other primitives are redefined on Riemannian manifolds, supporting manifold-aware pretraining, hybrid schemes, and dynamic geometry selection (He et al., 11 Apr 2025).

6. Applications: Scientific Discovery, Safety, and Interpretability

Equivariant foundation models have catalyzed advances in molecular dynamics, materials discovery, and structural prediction. For instance, Allegro-FM, an E(3)-equivariant foundation model trained by total energy alignment, delivers linear scaling and near-DFT accuracy for emergent phenomena in exascale simulations (e.g., fracture, reaction kinetics, solid–liquid dissolution) (Nomura et al., 9 Feb 2025). QET integrates an analytically solvable, linear-scaling charge-equilibration block into an equivariant TensorNet backbone, addressing charge transfer and reactivity bottlenecks in atomistic potentials, and enabling electrochemical simulations at scale (Ko et al., 10 Nov 2025). GL(r)-equivariant “Learning on LoRAs” models provide direct performance and membership diagnostics on finetuned weight spaces of large models (Putterman et al., 2024). For explainability, surrogate models built from group equivariant operators (GEOs) and their diagrammatic complexity enable rigorous, observer-adapted quantification of model interpretability, with empirical verification of accuracy-complexity tradeoffs (Colombini et al., 3 Mar 2025).

In vision-language settings, equivariant similarity models (e.g., EqSim) and specialized benchmarks (EqBen) enable rigorous assessment and enhancement of compositional generalization, supporting robust multimodal matching under nuanced perturbations (Wang et al., 2023). In computational imaging and time-series analysis, equivariant architectures yield sample-efficient, robust predictions in inverse problems and multi-way forecasting (Chen et al., 2022, Jeong et al., 7 Mar 2026).

7. Open Challenges and Future Directions

Key limitations include the need for richer semantic change models in multimodal settings (beyond “minimal” perturbations), efficient mining of hard negatives at scale for training equivariant losses, and integration of active learning or online adaptation in disordered and out-of-distribution regimes (Wang et al., 2023, Sivaraman et al., 2024). Fine-grained subgroup symmetry handling (e.g., for anatomical or chemical subsets) remains an open computational challenge (Goel et al., 19 Mar 2026). Expanding standardized downstream benchmarks, particularly in regression and scientific domains, is needed to further evaluate generalization and universal scaling (Finkelshtein et al., 17 Jun 2025).

Foundational work is ongoing in (1) instruction tuning of multimodal generative models for equivariant responses, (2) exact group-equivariant layer integration (e.g., steerable CNNs, tensor networks, group attention) into large-scale pretraining, and (3) full realization of hybrid and adaptive non-Euclidean architectures (He et al., 11 Apr 2025). Interpretable design via observer-parameterized complexity, safety auditing using equivariant weight diagnostics, and routine inclusion of explicit charge or higher multipole data in open datasets are emerging as future best practices (Colombini et al., 3 Mar 2025, Ko et al., 10 Nov 2025, Putterman et al., 2024).

---

References

• Equivariant Similarity for Vision-Language Foundation Models (Wang et al., 2023)
• Imaging with Equivariant Deep Learning (Chen et al., 2022)
• Permutation-Equivariant 2D State Space Models (Jeong et al., 7 Mar 2026)
• Efficient Equivariant Transfer Learning from Pretrained Models (Basu et al., 2023)
• Any-Subgroup Equivariant Networks via Symmetry Breaking (Goel et al., 19 Mar 2026)
• Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces (Putterman et al., 2024)
• The Lie Derivative for Measuring Learned Equivariance (Gruver et al., 2022)
• Equi-ViT: Rotational Equivariant Vision Transformer (Chen et al., 14 Jan 2026)
• Mathematical Foundation of Interpretable Equivariant Surrogate Models (Colombini et al., 3 Mar 2025)
• Allegro-FM: Towards Equivariant Foundation Model for Exascale Molecular Dynamics Simulations (Nomura et al., 9 Feb 2025)
• A Materials Foundation Model via Hybrid Invariant-Equivariant Architectures (Yan et al., 25 Feb 2025)
• Equivariance Everywhere All At Once: A Recipe for Graph Foundation Models (Finkelshtein et al., 17 Jun 2025)
• Deciphering diffuse scattering with machine learning and the equivariant foundation model (Sivaraman et al., 2024)
• Geometric Deep Learning and Equivariant Neural Networks (Gerken et al., 2021)
• Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries (He et al., 11 Apr 2025)
• A Fast, Accurate, and Reactive Equivariant Foundation Potential (Ko et al., 10 Nov 2025)