Invariance Pre-training Principle

• Invariance pre-training is the practice of designing representations that remain unchanged under specific data transformations to enhance downstream performance.
• It employs methods like invariant integration layers, learned augmentation distributions, and contrastive objectives to hard-wire invariance prior to or along with supervision.
• Empirical evidence shows that aligning induced invariances with task-relevant transformations boosts robustness, transfer learning, and data efficiency.

Searching arXiv for recent and foundational papers related to invariance pre-training and invariance-oriented representation learning. Across several lines of work, the Invariance Pre-training Principle can be understood as the practice of constructing, selecting, or preserving invariances in the representation before, or jointly with, downstream supervision, rather than expecting a classifier to discover them from task data alone. In this sense, invariance may be hard-wired by architecture, induced by a learned augmentation distribution, enforced by contrastive or IRM-style objectives, measured as a transferable layer-wise property of pretrained models, or added by parameter-efficient tuning to foundation models (Rath et al., 2020, Benton et al., 2020, Wang et al., 2022, Merlin et al., 2023, Ruan et al., 2024). This suggests a unifying view: pre-training is not only feature acquisition, but also invariance acquisition.

1. Conceptual and mathematical basis

At its most basic, invariance is the requirement that a representation or predictor remain unchanged under a transformation. For translation, this is written as

$f(T_{\theta}(I)) = f(I),$

and, more generally, for a transformation $T$,

$g(Tx) = g(x).$

Equivariance is the weaker requirement that the representation transform predictably with the input; in the semantic formulation of EqInv, a representation is equivariant if

$$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$

Several works therefore treat equivariance as a precursor to invariance: preserve all semantic changes first, then selectively discard nuisance variation (Wang et al., 2022, Biscione et al., 2021).

A stronger notion appears in invariant integration. A feature space is defined as complete if all patterns equivalent with respect to a transformation group $G$ are mapped to the same point in the feature domain, while all distinct patterns are mapped to different points. For finite groups, invariant integration uses the group average

$$A[f](\mathbf{x}) = \frac{1}{|G|}\sum_{g \in G} f(g\mathbf{x}),$$

and, with appropriate monomials $m(\mathbf{x})$, constructs a complete invariant feature map

$$\Phi(\mathbf{x}) = \big(A[m_1](\mathbf{x}), \ldots, A[m_M](\mathbf{x})\big).$$

This notion is stricter than heuristic pooling, because it aims at both invariance and separation of non-equivalent inputs (Rath et al., 2020).

A second formalization treats invariance as averaging over a learned transformation distribution. Augerino defines an approximately invariant predictor

$$\bar{f}(x) = \mathbb{E}_{g \sim \mu_\theta} f_w(gx),$$

and jointly optimizes network parameters $w$ and augmentation parameters $T$0. In this view, the invariance profile is a learnable part of the model, not a fixed preprocessing choice (Benton et al., 2020). A related Bayesian line of work goes further and treats invariance parameters as hyperparameters of a prior over functions, learned by maximizing an approximate marginal likelihood via a differentiable Laplace approximation (Immer et al., 2022), or, in the GP setting, by maximizing the marginal likelihood of invariant kernels defined by augmentation distributions (Wilk et al., 2018).

The transfer-learning perspective makes invariance measurable at the representation level. In synthetic settings, the sensitivity metric

$T$1

quantifies how much the representation changes when only the transformation changes. Lower sensitivity means higher invariance, and this metric is used to show that transfer performance depends strongly on whether pre-training induces the right transformations as invariances (Speicher et al., 2024).

2. Principal technical realizations

One realization is architectural invariance by construction. The invariant integration layer is inserted after an equivariant convolutional backbone and before fully connected layers. It performs channel-wise invariant integration over learned feature maps using monomials and group averaging, thereby enforcing a complete feature space invariant to finite transformation groups. In the reported rotation-invariant instantiation, integration is performed over 8 sampled rotation angles using bilinear interpolation (Rath et al., 2020).

A second realization is objective-level invariance learning. Augerino learns the extent of invariance by optimizing a distribution over transformations jointly with the network, recovering full, partial, or absent invariance depending on the task (Benton et al., 2020). The Laplace-based approach similarly learns transformation parameters by optimizing a differentiable approximation to the marginal likelihood, with invariance treated as a property of the prior over functions rather than as an external augmentation heuristic (Immer et al., 2022, Wilk et al., 2018).

A third realization is representation conditioning by explicit invariance descriptors. Amortised invariance learning parameterizes the encoder as $T$2, where $T$3 controls which augmentations should be treated as invariant. Pre-training samples many binary descriptors $T$4, and downstream tasks learn both a linear head and a task-specific continuous invariance vector $T$5 by gradient descent, avoiding task-specific re-pretraining (Chavhan et al., 2023).

A fourth realization is training-geometry invariance. ND++ applies a feature transform

$T$6

where $T$7 uses local statistics to enforce scale invariance and $T$8 uses global batch covariance to approximate $T$9, yielding a GL(n)-invariant training algorithm in the linear case. If $g(Tx) = g(x).$0, then gradient descent with learning rate $g(Tx) = g(x).$1 reaches the optimum in one step for the quadratic objective, which motivates the claim that whitening aligns gradient descent with the underlying basis-free optimum (Ye et al., 2021).

A fifth realization is learning non-parametric invariances from data itself. PRC-NPTN uses permanent random connectomes: randomly initialized but fixed channel-pooling supports over filter responses. The model then learns invariance to complex, non-parametric transformations through max pooling over these random subsets, without requiring a predefined transformation group (Pal et al., 2019).

3. Staged pipelines and hierarchical organization

Many papers instantiate the principle not as a single operation but as a staged pipeline. EqInv is explicit: first learn an equivariant representation by contrastive self-supervision, then infer environments from feature similarity, then impose class-wise IRM or REx on a masked feature subspace while freezing the backbone. The overall objective combines standard cross-entropy with class-wise invariance regularizers, and the frozen $g(Tx) = g(x).$2 is crucial because equivariance aims to preserve semantic variation while invariance aims to discard environment-specific variation (Wang et al., 2022).

The same staged structure appears in the convolutional invariant integration model. A backbone such as HCNN or SFCNN first builds equivariant features; the invariant integration layer then constructs invariant descriptors in the learned feature space; the classifier operates only after this transition from equivariance to invariance (Rath et al., 2020). The authors explicitly report that this is particularly beneficial when the learned feature representation is not exactly equivariant, for example due to sampling effects (Rath et al., 2020).

Self-supervised pre-training also exposes a hierarchy of invariances. E-DIY argues that pure image-level invariance is insufficient because it can produce region-level collapse. It therefore couples region-level invariance across views with region-level diversity within an image, using teacher-guided most-similar and most-dissimilar regions. This suggests that an invariance-oriented pre-training pipeline must preserve discrimination among non-corresponding regions while aligning corresponding ones (Wei et al., 2021).

The 3D literature likewise organizes invariance hierarchically. DPCo and related methods combine local dense correspondences with global instance discrimination, and compare perspective-invariance, modality-invariance, and format-invariance within a single contrastive framework. Their strongest results come from depth–point or RGB–point couplings, indicating that 2D–3D invariance is more useful than purely 3D–3D format invariance for downstream 3D detection (Li et al., 2022).

Foundation-model tuning introduces a further staged variant. Omniview-Tuning keeps the core VLP model largely frozen, adds LoRA and VIFormer on the visual side, preserves the original image–text contrastive objective, and augments it with a cross-viewpoint alignment loss optimized in a minimax-like way over outlier viewpoints. This is a parameter-efficient invariance-refinement stage on top of broad pre-training (Ruan et al., 2024).

4. Empirical profile across domains

The empirical record is heterogeneous but consistent on one point: when the induced invariances match the task, transfer and robustness usually improve.

On Rotated-MNIST, the invariant integration layer improves both HCNN and SFCNN baselines. Mean test error over ten runs changes from 1.730 to 1.548 for HCNN without augmentation, from 1.606 to 1.377 for HCNN with augmentation, and from 0.880 to 0.799 for SFCNN without augmentation. With augmentation, SFCNN without IIL is slightly better than II-SFCNN, 0.714 versus 0.722, but the gap is small (Rath et al., 2020).

In insufficient-data regimes, EqInv reports large gains. On VIPriors-50 test, MoCo-v2 improves from 46.98% to 52.09% with EqInv, SimSiam from 40.75% to 50.36%, and IP-IRM SSL from 48.90% to 52.27%. On VIPriors-10 test, MoCo-v2 improves from 16.97% to 25.20%, SimSiam from 6.68% to 23.33%, and IP-IRM SSL from 27.88% to 30.36%. On NICO test, IP-IRM SSL improves from 60.26% to 64.14%. These gains grow as data becomes more scarce (Wang et al., 2022).

Translation invariance provides a direct pre-training example. Standard CNNs are not architecturally translation-invariant, but pre-training on ImageNet or on fully-translated datasets can induce strong representation-level invariance. After pre-training on fully-translated simple datasets and before fine-tuning, the Invariance Metric reaches

$g(Tx) = g(x).$3

across several displacements, indicating that translated versions of the same object are substantially more similar than different objects (Biscione et al., 2021). The same line of work reports that invariance can then be disrupted by fine-tuning on one-location datasets due to catastrophic interference (Biscione et al., 2021).

In vision-language pre-training, Omniview-Tuning produces large viewpoint gains while keeping original performance largely intact. OVT-OpenCLIP ViT-B/32 reports an average viewpoint-OOD Top-1 improvement of +9.6%, OVT-OpenCLIP ViT-B/16 +10.2%, and OVT-OpenCLIP ViT-L/14 +8.9%. On 2D-OOD benchmarks the average sacrifice is small: $g(Tx) = g(x).$4, $g(Tx) = g(x).$5, and $g(Tx) = g(x).$6, respectively (Ruan et al., 2024).

In 3D self-supervision, DPCo obtains 59.8/35.6 AP25/AP50 on SUN RGB-D and 64.2/41.5 on ScanNet, compared with 58.4/33.3 and 60.0/37.6 from scratch, and it surpasses PointContrast, PVCo, and IPCo on ScanNet AP50 within the reported setting (Li et al., 2022). In dense visual transfer, E-DIY improves COCO detection with Mask R-CNN R50-C4 1× from AP$g(Tx) = g(x).$7 for BYOL to 40.7, a +2.1% improvement reported in the abstract, and on VOC detection improves AP from 53.1 to 58.4 and AP$g(Tx) = g(x).$8 from 58.6 to 65.7 (Wei et al., 2021).

These results are complemented by a mechanistic study of fine-tuning in ViTs. STIR-based analyses show that pretraining induces transferable invariances in shallow layers and that deeper pretrained invariances are compressed towards shallower layers during finetuning (Merlin et al., 2023). This suggests that the pre-training benefit is not merely feature reuse in the CKA sense, but reuse and redistribution of invariance sets across depth.

5. Misconceptions, limits, and failure modes

A common misconception is that standard CNNs are architecturally invariant to translation. The translation studies reject this for most tested architectures; apart from DenseNet-121, the reported models fail to recognize objects at novel locations unless the training environment itself teaches translation invariance (Biscione et al., 2021). Learned invariance is therefore an environmental or pre-training effect, not an automatic consequence of convolution and pooling.

A second misconception is that more invariance is always better. Transfer experiments show the opposite when invariance suppresses downstream-relevant features. In the CIFAR-plus-object setting, a model pretrained on $g(Tx) = g(x).$9 reaches $$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$0 on the same CIFAR target but only $$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$1 on $$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$2, because the representation becomes insensitive to the pasted objects when those objects are irrelevant during pre-training (Speicher et al., 2024). The same paper shows that a superset of required invariances is usually harmless only if the extra invariances do not remove task-relevant information (Speicher et al., 2024).

A third limit concerns collapse and destructive interactions. E-DIY reports that using R-IEM alone leads to model collapse, with AP$$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$3, and that R-DEM alone also fails; only the combination of BYOL, R-DEM, and R-IEM produces strong performance (Wei et al., 2021). In 3D pre-training, depth-only DDCo is harmful: AP25/AP50 for 2.5D VoteNet on SUN RGB-D drop from 60.8/36.9 to 56.0/31.2, indicating that the wrong invariance can remove geometry-relevant structure (Li et al., 2022).

A fourth limit is that fine-tuning may overwrite useful invariances. Translation-invariance studies attribute post-pretraining degradation to catastrophic forgetting or interference (Biscione et al., 2021). The ViT study formalizes this as layer-wise forgetting and learning of invariances, and shows that representational similarity can remain high while forgetting and learning of invariances are substantial (Merlin et al., 2023). EqInv likewise finds that freezing the equivariant backbone is crucial; updating $$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$4 under strong invariance penalty can sharply degrade performance (Wang et al., 2022).

Finally, evidence-based invariance learning remains computationally demanding. The differentiable Laplace approach is restricted by the chosen invariance families, incurs runtime that grows linearly with the number of augmentation samples $$f(g \cdot u) = g \cdot f(u) \quad \forall g \in \mathcal{G},\, u \in \mathcal{U}.$$5, and reports that diagonal GGN or empirical Fisher approximations perform poorly for invariance learning compared with KFAC-based alternatives (Immer et al., 2022). This suggests that principled invariance selection is still easier to formulate than to scale.

6. Synthesis and research directions

Taken together, the literature supports a recurrent pattern. First, identify candidate nuisance transformations or environments. Second, learn or impose an equivariant or information-rich representation. Third, convert this representation into an invariant one by architectural integration, contrastive alignment, risk invariance, learned augmentation distributions, or parameter-efficient alignment. Fourth, protect useful invariances during adaptation, because downstream fine-tuning can compress, relocate, or erase them (Rath et al., 2020, Wang et al., 2022, Merlin et al., 2023, Ruan et al., 2024).

This suggests a compact formulation of the Invariance Pre-training Principle: pre-training should endow the model with an invariance structure aligned with the transformations under which downstream labels should remain stable, while preserving sensitivity to semantically relevant variation. In the surveyed work, this structure may be complete and group-theoretic, as in invariant integration; learned and task-adaptive, as in Augerino or amortised invariance learning; Bayesian and prior-driven, as in marginal-likelihood or Laplace-based methods; hierarchical and layer-wise, as in STIR analyses; or injected after broad pre-training, as in Omniview-Tuning (Rath et al., 2020, Benton et al., 2020, Chavhan et al., 2023, Wilk et al., 2018, Immer et al., 2022, Merlin et al., 2023, Ruan et al., 2024).

Open directions are correspondingly diverse. The invariant integration work suggests extending invariant integration to other transformations such as scale and exploring hand-designed monomials (Rath et al., 2020). EqInv points to improving class-muted clustering and constructing more unique and fine-grained environments (Wang et al., 2022). The ViT analysis raises explicit STIR-based regularization of finetuning as an open question (Merlin et al., 2023). Amortised invariance learning asks how to model more complex invariance structures and better conditioning mechanisms (Chavhan et al., 2023). Omniview-Tuning highlights the need for large multi-condition datasets beyond viewpoint (Ruan et al., 2024). Laplace-based invariance learning identifies richer transformation parameterizations and better curvature approximations as central unresolved issues (Immer et al., 2022).

The common implication is not that invariance should always be maximized, but that it should be treated as a first-class pre-training object. The strongest results arise when invariance is explicitly specified, learned, measured, and preserved as part of the representation design itself (Speicher et al., 2024).