---
title: Test-Time Canonicalization for Robust Inference
url: https://www.emergentmind.com/topics/test-time-canonicalization
type: topic
---

# Test-Time Canonicalization for Robust Inference

Searching arXiv for recent papers on test-time canonicalization and closely related canonicalization methods.
Found relevant papers, including "Test-Time Canonicalization by Foundation Models for Robust Perception" [2507.10375], "Zero-Shot Test-Time Canonicalization using Out-of-Distribution Scoring" [2606.24178], "Improved Canonicalization for Model Agnostic Equivariance" [2405.14089], "Equivariance with Learned Canonicalization Functions" [2211.06489], and domain-specific canonicalization papers in NRSfM, language modeling, motion retargeting, and generative modeling.
Test-time canonicalization is the inference-time practice of mapping an observed input, an intermediate representation, or a generated sample to a canonical representative of its symmetry class before applying a downstream predictor or after sampling from a canonical slice. Across recent work, the object being canonicalized varies: images may be rotated or relit to a visually typical view, token prefixes may be restricted to canonical tokenizations, point clouds and skeletons may be mapped to canonical frames, predicted 3D sequences may be aligned to remove rigid ambiguity, and generative models may sample in canonical coordinates and only afterward restore symmetry by randomization over the group [2507.10375], [2606.24178], [2506.06446], [2602.15022]. The common objective is to factor out nuisance variation without redesigning the main predictor around strict equivariance or retraining it on exhaustive augmentations.

## 1. Formal definition and mathematical setting

A standard formulation treats canonicalization as orbit selection under a group action. If a group \(G\) acts on inputs \(x\in\mathcal X\), a canonicalizer seeks a representative from the orbit \(Gx\) that is consistent across transformed versions of the same underlying object. In architecture-agnostic equivariant adaptation, the canonicalized predictor is written as
\[
f(x) = c'(x)\, p\!\left(c(x)^{-1}x\right),
\]
where \(p\) is the downstream predictor, \(c(x)\) is the estimated canonicalizing transformation, and \(c'(x)\) is the corresponding output-space action [2405.14089]. In learned canonicalization for equivariant learning, the same idea appears as
\[
\phi(x)=h'(x)\,f\!\left(h(x)^{-1}x\right),
\]
with \(h\) a canonicalization function and \(h'\) the output-side inverse transport [2211.06489].

A complementary formulation defines canonicalization by energy minimization over transformations:
\[
h(x)=\arg\min_{t \in \mathcal{T}} E(t(x)),
\]
which underlies test-time search methods that score transformed candidates and select the minimum-energy representative [2507.10375]. A closely related zero-shot formulation writes
\[
h(x) = (g^*)^{-1} \cdot x \quad \text{with} \quad g^* = \arg\min_{g \in G} E(g^{-1} \cdot x),
\]
and interprets \(E\) as an out-of-distribution score, so that canonicalization becomes OOD minimization over inverse transformations [2606.24178].

Not all canonicalization is single-valued. A canonicalization perspective on invariant and equivariant learning defines a set-valued map
\[
\mathcal C\colon V\to 2^V\backslash \varnothing
\]
satisfying
\[
\mathcal C(\rho_1(g)X)=\mathcal C(X),
\]
with invariant prediction obtained by canonical averaging,
\[
\varPhi_\mathrm{CA}(X;\mathcal C,\phi) = \frac1{|\mathcal C(X)|}\sum_{X_0\in \mathcal C(X)}\phi(X_0).
\]
This accommodates cases where symmetry or stabilizers prevent a unique canonical representative [2405.18378].

The same structural idea extends beyond geometric groups. In autoregressive language modeling, canonicalization is defined relative to a tokenizer: a next token \(t\) is admissible exactly when the extended prefix \(s\shortmid t\) remains canonical, and canonical decoding masks all continuations that violate this condition [2506.06446].

## 2. Inference-time algorithmic patterns

Recent work realizes test-time canonicalization through several distinct mechanisms. Some methods run a small learned canonicalizer once and then transform the input into a canonical frame. Others search over transformed candidates, using either a learned score, a foundation-model energy, or an OOD score. Language-model work canonicalizes incrementally at every decoding step. Generative modeling often inverts the chronology: it trains on canonicalized data, generates directly in canonical space, and restores symmetry only after sampling [2211.06489], [2405.14089], [2507.10375], [2606.24178], [2506.06446], [2602.15022].

| Setting | Canonicalized object | Test-time mechanism |
|---|---|---|
| Architecture-agnostic equivariance | Input sample | Predict \(c(x)\), apply \(c(x)^{-1}x\), run backbone once |
| Search-based perception | Transformed image candidates | Enumerate or optimize over \(t(x)\), choose minimum-energy view |
| OOD-based canonicalization | Inverse-transformed input | Minimize OOD score over \(g^{-1}\!\cdot x\) |
| Canonical autoregressive generation | Token prefix | Mask next tokens that make the prefix non-canonical |
| Canonical diffusion / flow | Generated sample on slice | Sample in canonical space, optionally project during sampling, randomize over group at end |

Direct learned canonicalizers are the simplest operationally. In the main learned-direct variant of canonicalization for equivariant learning, inference is one extra forward pass through a small canonicalization module plus one transformation of the input, rather than repeated evaluation over group elements [2211.06489]. By contrast, EquiOptAdapt explicitly evaluates all transformed candidates for a discrete group using a non-equivariant scorer \(s_\theta\), but still runs the large downstream model only once on the chosen canonicalized input [2405.14089].

Search-based perception generalizes this pattern. FoCal adopts a “Vary and Rank” pipeline: generate candidate transformations, score each candidate with a combined CLIP and diffusion energy,
\[
E_{\textnormal{FoCal}(t(\boldsymbol{x})) = \gamma_1 E_{\text{CLIP}(t(\boldsymbol{x})) + \gamma_2 E_{\text{diff}(t(\boldsymbol{x})),
\]
then choose
\[
t^* = \argmin_{t \in \mathcal{T}} E_{\textnormal{FoCal}(t(\boldsymbol{x})).
\]
For continuous spaces it uses Bayesian Optimization rather than exhaustive enumeration [2507.10375]. OOD-based canonicalization retains the same outer optimization form but broadens the energy design space by letting any OOD score serve as \(E\), and adds selection and acceptance gates so already aligned inputs can bypass canonicalization [2606.24178].

## 3. Relation to invariance, equivariance, and symmetrization

Canonicalization is often presented as an alternative to equivariant architecture design rather than a replacement for symmetry as such. If the canonicalization function is itself appropriately equivariant, then the transformed predictor becomes equivariant for any downstream model \(p\), which is precisely why canonicalization is attractive for adapting pretrained models at inference time [2405.14089]. A corresponding universality result states that with a continuous \(G\)-equivariant canonicalizer and a universal backbone, the overall model is universal for continuous \(G\)-equivariant functions [2211.06489].

A more structural result is that frame averaging and canonicalization are equivalent at the level of invariant averaging. For any frame \(\mathcal F\), there exists an orbit canonicalization \(\mathcal C_\mathcal F\) such that
\[
\varPhi_\mathrm{FA}(X;\mathcal F,\phi)=\varPhi_\mathrm{CA}(X;\mathcal C_\mathcal F,\phi),
\]
and the induced canonicalization removes stabilizer-induced duplication by reducing the effective representation count from \(|\mathcal F(X)|\) to \(|\mathcal F(X)|/|G_X|\) [2405.18378]. This reframes many “frame” methods as canonicalization procedures implemented in group space rather than input space.

Canonicalization also differs from augmentation and test-time augmentation. Augmentation asks the model to absorb nuisance variability during training; TTA averages predictions over multiple transformed copies at inference. Canonicalization instead attempts to identify one representative before prediction. The literature repeatedly treats this as the main practical advantage: the downstream model can remain non-equivariant and often needs to be evaluated only once at test time [2405.14089], [2507.10375], [2510.08178].

The distinction between training-time and test-time use is nevertheless important. Some systems use canonicalization machinery only as training scaffolding. DRACO uses C3DPO-derived sparse canonicalization during training to teach a dense NOCS predictor, but at inference the network directly predicts dense canonical geometry from RGB without explicitly rerunning C3DPO [2011.12912]. In deep NRSfM, the GPA layer is explicitly a training regularization module and “can be left out of the computation during testing,” even though the underlying alignment procedure is naturally interpretable as a per-sequence inference-time canonicalizer [2412.07230].

## 4. Representative domain instantiations

In robust visual perception, test-time canonicalization is used to move images toward visually typical states. FoCal canonicalizes images by optimizing over 2D rotations, 3D viewpoints, illumination shifts, and day-night variations using foundation-model priors. On Objaverse-LVIS it improves accuracy on the hardest viewpoints from \(12.0\%\) to \(62.0\%\), and on rotated COCO segmentation it matches PRLC’s mAP while improving pose accuracy [2507.10375]. A related zero-shot line treats transformed inputs as OOD relative to the training pose distribution and finds that distance-based OOD scores, especially kNN- and prototype-style methods, outperform prior logit-based energies, with random search plus local refinement performing best overall [2606.24178].

For adapting pretrained models under discrete transformations, EquiOptAdapt canonicalizes by scoring all inverse-transformed candidates \(\rho(g)^{-1}x\) with a non-equivariant network \(s_\theta\), selecting
\[
g^* \in \arg\min_{g\in G} v_R \cdot s_\theta(\rho(g)^{-1}x),
\]
and then running the downstream model once on \(x_{\mathrm{can}}=\rho(g^*)^{-1}x\). On rotated-image benchmarks it closes the gap between standard accuracy and transformed-test accuracy, and the paper reports that the canonicalization process is up to \(2\times\) faster than EquiAdapt [2405.14089].

In 3D perception and geometry, test-time canonicalization often means explicit estimation of pose, translation, and scale or direct prediction of canonical coordinates. DRACO predicts dense object-centric depth and a dense NOCS map from one or more RGB images at inference, yielding a canonical category-level coordinate map without requiring keypoints or camera poses at test time [2011.12912]. ShapeMatcher takes a partial target point cloud \(S_{tgt}\), predicts
\[
R_{tgt}, T_{tgt}, F_{tgt}^{*} = \mathbf{VN\mbox{-}MLP}(S_{tgt}), \qquad s_{tgt}, F_{tgt} = normalize(F_{tgt}^{*}),
\]
and can explicitly form a canonicalized shape
\[
S^{c}_{tgt} = s_{tgt}R_{tgt}S_{tgt} + T_{tgt},
\]
which then supports segmentation, retrieval, and deformation [2311.11106].

Motion and sequence modeling provide a different interpretation. MoCaNet canonicalizes 2D skeleton sequences along two axes: structure canonicalization removes body-shape variation and view canonicalization removes camera pose variation, enabling retargeting from monocular 2D video to 3D motion without 3D labels [2112.10082]. Deep NRSfM canonicalizes predicted 3D shape sequences by generalized Procrustes alignment,
\[
\left\{\hat{\mathbf R}_i\right\}_{i=1}^{F} = \arg\min_{\mathbf R_i} \sum_{i=1}^{F} \left\| \mathbf R_i \tilde{\mathbf S}_i - \bar{\mathbf S} \right\|_F^2,
\]
so that low-rank sequence regularization acts in a coherent shared frame rather than being corrupted by framewise rotational ambiguity [2412.07230].

In language modeling, the canonicalization target is the token sequence rather than a geometric pose. Canonical autoregressive generation observes that a tokenizer presents only one canonical tokenization during training, yet a model can still generate non-canonical token sequences at inference. Its central theorem states that for BPE, if a prefix \(s\) is non-canonical, then \(s\shortmid t\) is also non-canonical for any token \(t\). Canonical sampling therefore masks every token that would make the current prefix non-canonical and renormalizes over the rest, guaranteeing that the full generated sequence is canonical [2506.06446].

In symmetry-aware generative modeling, canonicalization often moves from input preprocessing to generative gauge fixing. Learned canonicalization for diffusion defines
\[
c(x)=h(x)^{-1}x,
\]
trains a non-equivariant denoiser on \(c(x)\), and samples directly in canonical space [2501.07773]. Canonical diffusion and CanonFlow make this explicit at test time: sample from a canonical slice prior, integrate reverse dynamics in canonical space, optionally reproject intermediate states with projected canonical sampling, and finally recover the invariant distribution by sampling a random group element \(g\) from Haar measure and outputting \(g\tilde Z\) [2602.15022].

## 5. Empirical advantages and recurring benefits

The principal advantage claimed across the literature is symmetry handling without redesigning the main predictor. FoCal argues that transform-specific retraining is not necessary for robust perception and uses foundation-model priors to canonicalize inputs at test time without re-training or architectural changes [2507.10375]. OOD-based canonicalization is similarly designed for a fixed pretrained classifier, with robustness added post hoc by optimizing over transformations and optionally gating the procedure when the input already appears in-distribution [2606.24178].

A second benefit is computational asymmetry: the canonicalizer can be much smaller or cheaper than the predictor. Learned canonicalization functions are proposed specifically as a way to obtain exact equivariance with ordinary backbones while paying the symmetry cost once up front rather than throughout the network [2211.06489]. In point-cloud and \(N\)-body experiments, the canonicalizer is deliberately shallow and inference overhead is reported as negligible relative to the prediction network [2211.06489]. EquiOptAdapt makes the same design choice by evaluating multiple transformed candidates only through the small canonicalizer rather than through the large pretrained model [2405.14089].

A third benefit is statistical or geometric simplification. In NRSfM, canonicalization is necessary because applying a nuclear norm directly to arbitrarily rotated sequences would inflate rank and spoil the subspace prior; alignment removes rigid inter-frame motion first [2412.07230]. In canonical diffusion, the theoretical advantage is framed as elimination of symmetry-induced mixture complexity in diffusion scores and removal of a nonnegative symmetry-ambiguity term in flow-matching conditional variance, which in turn improves few-step generation quality [2602.15022].

Some works also provide formal distributional guarantees. Canonical autoregressive generation proves that the token-sequence distribution induced by canonical sampling is strictly closer in KL divergence to the true training distribution than standard sampling under the theorem’s assumptions,
\[
\mathrm{KL}(p,\tilde d) < \mathrm{KL}(p,d),
\]
because non-canonical token mass is removed and redistributed onto canonical continuations [2506.06446].

## 6. Ambiguity, failure modes, and scope limitations

A central limitation is non-uniqueness. Inputs with nontrivial stabilizers may not admit a unique canonical form. The canonicalization literature therefore distinguishes canonicalizable from uncanonicalizable inputs and often falls back to a set of representatives or averaging when uniqueness is impossible [2405.18378]. Learned canonicalization functions face a related continuity problem: small changes in nearly symmetric inputs can produce large changes in the selected canonical pose, and the literature explicitly notes that smoothness is not guaranteed [2211.06489].

Search-based methods add their own caveats. FoCal’s exact invariance intuition assumes invertible transformations, whereas 3D viewpoint changes are not invertible in image space, so viewpoint canonicalization is best understood as approximate canonicalization by search over synthesized views [2507.10375]. The same method is computationally heavy, with runtime dominated in some settings by transformation generation and energy evaluation rather than the downstream predictor [2507.10375]. OOD-based canonicalization shows that always-on canonicalization can hurt clean accuracy because canonicalizing an already aligned input may perturb discriminative features; this motivates the paper’s selection and acceptance gates [2606.24178].

Training assumptions are equally important. Canonicalization-prior training typically presumes that the dataset already shares a global canonical mode. Robust Canonicalization through Bootstrapped Data Re-Alignment argues that real-world datasets violate this assumption, so the learned canonicalizer becomes brittle; its remedy is an iterative training-time re-alignment procedure that contracts the Fréchet variance of the pose distribution under mild assumptions for compact groups [2510.08178]. This suggests that test-time canonicalization quality can be limited as much by training-set alignment as by inference algorithm design.

Several papers also delimit the scope of their guarantees. Canonical autoregressive generation proves improvement in token space, not string-space quality, and explicitly does not claim a string-space KL inequality [2506.06446]. The deep NRSfM paper does not actually deploy GPA at test time in its final evaluation pipeline, even though the alignment operator is defined per sequence and could be applied to a test sequence independently [2412.07230]. In symmetry-aware generation, canonicalization may intentionally discard nuisance degrees of freedom and return only canonical-pose samples unless a random group action is applied afterward [2501.07773], [2602.15022].

## 7. Conceptual synthesis and current directions

Across domains, test-time canonicalization now spans three recurring templates. The first is **canonicalize–predict**: estimate a pose, ordering, token constraint, or orbit representative and then run an ordinary model in that canonical frame [2405.14089], [2506.06446]. The second is **search–rank–predict**: generate transformed candidates, score them with a learned or pretrained energy, and select the most typical or least OOD representative before inference [2507.10375], [2606.24178]. The third is **canonicalize–generate–randomize**: learn a model on a canonical slice and restore symmetry only after sampling [2501.07773], [2602.15022].

This suggests a broader conceptual shift. Canonicalization is no longer confined to rigid geometric preprocessing. It has become a general inference-time strategy for quotienting nuisance structure, whether the nuisance is camera pose, body morphology, frame ambiguity, tokenization multiplicity, or permutation–rotation redundancy in molecule generation. The strongest formal developments treat canonicalization as a complete view of frame design, a universal route to invariant and equivariant learning, or an exact factorization device for invariant generative distributions [2405.18378], [2211.06489], [2602.15022].

The open questions identified in the literature are correspondingly broad. Search-based perception highlights automatic transformation-family selection and more efficient optimization as unresolved practical issues [2507.10375]. Architecture-agnostic image canonicalization currently focuses on discrete groups and explicitly leaves continuous rotations as future work [2405.14089]. Language-model work identifies practical string-level evaluation as future work [2506.06446]. Generative modeling raises unresolved questions about discontinuity, stabilizers, and the choice of canonical slice [2501.07773], [2602.15022]. What is already clear, however, is that test-time canonicalization has become a unifying operational principle: rather than forcing the predictor to internalize all symmetry, it externalizes nuisance removal into an inference-time canonicalization step and lets the main model operate in a more stable coordinate system.

Source: https://www.emergentmind.com/topics/test-time-canonicalization