Deep Equilibrium Canonicalizer (DEC) Explained

• DEC is a deep equilibrium approach that canonicalizes latent variables to improve local scale equivariance in image-based tasks.
• It maps nonlinear latent coordinates to canonical parameters via fixed-point computation, using implicit differentiation for efficient inference.
• Empirical results on benchmarks like ImageNet and MNIST show enhanced accuracy and reduced scale variance in architectures such as ViT, DeiT, Swin, and BEiT.

Deep Equilibrium Canonicalizer (DEC) denotes a use of deep equilibrium models in which equilibrium computation is the mechanism of canonicalization. In a foundational statistical sense, a DEQ can be interpreted as a canonicalizer for continuous latent-variable models: latent coordinates are mapped nonlinearly to exponential-family canonical parameters, and the equilibrium state solves maximum a-posteriori (MAP) inference for the latent representation (Tsuchida et al., 2022). In later computer-vision work, DEC is introduced as a plug-in module for improving local scale equivariance by replacing per-example optimization-based canonicalization with a fixed-point computation, and by inserting that computation into latent stages of pretrained architectures such as ViT, DeiT, Swin, and BEiT (Rahman et al., 19 Aug 2025).

1. Definition through canonicalization and local scale equivariance

The canonicalization viewpoint starts from a standard formulation in which an input is transformed into a canonical form before task processing. The paper on local scale equivariance states this as

$$h(f) \triangleq \text{arg min}_{g \in G} E( g^{-1}(f)),$$

where $E$ is a learnable energy function and $h(f)$ returns the canonicalizing transformation (Rahman et al., 19 Aug 2025). In this formulation, canonicalization is not merely feature normalization; it is an input-dependent estimation problem over a transformation group.

The specific problem addressed in DEC is local scale variation. The motivating observation is that objects of the same class can have different sizes, that perceived size also depends on distance from the camera, and that these changes are local rather than globally uniform. The paper distinguishes this setting from global scaling, where the entire image is resized uniformly. Standard architectures are described as struggling because they are built around fixed receptive fields or fixed tokenization or patching patterns, and because they are typically trained on global augmentations rather than transformations that independently distort different regions (Rahman et al., 19 Aug 2025).

A central technical point is that real-world local scaling does not naturally form a group in the strict mathematical sense, since it is not generally invertible when objects occlude one another. DEC therefore approximates local scaling with a monotone scaling group. For a 1D function $f$, the action is written as

$(f;l)(x) \triangleq f(l^{-1}(x)),$

where $l:[0,1]\to[0,1]$ is continuous, strictly increasing, and invertible, and the group law is composition,

$$\left(l_1 \cdot l_2 \right) (x) =(l_1 \circ l_2)(x) = l_1(l_2(x)).$$

Within this group-theoretic approximation, equivariance and invariance are defined in the usual way: $$\mathcal{N}((f;g)) = (\mathcal{N}(f);g), \qquad \mathcal{N}((f;g)) = \mathcal{N}(f).$$ The canonicalization condition is then

$h((f;g)) = g \cdot h(f), \forall g \in G.$

This makes DEC a transformation-estimation mechanism whose purpose is to stabilize downstream representations under local monotone scaling (Rahman et al., 19 Aug 2025).

2. Statistical interpretation: DEQ as a canonicalizer for latent variables

The earlier statistical treatment of DEQs provides the conceptual basis for calling such a mechanism a “canonicalizer.” That paper starts from PCA and exponential-family PCA, with observations $x$, latent variables $E$0, and transformation parameters, and generalizes the linear canonical map by replacing it with a neural network: $E$1 where $E$2 is the canonical parameter of the observation distribution (Tsuchida et al., 2022). The resulting model is

$E$3

with $E$4 an exponential-family likelihood whose canonical parameter is given by $E$5.

The associated MAP estimate is

$E$6

or equivalently the minimizer of the corresponding negative log-posterior energy. For exponential-family observations, the energy has the standard form

$E$7

The paper’s central result is that the DEQ equilibrium condition corresponds to the first-order optimality condition for this MAP problem. When the latent representation is defined implicitly by a fixed point such as

$E$8

solving the equilibrium equation is equivalent to solving the stationarity condition

$E$9

for the latent posterior objective (Tsuchida et al., 2022).

In this sense, a DEQ is both a canonicalizer and an implicit optimizer. It maps latent variables to canonical parameters of an observation model, and its equilibrium computation returns the MAP latent state consistent with the observations. The same paper further states that activation functions, dropout, and layer structure correspond to statistical assumptions about the observations and latent prior, and that in hierarchical settings individual neurons can be interpreted as nodes in a deep graphical model. This gives DEC a precise probabilistic semantics rather than treating it only as an implicit neural network (Tsuchida et al., 2022).

3. The local-scale DEC mechanism

The local-scale DEC paper turns optimization-based canonicalization into a DEQ problem by using the monotone-scale parameter $h(f)$0 as the equilibrium hidden state. Rather than explicitly unrolling many optimization steps for every example, DEC learns an update rule whose equilibrium corresponds to a stationary point of the canonicalization energy (Rahman et al., 19 Aug 2025). The paper states that at each iteration the current parameter $h(f)$1 is used to inversely scale the input feature or image, the energy is evaluated, and the parameter is updated; the process repeats until convergence to an equilibrium $h(f)$2, which satisfies the usual DEQ fixed-point condition.

This construction is implemented with standard DEQ machinery. The equilibrium can be solved using Anderson acceleration, and backpropagation is done via implicit differentiation (Rahman et al., 19 Aug 2025). That choice is directly aligned with the broader DEQ interpretation in which the equilibrium computation stands in for iterative inference or optimization.

A key design choice is latent canonicalization rather than input-only canonicalization. The paper motivates this by noting that local scaling is only approximated by the monotone group, so canonicalizing in latent space better matches the learned representation and avoids the brittleness of forcing the raw image into a fully canonical form (Rahman et al., 19 Aug 2025). Operationally, each latent feature is first inverse-warped by the learned monotone scaling, the layer is applied, and then the forward warp is applied back so that the module is equivariant. For invariance, the output is canonicalized and passed onward without the final reapplication of the transformation.

For 2D inputs, the transformation is parameterized as a piecewise-linear monotone warp over a grid, with image action

$h(f)$3

where $h(f)$4. The appendix further specifies that the 2D formulation uses locally monotone functions whose Jacobians are symmetric positive definite and commute, ensuring a valid group under composition. As an implementation detail, the DEC module is modeled as a 2-layer CNN with 64 and 128 channels, followed by adaptive pooling to output the monotone-scaling parameters, and it is designed to be inserted into pretrained architectures rather than requiring a new equivariant model to be trained from scratch (Rahman et al., 19 Aug 2025).

4. Iterative refinement, equilibrium dynamics, and relation to DEQ-style spatial models

A related line of work connects DEQs to Neural Cellular Automata (NCA) and provides a useful dynamical interpretation for DEC-like mechanisms. That essay characterizes NCA as repeatedly applying a spatially invariant, locally defined transition to every cell, and DEQs as implicit models in which the forward pass is a root-finding process for an equilibrium function and the backward pass uses implicit differentiation (Jia, 7 Jan 2025). Its central claim is that the paradigms are structurally similar: NCA applies repeated local rules until a stable pattern emerges, whereas DEQ solves for the fixed point of a recurrent or local update.

The concrete DEQ-style convolutional model in that paper consists of one explicit convolutional layer, one convolutional DEQ layer, global average pooling, and a two-layer MLP decoder. Its equilibrium hidden state $h(f)$5 is defined implicitly by a residual-style update involving convolutions, concatenation of input features with the hidden state, and a $h(f)$6 nonlinearity (Jia, 7 Jan 2025). When the implicit layer is repeatedly applied, digit-like structures emerge from noise, different regions settle into distinct stable states, and the evolution resembles self-organizing NCA dynamics.

Although that essay does not introduce a method explicitly called Deep Equilibrium Canonicalizer, it identifies several ingredients that are directly relevant to DEC: weight tying, iterative refinement, convergence to a stable hidden state, solver-based training, and the possibility of treating a spatial DEQ as the limit of repeated local updates (Jia, 7 Jan 2025). This suggests that DEC-style canonicalization can be understood not only as an energy-based fixed-point estimator, but also as a self-organizing equilibrium process in which local evidence is iteratively refined into a stable representation.

5. Empirical performance and evaluation protocol

The empirical evaluation of DEC is centered on local scale consistency and local scale invariance or equivariance. The paper constructs synthetic datasets with controlled local scaling, including a locally scaled object segmentation dataset from rendered Google Scanned Objects on random HDRI backgrounds, a locally scaled MNIST dataset of 3-digit images with independently scaled digits, and a locally scaled ImageNet benchmark in which only the target object is rescaled while the background is kept unchanged (Rahman et al., 19 Aug 2025).

For segmentation, the reported metrics are mIoU and EquE, a local scale equivariance error. For MNIST and ImageNet classification, the metric is InvE, a local scale invariance error; in both cases, lower values indicate more stable behavior under local scale changes (Rahman et al., 19 Aug 2025). The baselines include training with random monotone scaling augmentation, vanilla optimization-based canonicalization using an energy model and discrete transformation bins, and variants further fine-tuned with an equivariance or invariance loss.

On ImageNet, DEC is evaluated on ViT, DeiT, Swin, and BEiT. The reported results are as follows.

Backbone	Baseline acc / InvE	DEC acc / InvE
ViT	80.15 / 8.16	80.36 / 8.10
DeiT	69.06 / 11.12	69.27 / 11.08
Swin	77.96 / 8.92	78.32 / 8.82
BEiT	84.66 / 6.27	85.08 / 6.24

The per-scale table for Swin shows improvement at every local scale from 0.7 to 1.3, and the standard deviation of accuracy decreases from 1.65 to 1.49, indicating more consistent predictions across scales (Rahman et al., 19 Aug 2025).

On locally scaled MNIST, DEC is evaluated on ResNet, ViT, DeiT, Swin, BEiT, and DINOv2, and is reported to achieve the best accuracy and lowest InvE across all six architectures. The paper highlights reductions such as 18.08 to 9.85 in InvE on ResNet and 6.30 to 1.31 on DINOv2. Averaged across architectures, DEC improves the baseline by 4.10% accuracy and $h(f)$7 in InvE. For per-scale MNIST results on Swin, DEC achieves the highest accuracy in every scale range and lowers the accuracy standard deviation from 1.91 to 1.09 (Rahman et al., 19 Aug 2025).

The semantic segmentation results show the same pattern: across ViT, Swin, and DINOv2, DEC achieves the highest mIoU and lowest EquE, and on average outperforms the best baseline by 1.17% mIoU (Rahman et al., 19 Aug 2025). The paper explicitly notes that data augmentation alone is not sufficient to guarantee equivariance, and that directly canonicalizing the input image with a discretized optimization baseline can even hurt performance. Within the paper’s framing, this is evidence that latent canonicalization with a DEQ-based amortized solver is more effective than input-only optimization.

6. Efficiency, practical scope, and terminological ambiguity

The practical motivation for DEC includes efficiency as well as equivariance. The paper reports that the optimization-based canonicalizer uses 43.30 GB of GPU memory and 0.41s per iteration, whereas DEC uses 5.75 GB and 0.19s (Rahman et al., 19 Aug 2025). For DINOv2, DEC accounts for about 24% of total inference time on a batch of 128 images. The paper also reports no drop in performance on unmodified scale-1 ImageNet images; it instead states that DEC slightly improves them. Ablations in the appendix report that increasing the number of DEC modules, increasing the number of layers per DEC module, and increasing the grid size used to parameterize local scaling generally improve performance (Rahman et al., 19 Aug 2025).

These results delimit the current scope of DEC. The method is presented as a practical adaptation mechanism for pretrained backbones under approximate local scaling, not as a claim that arbitrary real-world local deformations have been reduced to an exact group action. The emphasis on latent canonicalization reflects that limitation directly: local scaling is only approximated by the monotone group, so canonicalization is moved into representation space, where the approximation better matches the learned features (Rahman et al., 19 Aug 2025). A plausible implication is that DEC is best understood as a representation-level normalization mechanism grounded in equilibrium inference, rather than as a complete geometric solution to all scale variation.

The acronym itself is also ambiguous outside machine learning. In differential geometry and general relativity, “DEC” can denote the dominant energy condition, as in “Nonexistence of DEC spin fill-ins” (Raulot, 2022). That usage is unrelated to Deep Equilibrium Canonicalizer. Within the machine-learning literature represented here, DEC refers to equilibrium-based canonicalization: either the general statistical interpretation of DEQs as latent MAP canonicalizers (Tsuchida et al., 2022), or the concrete latent module for local scale equivariance in vision systems (Rahman et al., 19 Aug 2025).