Deformable Steerable Convolution

• Deformable steerable convolution is a framework that fuses group-structured steering with spatially adaptive sampling to improve neural network performance.
• It combines representation theory, harmonic analysis, and learned offset fields to modulate the sampling geometry in image, video, and point cloud tasks.
• The approach achieves approximate equivariance through interpolation, discretization strategies, and deformation regularization, balancing algebraic structure and adaptive filtering.

Deformable steerable convolution denotes a class of operators that aim to combine the geometric, representation-theoretic structure of steerable convolutional neural networks with spatially varying deformations, in the sense of deformable convolution or spatial transformers. In the rigid steerable setting, convolution is organized by a transformation group such as $\mathrm{SE}(d)$ or a discrete rotation/reflection group, and feature channels transform according to prescribed representations. In the deformable setting, sampling locations, kernel support, or local transformation parameters vary with position. Contemporary formulations therefore sit at the intersection of group equivariance, adaptive filtering, interpolation theory, and learned offset fields rather than within a single universally standardized architecture (Kundu et al., 21 Oct 2025).

1. Historical and conceptual lineage

Two technical lineages underlie the modern notion. The first is steerability: for a transformation group $G$, a filter family is steerable if transformed copies of a filter can be expressed through a structured basis, classically by rotated copies and, in more recent work, by representation-theoretic kernel constraints. The second is spatial adaptation: the filter, or the sampling pattern through which it is evaluated, is allowed to vary with spatial position. Efficient spatially adaptive convolution and correlation formalize this second idea through a transformation field $\Phi : \text{space} \to G$, producing extended correlation

$$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$

and extended convolution

$$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$

with efficiency obtained by expanding filters in bases adapted to the representation theory of the chosen group (Mitchel et al., 2020).

In image CNNs, deformable convolution introduced a complementary mechanism: a fixed sampling grid $\mathcal R$ is augmented with learned offsets $\Delta \mathbf p_n$, so that

$$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$

with bilinear interpolation used at fractional locations. This replaces a rigid kernel footprint by an input-dependent one while retaining end-to-end training by standard back-propagation (Dai et al., 2017).

A third strand clarifies how classical rotate-and-stack constructions fit the same representation-theoretic picture. Filter-transform methods build a kernel from a base filter and its transformed copies; FILTRA shows that such kernels are specific solutions of the group-theoretic kernel constraint

$$\kappa(gx)=\rho_{\mathrm{out}}(g)\,\kappa(x)\,\rho_{\mathrm{in}}(g)^{-1},$$

and interprets them through regular and irreducible representations of $C_N$ and $G$0 (Li et al., 2021).

Taken together, these developments suggest that deformable steerable convolution is best understood not as a single layer type, but as an overview: group-structured steering provides the channel law and kernel basis, while deformability introduces position-dependent transformation parameters or sampling coordinates.

2. Geometric and representation-theoretic structure

A geometric derivation of steerable convolution starts from equivariant bounded linear maps and arrives at generalized convolutions with group-structured kernels. For the special Euclidean group $G$1, the first steerable layer for a scalar field $G$2 has the form

$G$3

where a single canonical filter $G$4 is reused across positions $G$5 and orientations $G$6. Higher layers act on fields defined on $G$7 and couple relative position and relative orientation through kernels on $G$8 (Kundu et al., 21 Oct 2025).

The channel structure is governed by a representation $G$9 of $\Phi : \text{space} \to G$0. A feature map is $\Phi : \text{space} \to G$1-steerable when

$\Phi : \text{space} \to G$2

and a filter $\Phi : \text{space} \to G$3 is $\Phi : \text{space} \to G$4-steerable when

$\Phi : \text{space} \to G$5

This expresses steerability as a linear constraint on kernel weights (Kundu et al., 21 Oct 2025).

Angular dependence is then organized by harmonic analysis. In the continuous $\Phi : \text{space} \to G$6 construction, spherical harmonics furnish the angular basis, while higher-order channel interactions are resolved through the Clebsch–Gordan decomposition

$\Phi : \text{space} \to G$7

In discrete planar settings, the analogous structure appears through regular and irreducible representations of cyclic and dihedral groups, with DCT-like basis matrices $\Phi : \text{space} \to G$8 and $\Phi : \text{space} \to G$9 block-diagonalizing the regular representation (Li et al., 2021).

This structure matters because deformation is not ordinarily introduced in channel space but in spatial sampling. A plausible deformable extension, suggested by the geometric formulation, is therefore to preserve the representation coupling in $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$0-indexed channels and to replace the rigid map $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$1 by a position-dependent sampling map $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$2. In that synthesis, the “steerable” content remains algebraic, while the “deformable” content lives in the sampling geometry.

3. Deformation models and adaptive sampling

The most direct deformation model is the learned offset field of deformable convolution. On a regular image lattice, each canonical kernel offset acquires its own learned perturbation, and fractional sampling is resolved by bilinear interpolation

$$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$3

When offsets are zero, the operator reduces to standard convolution; when offsets are constant and structured, it can emulate different dilation, anisotropic aspect ratios, or rotated sampling patterns (Dai et al., 2017).

In video alignment, deformable convolution admits an especially revealing decomposition. Writing

$$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$4

the deformable layer becomes

$$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$5

which is exactly multiple spatial warpings followed by a $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$6 convolution across the warped features. This exposes the relation to flow-based alignment: a single offset per group behaves like feature warping by a learned task-specific optical flow, whereas multiple offsets provide “offset diversity,” i.e. a multi-hypothesis alignment model (Chan et al., 2020).

On irregular domains, deformability can be expressed through continuous kernel support rather than grid offsets. KPConv places weight matrices $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$7 at kernel points $$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$8 and defines

$$\bigl(\{H,\Phi\}\star F\bigr)(p)=\int H(q)\,\overline{\rho_p(\Phi(p)F)(q)}\,dq$$9

Its deformable form replaces $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$0 by $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$1:

$$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$2

Here the deformation acts directly in Euclidean space and adapts kernel points to local geometry (Thomas et al., 2019).

A common misconception is that any such adaptive operator is automatically “steerable” in the representation-theoretic sense. The literature distinguishes two meanings. In deformable image CNNs, steerability is often only conceptual: the sampling grid is “spatially steerable” by offsets (Dai et al., 2017). In steerable CNNs proper, channel transformations obey explicit representation laws, and kernel spaces are constrained accordingly (Kundu et al., 21 Oct 2025). Deformable steerable convolution, in the stricter sense, requires both.

4. Kernel construction and implementation strategies

One implementation strategy begins from interpolation kernels and harmonic bases. Interpolation is treated as a linear operator from $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$3 to $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$4 with kernel $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$5 satisfying translation-invariance, Hölder regularity, and uniform boundedness. The first-layer steerable basis then has the discrete form

$$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$6

where $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$7 is a precomputed basis determined by the interpolation kernel $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$8, spherical harmonic $$\bigl(\{H,\Phi\}*F\bigr)(p)=\int H(q)\,\rho_q(\Phi(q)F)(p)\,dq,$$9, quadrature weights, and radial grid points. Higher layers reuse the same construction together with Clebsch–Gordan coupling matrices $\mathcal R$0 (Kundu et al., 21 Oct 2025).

A second strategy is filter transform. For a cyclic rotation group $\mathcal R$1, a base filter $\mathcal R$2 is rotated and stacked as

$\mathcal R$3

with an analogous reflected stack for $\mathcal R$4. FILTRA shows that these rotate-and-stack kernels implement equivariant maps from trivial to regular representations and that regular-to-regular kernels can be assembled by combining such transformed copies with DCT-like basis changes $\mathcal R$5 or $\mathcal R$6 (Li et al., 2021).

In the spatially adaptive framework of extended convolution and correlation, the efficient implementation principle is similar even though the presentation differs: the filter space is restricted to a finite-dimensional representation space

$\mathcal R$7

so that locally transformed filters can be synthesized from a fixed set of standard convolutions or correlations followed by position-dependent linear combinations derived from the transformation field $\mathcal R$8 (Mitchel et al., 2020).

These lines of work imply two main design patterns for deformable steerable convolution. One pattern keeps the steerable basis fixed and deforms only the input sampling locations. The other deforms the kernel support itself while preserving the representation coupling in channel space. The first preserves the algebraic Clebsch–Gordan or regular-representation structure most directly; the second relaxes exact equivariance more aggressively but can adapt the support geometry itself. This suggests a spectrum rather than a binary distinction between rigid steerable and fully deformable operators.

5. Approximate equivariance, discretization, and regularization

Exact equivariance is fragile under discretization. In interpolation-based steerable CNNs, two error sources are emphasized: interpolation errors and finite angular or radial discretization. For a steerable field $\mathcal R$9 and an assembled steerable filter $\Delta \mathbf p_n$0, the discrete equivariance error obeys

$\Delta \mathbf p_n$1

where $\Delta \mathbf p_n$2 measures interpolation equivariance error, $\Delta \mathbf p_n$3 is the Hölder exponent of the interpolation kernel, and $\Delta \mathbf p_n$4 is angular resolution. Smoother interpolation kernels and higher angular resolution therefore improve robustness to off-grid transformations (Kundu et al., 21 Oct 2025).

This provides an objective correction to the common assumption that steerability implies exact numerical equivariance after discretization. The literature instead distinguishes exact continuous equivariance from approximate discrete equivariance. FILTRA makes the same point from a different angle: on discrete grids, rotated kernels require interpolation, and exact equalities needed for perfect equivariance fail except in special cases such as $\Delta \mathbf p_n$5 rotations on square grids (Li et al., 2021).

Deformability adds further stability issues. In video super-resolution, deformable alignment can suffer from “overflow of offsets,” where sampling points move too far and aligned features degenerate. An offset-fidelity loss regularizes learned offsets toward optical flow within a thresholded margin, stabilizing training and improving PSNR from 28.753 dB to 30.480 dB on REDS4 and from 33.632 dB to 35.223 dB on Vimeo-90K-T (Chan et al., 2020).

On point clouds, sparse sampling produces a different pathology: “lost kernel points,” whose deformed positions move into empty space and cease to receive gradients. KPConv addresses this with a deformation regularization

$\Delta \mathbf p_n$6

where the fitting term keeps kernel points close to some input neighbor and the repulsive term prevents them from collapsing together (Thomas et al., 2019).

The practical implication is that deformable steerable convolution is usually approximate twice over: first because discretized steerable implementations are only approximately equivariant, and second because spatial deformation departs from rigid group action. Stability therefore depends not only on representation theory but also on interpolation quality, resolution, and deformation regularization.

6. Empirical behavior, applications, and scope

Spatially adaptive steerable operators already support applications to pattern matching, image feature description, vector field visualization, and adaptive image filtering. In the 2D rotation case, the adaptive response can be written as

$\Delta \mathbf p_n$7

showing explicitly how a local angle field $\Delta \mathbf p_n$8 modulates precomputed basis responses (Mitchel et al., 2020). This formulation also reinterprets previous methods including classical steerable filters and the generalized Hough transform (Mitchel et al., 2020).

In image recognition, learned deformation substantially improves geometric adaptation. On semantic segmentation with a DeepLab-like architecture, replacing the last three standard layers by deformable convolution raises mIoU on PASCAL VOC from 69.7 to 75.2 and on Cityscapes from 70.4 to 75.2. In object detection, deformable convolution and deformable RoI pooling improve high-IoU localization, for example lifting R-FCN on VOC 2007 from 80.0/61.8 to 82.6/68.5 at mAP@0.5/[email protected] when both modules are used (Dai et al., 2017).

On point clouds, deformability is more task-dependent. KPConv reports 92.9% OA for rigid and 92.7% OA for deformable classification on ModelNet40, but scene segmentation on S3DIS Area 5 improves from 65.4 to 67.1 mIoU and Paris-Lille-3D from 72.3 to 75.9 mIoU. A particularly informative ablation shows that deformable KPConv loses only about 1.5% mIoU when reduced to $\Delta \mathbf p_n$9 kernel points, whereas rigid KPConv loses about 3.5% mIoU under the same reduction (Thomas et al., 2019).

In video super-resolution, the key empirical variable is offset diversity. Increasing the number of offsets per group from $$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$0 to larger values improves alignment quality, with PSNR on REDS4 rising rapidly up to about $$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$1 and then saturating around $$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$2 dB; the measured diversity correlates strongly with PSNR, with Pearson correlation 0.9418 across the tested $$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$3 values (Chan et al., 2020).

For rigid steerable CNNs, implementation details materially affect robustness. Interpolation-based steerable filters are reported to yield lower equivariance errors under off-grid rotations than Cartesian filters and to be more robust to Gaussian noise. Filter-transform implementations achieve performance comparable to harmonic R2Conv on multiple datasets, with especially strong angle-regression results; for example on rotated MNIST with $$\mathbf y(\mathbf p_0)=\sum_{\mathbf p_n\in\mathcal R}\mathbf w(\mathbf p_n)\cdot \mathbf x(\mathbf p_0+\mathbf p_n+\Delta \mathbf p_n),$$4 equivariance, FILTRA reports 98.9% classification accuracy and 3.3° mean angle error, compared with 98.8% and 4.8° for R2Conv (Kundu et al., 21 Oct 2025, Li et al., 2021).

The overall empirical picture is consistent. Deformability improves adaptation to complex local geometry, steerability improves inductive bias under structured transformations, and interpolation-centered implementations matter whenever sampling is off-grid. A plausible implication is that the most effective deformable steerable convolutions will preserve explicit representation structure in channels while introducing carefully regularized, interpolation-aware deformation in spatial sampling.