Overview of Position Adaptive Convolution (PAC)

• Position Adaptive Convolution (PAC) is a family of operators that adapt convolutional behavior based on local spatial features to improve image and 3D data processing.
• Variants modulate fixed filters with guidance features, learn continuous offsets, or predict per-pixel kernels, yielding flexible receptive fields and improved accuracy.
• PAC methods are applied in image restoration, interpolation, and point cloud segmentation, demonstrating enhanced performance and efficiency over standard convolutions.

Position Adaptive Convolution (PAC) denotes a family of convolution operators in which the effective kernel, the sampling geometry, or both depend on spatial position or local content rather than remaining fully shared across the domain. On image grids, PAC is often used synonymously with pixel-adaptive convolution, where a shared spatial filter is modulated by a spatially varying kernel derived from guidance features; related formulations instead learn continuous sampling offsets, predict full per-pixel kernels, or multiplicatively gate a base kernel. On irregular domains such as point clouds, the same principle appears as dynamically assembled kernels conditioned on local geometry. The literature therefore treats PAC less as a single canonical operator than as a design pattern for position-conditioned convolution (Su et al., 2019, Jeon et al., 2017, Xu et al., 2021).

1. Scope and operator family

The canonical reference formulation on image grids starts from standard spatial convolution

$$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b},$$

where $\Omega(i)$ is an $s \times s$ neighborhood, $\mathbf{W}[\Delta]$ is the filter slice at spatial offset $\Delta$, and $\mathbf{v}'_i$ is the output at pixel $i$. Pixel-Adaptive Convolution modifies this by introducing a spatially varying kernel $K(\mathbf{f}_i,\mathbf{f}_j)$ derived from adapting features $\mathbf{f}$: $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$ In this form, PAC generalizes spatial convolution, bilateral filtering, and pooling (Su et al., 2019).

The same underlying idea recurs in several distinct parameterizations. Some works adapt the kernel weights while keeping the sampling grid fixed; some learn the sampling positions themselves; some predict a full local kernel for each pixel; and some factorize the dynamic kernel through bases, atoms, or a weight bank. This suggests that “position adaptive convolution” is best understood as an umbrella term for convolutions whose effective receptive pattern is conditioned on position-dependent information.

Variant	Adaptation mechanism	Representative paper
Pixel-Adaptive Convolution	$\Omega(i)$0 modulates shared $\Omega(i)$1	(Su et al., 2019)
Active Convolution Unit	Learnable continuous offsets $\Omega(i)$2	(Jeon et al., 2017)
Adaptive Convolution for interpolation	Per-pixel kernels over two input frames	(Niklaus et al., 2017)
LAConv	Patch-conditioned $\Omega(i)$3 gate scales base kernel	(Jin et al., 2021)
PPAC	PAC plus confidence gating and advanced normalization	(Wannenwetsch et al., 2020)
PAConv	Dynamic kernel assembling from a Weight Bank	(Xu et al., 2021)
ACDA	Per-pixel dynamic atoms over multi-scale bases	(Wang et al., 2021)

2. Feature-conditioned PAC on image grids

In the formulation of Su et al., the adapting signal enters through local features $\Omega(i)$4, typically with a Gaussian kernel

$\Omega(i)$5

The modulation is usually scalar per neighbor and then broadcast across channels, although the framework allows channel-aware variants. Because the operator preserves the standard convolutional neighborhood specification—filter size, stride, padding, and dilation—it can be used as a drop-in replacement for pretrained convolutions. The same work also defines PAC-$\Omega(i)$6, where learned spatial compatibility tensors and Gaussian kernels over features replace the high-dimensional filtering of Fully-Connected CRFs, yielding an efficient mean-field update on fixed windows with optional dilation (Su et al., 2019).

LAConv instantiates the same PAC principle in a structured multiplicative form. For a local patch $\Omega(i)$7, a shared kernel bank $\Omega(i)$8, and a patch-conditioned gate $\Omega(i)$9, the paper defines

$s \times s$0

where $s \times s$1 duplicates $s \times s$2 across channels. In explicit PAC language, this corresponds to

$s \times s$3

with $s \times s$4 produced from the local patch. LAConv augments this with a dynamic bias $s \times s$5 generated from $s \times s$6, so that

$s \times s$7

The paper positions LAConv as a per-pixel, patch-conditioned alternative to sample-conditioned dynamic convolutions such as CondConv and DYConv, and as distinct from deformable convolution because the sampling grid remains fixed while the kernel values are adapted (Jin et al., 2021).

Probabilistic Pixel-Adaptive Convolution extends PAC by incorporating explicit per-pixel confidence. If $s \times s$8 denotes the confidence associated with neighbor $s \times s$9, then PPAC uses

$\mathbf{W}[\Delta]$0

Its purpose is not merely image-adaptive smoothing but also preferential propagation from high-confidence neighbors into less reliable regions while respecting boundaries. This is paired with an “advanced normalization” in which an auxiliary constant input is passed through the same operator using a strictly positive normalization kernel $\mathbf{W}[\Delta]$1, and the output is divided by the auxiliary response before adding bias. The paper argues that kernel-only normalization is insufficient because the effective magnitude depends jointly on $\mathbf{W}[\Delta]$2, $\mathbf{W}[\Delta]$3, and local neighborhood structure (Wannenwetsch et al., 2020).

3. Learnable sampling geometry and full kernel synthesis

A distinct PAC line adapts the sampling positions rather than—or in addition to—the kernel weights. The Active Convolution Unit replaces the fixed grid $\mathbf{W}[\Delta]$4 of standard convolution with learnable continuous synapse positions $\mathbf{W}[\Delta]$5. With position parameters $\mathbf{W}[\Delta]$6, ACU computes

$\mathbf{W}[\Delta]$7

Because $\mathbf{W}[\Delta]$8 and $\mathbf{W}[\Delta]$9 are generally fractional, the sampled feature $\Delta$0 is obtained by bilinear interpolation from the four neighbors $\Delta$1. This makes the output differentiable with respect to $\Delta$2 almost everywhere, enabling backpropagation of the sampling geometry itself. The operator generalizes conventional convolution, dilated convolution, shifted kernels, and arbitrary sparse or fractional sampling patterns. In the reported implementation, $\Delta$3 is fixed at $\Delta$4, offsets are shared across spatial positions and typically across kernels in a layer, initialization uses the regular $\Delta$5 grid, and position updates use a normalized gradient with a smaller learning rate than weights (Jeon et al., 2017).

Video frame interpolation via adaptive convolution represents an even more explicit per-pixel synthesis operator. For each output location $\Delta$6, the interpolated midpoint frame is defined by two spatially adaptive sub-kernels $\Delta$7 and $\Delta$8: $\Delta$9 The paper instantiates $\mathbf{v}'_i$0, so the concatenated kernel has $\mathbf{v}'_i$1 elements. A fully convolutional network predicts these logits from co-centered $\mathbf{v}'_i$2 receptive-field patches from the two frames, followed by a spatial softmax across both sub-kernels jointly. This imposes non-negativity and a sum-to-one constraint over the concatenated kernel, while leaving the two sub-kernels free to allocate negligible mass to one frame under occlusion. In this formulation, kernel mass concentration encodes local motion and per-frame mass allocation encodes visibility and blending, thereby replacing the traditional motion-estimation-plus-warping pipeline with a single learned convolution process (Niklaus et al., 2017).

Adaptive Convolutions with Per-pixel Dynamic Filter Atom move further toward full per-pixel kernel construction while factorizing the dynamic kernel to control computation and memory. ACDA defines

$\mathbf{v}'_i$3

where $\mathbf{v}'_i$4 are shared channel-mixing coefficients and $\mathbf{v}'_i$5 are per-pixel dynamic atoms. Each atom is itself expanded over fixed multi-scale bases: $\mathbf{v}'_i$6 The atom coefficients $\mathbf{v}'_i$7 are produced by a lightweight two-layer CNN, while the bases $\mathbf{v}'_i$8 are fixed. This yields explicit adaptive receptive fields because different positions can upweight different scales, yet the operator preserves strict translation equivariance since the conditioning network is convolutional and the sampling positions are fixed (Wang et al., 2021).

4. Position-adaptive convolution on point clouds

On point clouds, the regular-grid assumptions of 2D convolution fail because positions are continuous, neighborhoods are unordered, and permutation invariance is required. PAConv addresses this by dynamically assembling a position-adaptive kernel from a small Weight Bank rather than predicting a full kernel per edge. For a neighborhood $\mathbf{v}'_i$9, basis matrices $i$0, and ScoreNet coefficients $i$1, the kernel is

$i$2

and the per-point output is

$i$3

In practice, PAConv uses MAX pooling as the symmetric aggregation, so permutation invariance is retained (Xu et al., 2021).

The ScoreNet input can include relative offsets, absolute coordinates of the center point, and Euclidean distance,

$i$4

and a small MLP followed by softmax yields $i$5 with $i$6. The implementation exploits a CUDA-friendly reformulation,

$i$7

so that transformed features $i$8 can be precomputed and then assembled with scalar coefficients. To encourage diversity among basis matrices, the paper adds a correlation loss

$i$9

PAConv is explicitly designed as a plug-and-play replacement for PointNet, DGCNN, and PointNet++ blocks without changing the surrounding network configurations (Xu et al., 2021).

5. Optimization, normalization, and systems considerations

The PAC literature devotes substantial attention to stabilization because position adaptation couples convolution with additional latent variables—features, offsets, confidence maps, or routing coefficients. In ACU, the gradients with respect to offsets follow directly from the bilinear interpolation terms, and the paper uses normalized offset gradients

$K(\mathbf{f}_i,\mathbf{f}_j)$0

with

$K(\mathbf{f}_i,\mathbf{f}_j)$1

together with a warm-up stage in which weights are trained while positions remain fixed for several iterations. This is used to avoid poor local minima caused by unstable early offset motion (Jeon et al., 2017).

Normalization is a central issue in PAC and PPAC. Standard PAC implementations may normalize only the kernel $K(\mathbf{f}_i,\mathbf{f}_j)$2, but the PPAC paper shows that the effective response magnitude also depends on the learned spatial filter $K(\mathbf{f}_i,\mathbf{f}_j)$3 and on how much mass is concentrated in a structured region. Its advanced normalization therefore computes an auxiliary response

$K(\mathbf{f}_i,\mathbf{f}_j)$4

with $K(\mathbf{f}_i,\mathbf{f}_j)$5 constrained to be strictly positive by learning it in log-space, and normalizes the actual filtered output by $K(\mathbf{f}_i,\mathbf{f}_j)$6. This choice is particularly important when convolution weights may contain negative entries (Wannenwetsch et al., 2020).

Per-pixel kernel prediction is often limited more by systems cost than by parameter count. In video interpolation, predicting a $K(\mathbf{f}_i,\mathbf{f}_j)$7-dimensional kernel per pixel is compute- and memory-intensive; the paper therefore exploits the fully convolutional structure through a “shift-and-stitch” strategy, reducing HD inference from $K(\mathbf{f}_i,\mathbf{f}_j)$8 seconds for $K(\mathbf{f}_i,\mathbf{f}_j)$9 per-pixel forward passes to approximately $\mathbf{f}$0 seconds on a Titan X. ACU reports a forward pass approximately $\mathbf{f}$1 slower than standard convolution in the provided Caffe implementation because of bilinear interpolation. PAConv addresses the analogous issue on point clouds by predicting only $\mathbf{f}$2 routing coefficients per edge and reusing a Weight Bank, while the released CUDA implementation reduces memory from more than $\mathbf{f}$3GB to approximately $\mathbf{f}$4GB for $\mathbf{f}$5 points. ACDA makes the same design move in dense images: instead of materializing a full per-pixel kernel, it stores compact atom coefficients and shared mixing weights; the paper notes that naive per-pixel gradients for a single layer with $\mathbf{f}$6 and $\mathbf{f}$7 would consume approximately $\mathbf{f}$8GB (Niklaus et al., 2017, Xu et al., 2021, Wang et al., 2021).

6. Empirical behavior, applications, and nomenclature

Empirically, PAC variants have been used in classification, dense prediction, image restoration, interpolation, fusion, and 3D geometric learning. On CIFAR-10 and CIFAR-100, replacing six $\mathbf{f}$9 convolutions with ACUs in a plain CNN improved test error from $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$0 to $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$1 and from $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$2 to $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$3, respectively, with only $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$4 added parameters; on Places365, replacing AlexNet $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$5 convolutions with ACUs improved Top-1 from $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$6 to $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$7 and Top-5 from $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$8 to $$\mathbf{v}'_i \;=\; \sum_{j \in \Omega(i)} K\!\left(\mathbf{f}_i, \mathbf{f}_j\right)\, \mathbf{W}\big[\mathbf{p}_i - \mathbf{p}_j\big]\, \mathbf{v}_j \;+\; \mathbf{b}.$$9 (Jeon et al., 2017). The original Pixel-Adaptive Convolution paper reports state-of-the-art performance for deep joint image upsampling and competitive PAC-$\Omega(i)$00 refinement, with PAC-$\Omega(i)$01 yielding $\Omega(i)$02 ms for a single $\Omega(i)$03 kernel with dilation $\Omega(i)$04, and a two-kernel configuration improving Pascal VOC mIoU by $\Omega(i)$05 on validation and $\Omega(i)$06 on test over FCN-8s unaries (Su et al., 2019).

The probabilistic extension shows that explicit uncertainty can further sharpen PAC behavior. For optical flow refinement, PPAC improves over PAC and over simple convolutional refinement, and on Pascal VOC 2012 it improves DeepLabv3+ from $\Omega(i)$07 to $\Omega(i)$08 mIoU while reducing boundary artifacts (Wannenwetsch et al., 2020). For image fusion, LAResNet equipped with LAConv and dynamic bias reports the best quantitative results across the listed WV3, GF2, QB, and CAVE settings, and Table 13 reports $\Omega(i)$09 parameters for LAResNet, the lowest among the listed CNNs, while the ablation identifies LAC+DYB as the strongest of the six tested configurations on WV3 Tripoli (Jin et al., 2021). In video frame interpolation, the adaptive-convolution model ranks best on Evergreen and Basketball, second on Dumptruck, and third on Backyard among Middlebury real-world scenes, while qualitative results emphasize sharp interpolation under occlusion, blur, and abrupt brightness change (Niklaus et al., 2017).

On point clouds, PAConv improves classical MLP-based backbones without redesigning them. The paper reports PointNet baseline $\Omega(i)$10 versus PointNet + PAConv $\Omega(i)$11 on ModelNet40, DGCNN baseline class mIoU $\Omega(i)$12 versus DGCNN + PAConv class mIoU $\Omega(i)$13 with voting on ShapeNet Part, and PointNet++ baseline $\Omega(i)$14 versus PAConv $\Omega(i)$15 mIoU with voting on S3DIS Area-5 (Xu et al., 2021). ACDA, although framed as adaptive convolutions with per-pixel dynamic atoms rather than explicitly as PAC, belongs to the same broader family and reports improvements across toy multi-scale pattern detection, CIFAR-10/100, ImageNet, crowd counting, real-world super-resolution, and real-world denoising, particularly on tasks with significant intra-image variance (Wang et al., 2021).

A recurring source of confusion is nomenclature. In vision papers, “PAC” commonly denotes Pixel-Adaptive Convolution, and some literature refers to “Position-Adaptive Convolution” synonymously because the adaptation depends on pixel position through guidance features. In 3D vision, PAConv uses the phrase “Position Adaptive Convolution” explicitly for point clouds. By contrast, PAC in coding theory denotes polarization-adjusted convolutional codes, where the acronym refers to a concatenation of convolutional and polar transforms and has no relation to adaptive convolution operators (Wannenwetsch et al., 2020, Sun et al., 2021).