4D Scale-Adaptive Filter

• 4D scale-adaptive filter is a mechanism that adjusts its effective support based on varying scales in space-time settings.
• It employs frequency-aware adaptive dilation and scale loss to control aliasing and optimize dynamic scene reconstructions.
• Implementations span extended convolutions, per-pixel adaptive convolutions, and learnable Gaussian envelopes to efficiently manage scale variations.

Searching arXiv for the cited works on scale-adaptive filtering and 4D Gaussian Splatting. A 4D scale-adaptive filter is a filtering mechanism whose effective support or smoothing strength varies with scale in a four-dimensional setting. In the strict sense used in alias-free 4D Gaussian Splatting, it is a filter and associated scale loss that regulate the sampling frequency of time-varying Gaussian primitives so that changes in camera focal length, camera-object distance, or rendering resolution do not produce high-frequency artifacts (Chen et al., 23 Nov 2025). In a broader research sense, closely related constructions include extended convolution and correlation with spatially varying transformations, per-pixel adaptive convolutions assembled from multi-scale bases, explicit scale filters in discriminative tracking, scale-aware residual fields for 4D Gaussian primitives, and learnable convolutional envelopes whose covariance controls filter scale and orientation (Mitchel et al., 2020).

1. Scope of the term

The literature uses the expression in several related but non-identical ways. In dynamic scene reconstruction, “4D” typically denotes space-time Gaussian primitives and the filter acts on their scale under changing sampling conditions. In adaptive convolution, the same phrase can be interpreted as a filter whose response depends jointly on spatial location, kernel support, and a scale axis. In tracking, it can describe filtering over translation, scale, and feature dimensions. In learnable CNN envelopes, the term is only an interpretation: the paper itself does not define a literal 4D tensor filter, but a 2D Gaussian envelope with covariance degrees of freedom that jointly govern scale and orientation (Cam et al., 2018).

Work	4D interpretation	Scale-adaptive mechanism
(Chen et al., 23 Nov 2025)	Space-time Gaussian primitives	Frequency-aware adaptive dilation and scale loss
(Mitchel et al., 2020)	Spatial position plus transformation/scale field	Extended convolution/correlation
(Wang et al., 2021)	Per-pixel filtering with a scale-basis axis	Dynamic atoms over multi-scale bases
(Danelljan et al., 2016)	Translation, scale, and feature dimensions	Separate discriminative scale filter
(Yan et al., 2024)	4D Gaussians with temporal activity	Scale-aware residual field
(Cam et al., 2018)	Editor’s term: “covariance-augmented filter geometry”	Learnable Gaussian envelope

A common misconception is that the term must refer to a literal 4D kernel tensor. Several papers support a broader interpretation. The representation-theoretic framework for extended convolution is not described as a 4D tensor construction, yet it explicitly supports adaptive scale transforms and can be interpreted as scale-adaptive filtering over a higher-dimensional signal/filter parameter space (Mitchel et al., 2020). Likewise, ACDA can be interpreted as a 4D scale-adaptive filter because the effective filter mixes across a scale axis while varying per pixel, but the paper formulates the method as dynamic atoms and multi-scale bases rather than as an explicit 4D tensor (Wang et al., 2021).

2. Representation-theoretic foundations

A general foundation for scale-adaptive filtering is the extension of convolution and correlation by allowing a transformation field to act on the filter at each output location. In this formulation, the signal is $H$, the filter is $F$, the spatially varying transformation is applied locally to the filter, and translation is represented by $(\rho_p F)(q)=F(q-p)$. The resulting operators are

$$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$

and

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$

The conceptual distinction is explicit: extended correlation is “gathering” at $p$, whereas extended convolution is “scattering” from each $q$. The importance for scale adaptation is that the filter is no longer globally fixed; it is transformed locally according to a field such as rotation or scale (Mitchel et al., 2020).

Efficiency is obtained by restricting the filter space to a finite-dimensional basis,

$$\mathcal F = \mathrm{Span}\langle F_1,\ldots,F_n\rangle,$$

so that the transformation acts linearly and becomes a matrix on coefficients. Representation theory is then used to choose a basis in which the transformation matrices become sparse or block-diagonal. The decomposition theorem

$${\mathcal F} = \,\bigoplus_{k=1}^{\gamma} \left(\bigoplus_{l=1}^{m_k}{\mathcal F}_{kl}\right)$$

permits reduction from $n^2$ standard convolutions or correlations to $F$0 in the reduced form. For practical approximation, the first $F$1 frequencies are retained, which the paper describes as band-limiting the filter before applying the adaptive operator (Mitchel et al., 2020).

Scale enters this framework through a basis adapted to multiplicative changes in radius: $F$2 This is the scale analogue of the rotational Fourier basis, with Fourier analysis performed on $F$3. For adaptive smoothing, the paper normalizes the signal as

$F$4

and defines the normalized smoothed output by

$F$5

That normalization is central because varying scale changes kernel support and integral; without it, local scale changes would induce intensity bias and blur-bleeding (Mitchel et al., 2020).

3. The explicit 4D scale-adaptive filter in alias-free 4D Gaussian Splatting

In alias-free 4D Gaussian Splatting, the 4D scale-adaptive filter is introduced to address two coupled problems: the absence of an explicit maximum-frequency constraint for each Gaussian primitive, and the scale mismatch induced by a fixed 2D dilated filter in screen space. The motivating observation is that changing focal length, camera-object distance, or rendering resolution changes the effective sampling rate, while fixed learned Gaussian scales can then expose previously harmless high-frequency content as visible artifacts (Chen et al., 23 Nov 2025).

The derivation uses the relationship between Gaussian-camera distance $F$6 and focal length $F$7. The maximum sampling frequency of a Gaussian primitive is determined by the most demanding visible view and time, i.e. by the smallest effective sampling interval across cameras and temporal states. Because evaluating all deformation states directly is expensive, the method first estimates a coarse minimum sampling interval from the static Gaussian position, and after deformation stabilizes refines it through a momentum-style update. A practical implication is that the filter is frequency-aware rather than purely image-space-aware (Chen et al., 23 Nov 2025).

The adaptive filter then modulates dilation according to current Gaussian scale, temporal deformation, and the estimated maximum sampling frequency. The scale-ratio coefficient is

$F$8

where $F$9 is the original scale and $(\rho_p F)(q)=F(q-p)$0 is the time-varying scale. Visibility-aware masking is defined by

$(\rho_p F)(q)=F(q-p)$1

This rule has two stated effects: small or invisible Gaussians are masked so that filtering does not make them visible, while visible Gaussians receive a scale-adaptive dilation (Chen et al., 23 Nov 2025).

The filtered 4D Gaussian is defined by replacing the covariance with a dilated, sampling-frequency-controlled covariance term: $(\rho_p F)(q)=F(q-p)$2 The associated scale regularization term is

$(\rho_p F)(q)=F(q-p)$3

and the total objective is

$(\rho_p F)(q)=F(q-p)$4

The stated optimization role is to split the anti-aliasing burden between the filter and the Gaussian’s own learned scale. Instead of forcing the filter alone to smooth all high-frequency content, the scale loss nudges Gaussians toward a size range where the adaptive filter can operate with a smaller minimum dilation scale. The paper reports that this suppresses high-frequency artifacts under increased rendering frequencies and reduces redundant Gaussians in multi-view video reconstruction (Chen et al., 23 Nov 2025).

4. Dynamic convolutional realizations

A second major line of work realizes scale-adaptive filtering as per-pixel adaptive convolution. ACDA starts from standard convolution

$(\rho_p F)(q)=F(q-p)$5

and replaces the shared filter $(\rho_p F)(q)=F(q-p)$6 with a local filter $(\rho_p F)(q)=F(q-p)$7,

$(\rho_p F)(q)=F(q-p)$8

Here $(\rho_p F)(q)=F(q-p)$9 is a convolutional filter generation network, and the key fact is that each spatial position receives a filter predicted from its local neighborhood. Direct generation of full per-pixel kernels is avoided by decomposing them into dynamic filter atoms, and adaptive receptive fields are supported by representing each atom over pre-fixed multi-scale bases (Wang et al., 2021).

The dynamic atoms are reconstructed from basis coefficients over $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$0 scale groups and $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$1 basis elements per scale. The interpretation given in the paper is explicit: the network does not directly learn a large kernel at each pixel, but instead learns how to mix pre-fixed basis filters at multiple scales. More weight on small-scale bases yields a smaller receptive field, and more weight on larger-scale bases yields a larger receptive field. The method preserves translation equivariance because both the atom-generation network and the composition coefficients are shared across spatial positions (Wang et al., 2021).

A related but architecturally distinct construction learns scale and orientation inside a single convolutional layer by masking a large base filter grid with a differentiable Gaussian envelope. On a grid $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$2, the envelope is

$$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$3

and the effective convolution becomes

$$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$4

The mean $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$5 is fixed at the grid center, while the covariance

$$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$6

is learned by backpropagation. The largest eigenvalue represents scale and the angle of the largest eigenvector represents orientation. The paper states that the envelope can grow, shrink, or rotate by updating its covariance matrix, and that only the weights in the envelope are updated during training (Cam et al., 2018).

These two approaches instantiate different design choices. ACDA makes scale adaptation data-dependent at each pixel and factors the kernel into atoms and shared coefficients. The Gaussian-envelope model keeps a conventional convolutional layout but learns the active support within a larger grid. A plausible implication is that both can be read as scale-adaptive filters over an augmented parameter space, even though neither paper defines the object primarily as a 4D Gaussian-splatting-style filter.

5. Scale filtering in tracking and scale-space estimation

In discriminative tracking, scale adaptation appears as an explicit correlation filter learned on scale space rather than as a spatially varying kernel field. DSST formulates the tracking state as $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$7 and replaces exhaustive scale search with separate discriminative correlation filters for translation and scale. The translation filter remains a standard 2D DCF, while the scale filter is a 1D DCF trained on target appearance sampled at different scales (Danelljan et al., 2016).

The multi-channel DCF objective is

$$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$8

For scale training, if the current target size is $$\{H,\ \} \star F = \int H(q)\,\overline{\rho_p\bigl((p)\,F\bigr)(q)} \, dq$$9 and the scale filter length is $$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$0, image patches of size

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$1

are extracted for

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$2

resized to a fixed size, and converted into a 1D sequence of feature descriptors across discrete scales. The same DCF update machinery is then applied in the scale domain (Danelljan et al., 2016).

The importance of DSST for the present topic is conceptual. It shows that scale adaptation can be treated as discriminative learning over a scale axis rather than as brute-force repeated evaluation of a fixed filter at multiple resolutions. The paper states that the explicit scale filter is learned online using target appearance sampled at a set of different scales, and that this directly learns the appearance change induced by variations in target scale (Danelljan et al., 2016). A plausible implication is that this is a task-specific realization of scale-adaptive filtering in a state space that is effectively higher-dimensional, even though the method is not framed as a general 4D filter.

6. Scale-aware 4D Gaussian representations

A separate but closely related development appears in SaRO-GS, which introduces a Scale-aware Residual Field for 4D Gaussian primitives. The primitive is

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$3

with $$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$4. At a sampled time $$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$5, a residual feature is computed by

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$6

a lifespan is predicted as

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$7

and temporal activity is modeled by

$$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$8

with $$\{H,\ \} * F = \int H(q)\,\rho_q\bigl((q)F\bigr)(p) \, dq .$$9. The opacity is gated by $p$0, and time-conditioned residuals

$p$1

produce the 3D Gaussian attributes used in the standard 3DGS rasterization pipeline (Yan et al., 2024).

Scale awareness enters through hexplanes. Spatial-only planes $p$2 are equipped with a MipMap stack, while spatiotemporal planes $p$3 use standard features. For a projected Gaussian scale $p$4, the level is estimated by

$p$5

and features are obtained by interpolation between neighboring MipMap levels. The paper’s stated rationale is that Gaussians are ellipsoids rather than points, so encoding only the center ignores the region actually occupied by the primitive and becomes especially problematic when large Gaussians later split into smaller children (Yan et al., 2024).

SaRO-GS also defines a per-Gaussian adaptive optimization schedule: $p$6 where $p$7 is a temporal integral of the state function over the observation range. Gaussians active for shorter intervals receive lower densification thresholds and higher learning rates, which the paper states expedites the reconstruction of dynamic regions (Yan et al., 2024). This suggests a broader view of 4D scale-adaptive filtering: scale awareness may be embedded not only in a dedicated anti-aliasing filter, but also in the feature extraction and optimization processes that determine how 4D Gaussian primitives are represented.

7. Constraints, ambiguities, and methodological boundaries

The concept is unified by local adaptation of filter scale, but the mathematical objects differ substantially across subfields. In the representation-theoretic framework, the filter space must be representable in a finite basis, the transformation must act linearly, and practical efficiency depends on sparse or block-diagonal representation matrices and often on band-limiting (Mitchel et al., 2020). In dynamic convolutions, naive per-pixel generation of full kernels is explicitly described as infeasible because of memory and compute costs; ACDA addresses this by generating low-dimensional atoms and composing them through shared coefficients (Wang et al., 2021). In the Gaussian-envelope CNN, the adaptive model is slower during training, and the Gaussian envelope introduces a smoothing bias because it is not a free-form support mask (Cam et al., 2018). In 4D Gaussian Splatting, a fixed 2D dilation filter is insufficient because it ignores time-varying Gaussian scale and can distort anisotropy or make imperceptible Gaussians visible (Chen et al., 23 Nov 2025).

Another boundary concerns terminology. Not every scale-aware mechanism in a 4D representation is itself a “4D scale-adaptive filter” in the narrow sense. The alias-free 4DGS paper defines the term explicitly as a frequency-aware adaptive filter and scale loss for 4D Gaussian primitives (Chen et al., 23 Nov 2025). By contrast, SaRO-GS introduces a Scale-aware Residual Field rather than an explicit anti-aliasing filter (Yan et al., 2024). ACDA introduces adaptive convolutions with per-pixel dynamic atoms and pre-fixed multi-scale bases rather than a literal 4D filter tensor (Wang et al., 2021). The CNN-envelope paper explicitly does not define a literal 4D tensor filter; the 4D interpretation there is only a practical one based on spatial coordinates plus covariance-controlled geometry (Cam et al., 2018).

Taken together, the literature supports a stable core definition. A 4D scale-adaptive filter is a mechanism that couples filtering to a scale variable in a higher-dimensional domain—whether the domain is position plus transformation, image position plus scale basis, translation-scale state space, or full space-time Gaussian primitives. The principal technical questions are then how scale is parameterized, how adaptation is normalized or regularized, how efficiency is preserved, and how artifacts from mismatch between sampling rate and filter support are controlled.