Spatio-Temporal Effective Receptive Field

• ST-ERF is the spatio-temporal volume of input data that influences a neuron, linking architecture design, dynamics, and invariance across space and time.
• It integrates theories from CNN geometry, scale-space models, and gradient-based analyses to distinguish between theoretical support and actual influence distribution.
• ST-ERF guides practical design choices in video networks, motion-adapted systems, and SNNs by informing kernel configuration, temporal causality, and spatial coverage.

Searching arXiv for the cited ST-ERF and receptive-field literature to ground the article. Spatio-Temporal Effective Receptive Field (ST-ERF) denotes the spatio-temporal volume of input data that can influence a unit, filter response, or neural activation over both space and time. In the geometric CNN sense, it is the input video volume $(T \times H \times W)$ that can possibly affect a neuron; in scale-space and biological-vision formulations, it is the effective space-time support of Gaussian- or affine-Gaussian-based receptive fields coupled with temporal kernels; and in gradient-based analyses of modern spiking networks, it is the derivative-defined influence distribution $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$ over spatial position and temporal delay (Le et al., 2017). Across these traditions, ST-ERF links architecture, dynamics, and invariance: it specifies what spatial region, what temporal interval, and, in motion-adapted formulations, what space-time trajectory a unit effectively integrates.

1. Conceptual definition and terminological scope

The term “receptive field” is used in several related but non-identical senses. In the CNN formulation, Receptive Field (RF) is “a local region (including its depth) on the output volume of the previous layer that a neuron is connected to,” while Effective Receptive Field (ERF) is “the area of the original image that can possibly influence the activation of a neuron” (Le et al., 2017). In that usage, RF is local between adjacent layers, whereas ERF is the projection of that dependency back to the original input. Extending this dimension-wise to video or 3D convolutions yields the natural notion of ST-ERF: the spatio-temporal volume in the original input that can possibly influence a unit (Le et al., 2017).

A second usage comes from scale-space and theoretical vision. There, spatio-temporal receptive fields are defined as linear kernels over 2D space and 1D time, often of the form

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$

with $g$ an affine Gaussian over space and $h$ a temporal smoothing kernel (Lindeberg, 2015). In this setting, ST-ERF refers to the effective support of that space-time kernel: an anisotropic spatial envelope, a temporally extended support, and, when $v \neq 0$, a tilt along motion trajectories (Lindeberg, 2017).

A third usage appears in modern gradient-based analyses. In spiking neural networks, the paper on Transformer-based SNNs defines the Spatio-Temporal Effective Receptive Field explicitly as

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$

thereby treating space and time on equal footing (Zhang et al., 24 Oct 2025). This definition makes ST-ERF a three-dimensional influence field over spatial coordinates and temporal delay.

A persistent source of ambiguity is the word “effective.” In the CNN geometry paper, ERF is the set of input positions that can influence a unit, assuming nonzero weights (Le et al., 2017). In later usage, especially following gradient analyses, “effective receptive field” often refers to the non-uniform influence distribution within that theoretical support, typically concentrated near the center. This suggests that ST-ERF has both a theoretical meaning, concerning support extent, and a distributional meaning, concerning how influence is apportioned within that support.

2. Geometric ST-ERF in convolutional architectures

For convolutional and pooling networks, ST-ERF follows from dimension-wise recurrences. In the 1D derivation used by the CNN receptive-field paper, if $R_k$ is the ERF size at layer $k$, $f_k$ the filter size, and $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$0 the stride, then

$\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$1

The equivalent top-down recurrence is

$\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$2

Both are applied independently per dimension in 2D or 3D, so a video/3D CNN obtains temporal, height, and width extents by tracking separate recurrences along $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$3 (Le et al., 2017).

The same source makes the dimension-wise generalization explicit. For each layer $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$4 and dimension $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$5, with kernel $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$6, stride $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$7, dilation $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$8, jump $\partial y_{(m,n)}[t]/\partial x_{(i,j)}[t-\tau]$9, and receptive-field size $$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$0,

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$1

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$2

Ignoring padding and dilation reduces this to the earlier recurrence. The ST-ERF at layer $$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$3 is then a block of size

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$4

This is the direct spatio-temporal analogue of the image-case ERF (Le et al., 2017).

The same paper also distinguishes Projective Field (PF), defined as “the set of neurons to which a neuron projects its output.” For spatio-temporal models, this extends to a 3D block in $$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$5, with possibly distinct sizes per dimension (Le et al., 2017). This is relevant because ST-ERF and spatio-temporal PF are dual notions: one asks what past inputs can affect a target unit, the other asks how a source voxel propagates forward through the network.

A second architecture-centered perspective appears in receptive-field search. RF-Next parameterizes receptive field mainly by dilation rates and searches for receptive-field combinations $$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$6, where each layer chooses a dilation from a candidate set (Gao et al., 2022). For a 1D TCN layer $$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$7, the incremental contribution is

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$8

and the cumulative theoretical receptive field is

$$T(x_1, x_2, t;\, s, \tau, v, \Sigma) = g(x_1 - v_1 t, x_2 - v_2 t;\, s, \Sigma)\, h(t;\, \tau),$$9

The paper interprets dilation schedules as direct control over temporal or spatial receptive fields and argues that searched combinations outperform hand-designed patterns on temporal action segmentation, object detection, instance segmentation, and speech synthesis (Gao et al., 2022). A plausible implication is that ST-ERF design is not only an analytical question but also an optimization variable.

3. Continuous spatio-temporal receptive-field models

In continuous spatio-temporal scale-space theory, ST-ERF is grounded in explicit kernel families. A central model is the separable or motion-adapted spatio-temporal receptive field

$g$0

where $g$1 is an affine Gaussian and $g$2 is a temporal smoothing kernel (Lindeberg, 2015). In the isotropic case, $g$3, the spatial kernel becomes

$g$4

The spatial standard deviation is $g$5, which serves as a natural measure of effective spatial receptive-field size (Lindeberg, 2015).

The temporal component in time-causal formulations is built from cascades of first-order integrators, equivalently truncated exponential filters. A single kernel is

$g$6

and a $g$7-stage cascade is

$g$8

with mean and variance

$g$9

This makes the temporal ST-ERF strictly causal and time-recursive, so it depends only on the present and past and can be updated from a finite temporal state (Lindeberg, 2015).

The full derivative-based family is obtained by applying spatial and temporal derivatives to the smoothing kernel. A general spatio-temporal derivative receptive field has the form

$h$0

with velocity-adapted temporal derivative

$h$1

This yields separable ST-ERFs when $h$2 and tilted, non-separable ST-ERFs when $h$3 (Lindeberg, 2017).

In these models, the effective spatio-temporal support is controlled by the Gaussian and temporal envelopes, while derivative order determines internal sign structure and selectivity. The papers explicitly connect such receptive fields to LGN and V1 models, including center-surround structure, oriented simple-cell profiles, and velocity-adapted inseparable receptive fields (Lindeberg, 2015).

4. Temporal causality, temporal scale, and ST-ERF dynamics

Time-causal ST-ERF theory differs from non-causal Gaussian time models by imposing temporal asymmetry and recursivity. The time-causal papers require that the temporal smoothing process be time-causal, that temporal scale levels be discrete, and that increasing temporal scale must not create new local extrema or zero-crossings (Lindeberg, 2015). Under those constraints, cascades of truncated exponentials are the admissible temporal scale-space kernels.

Two parameterizations of intermediate temporal scales are emphasized. The uniform distribution in variance uses

$h$4

which gives equal time constants. The logarithmic distribution uses

$h$5

with corresponding time constants

$h$6

$h$7

The logarithmic distribution is reported to lead to “significantly shorter delays” and “much faster temporal response properties (shorter temporal delays)” than the uniform distribution for the same $h$8 and $h$9 (Lindeberg, 2015).

Temporal dynamics can be summarized by the mean delay $v \neq 0$0, the delay at maximum response, and the kernel’s skewness. In the equal-$v \neq 0$1 case, the composed temporal kernel is

$v \neq 0$2

with maximum at

$v \neq 0$3

The papers describe logarithmic distributions as concentrating more mass near the present and producing “sharper effective temporal receptive fields” with shorter delay (Lindeberg, 2015).

This temporal structure is central to ST-ERF interpretation. In a separable kernel $v \neq 0$4, spatial support is governed by $v \neq 0$5, whereas temporal support is governed by $v \neq 0$6, the delay $v \neq 0$7, and the asymmetry induced by causality (Lindeberg, 2015). This suggests that ST-ERF should not be treated as a purely geometric cuboid in time-causal systems: its temporal axis has a distinct orientation toward the past and a characteristic reaction-time profile.

The time-causal limit kernel further introduces exact scale covariance over discrete temporal scales. With logarithmically distributed scales and $v \neq 0$8, the Fourier transform becomes

$v \neq 0$9

and the resulting kernel satisfies self-similar temporal scale covariance (Lindeberg, 2015). In ST-ERF terms, this gives a principled way to compare receptive fields across different event durations.

5. Motion adaptation, covariance, and space-time geometry

A major theoretical refinement is the treatment of ST-ERF under geometric image transformations. The generalized Gaussian derivative framework defines spatio-temporal kernels

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$0

and proves joint covariance under composed spatial scaling, spatial affine transformations, Galilean transformations, and temporal scaling (Lindeberg, 2023). Under

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$1

the parameters transform as

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$2

This means that matched ST-ERFs can be transported across changes in viewpoint, scale, speed, and local motion by appropriate parameter transformation (Lindeberg, 2023).

The geometric content of ST-ERF is especially clear in the joint space-time covariance matrix for the non-causal affine model. The hybrid Lie semi-group paper rewrites the kernel as a 3D Gaussian with covariance

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$3

Diagonal terms encode spatial and temporal variance, while off-diagonal space-time terms encode the tilt induced by motion (Lindeberg, 19 Sep 2025). In this formulation, the ST-ERF is a 3D Gaussian ellipsoid in $$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$4 whose principal axes reflect spatial anisotropy, temporal extent, and motion direction.

The same paper derives infinitesimal evolution equations over the parameter manifold. In the isotropic spatial case,

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$5

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$6

$$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$7

These relations describe how ST-ERFs deform continuously as spatial scale, temporal scale, or preferred velocity changes (Lindeberg, 19 Sep 2025). A plausible implication is that ST-ERF should be understood not merely as a single support volume but as a structured family evolving over a parameter space with explicit differential relations.

Motion selectivity analyses sharpen this view further. In the direction- and speed-selectivity paper, simple-cell spatio-temporal receptive fields are modeled as velocity-adapted affine Gaussian derivatives, and their effective support is described as a tilted, anisotropic region aligned with $$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$8 (Lindeberg, 11 Nov 2025). Direction and speed tuning then arise from overlap between stimulus motion and the ST-ERF’s preferred space-time orientation. This makes the tilt parameter $$\mathrm{ERF}^{(\mathcal{S},\mathcal{T})}_{(i,j)}[\,y_{(m,n)}[t], \tau; \mathbf{x}\,] = \frac{\partial y_{(m,n)}[t]}{\partial x_{(i,j)}[t-\tau]},$$9 a defining geometric attribute of motion-sensitive ST-ERFs.

6. Gradient-defined ST-ERF in modern neural systems

In gradient-based practice, ST-ERF is measured rather than prescribed. The SNN analysis paper begins from the classical ANN ERF

$R_k$0

and extends it to spiking models with discrete time: $R_k$1 This turns ST-ERF into a spatial-by-delay tensor indexed by output location and time (Zhang et al., 24 Oct 2025).

The same paper defines two useful marginals. The spatial ERF of an SNN layer averages the full spatio-temporal derivative tensor over time and delays with a weighting function $R_k$2, while the temporal ERF sums spatial contributions at the final timestep as a function of delay $R_k$3 (Zhang et al., 24 Oct 2025). The resulting analysis reveals that several Transformer-based SNNs “fail to establish a robust global ST-ERF,” with receptive fields that remain Gaussian-like and strongly centered rather than globally spread. On that basis, the paper proposes MLPixer and SRB channel mixers, whose effect is to enhance global spatial ERF through all timesteps in early network stages (Zhang et al., 24 Oct 2025).

This gradient-defined view closely parallels the now-standard distinction between theoretical and effective receptive field in CNNs. RF-Next explicitly cites the observation that “although the theoretical receptive field could be huge, the effective receptive field occupies a fraction of the theoretical receptive field” and then searches dilation schedules accordingly (Gao et al., 2022). This suggests that ST-ERF can function as a diagnostic object for model design: measured influence distributions can reveal whether a nominally large spatio-temporal field is actually used.

A related but distinct line arises in systems neuroscience, where spatio-temporal receptive fields are estimated statistically from data. The spline-basis STRF paper treats spatio-temporal receptive fields as linear filters $R_k$4 in linear-Gaussian, LNP, and LNLN models and imposes smoothness and sparsity by representing them in a natural cubic spline basis (Huang et al., 2021). Although that work does not use the phrase ST-ERF in the same formal sense, it describes estimated STRFs as the effective region in space-time where inputs matter for a neuron’s response, emphasizing smoothness, sparsity, confidence intervals, and significance tests (Huang et al., 2021). This suggests a complementary, estimation-theoretic interpretation of ST-ERF as a data-driven characterization of the space-time support of sensory computation.

7. Applications, misconceptions, and broader significance

ST-ERF has direct consequences for architecture design, biological modeling, and interpretation. In CNNs and video networks, knowing the ST-ERF per layer allows the selection of kernel sizes, strides, and dilations to match the spatio-temporal scale of a task, such as long-term action recognition or spatio-temporal localization (Le et al., 2017). In receptive-field search, learned dilation patterns alter the hierarchy of temporal and spatial fields across depth, often outperforming monotonic hand-designed schedules (Gao et al., 2022). In time-causal scale-space, parameter choices $R_k$5 explicitly control spatial width, temporal width, delay, and asymmetry (Lindeberg, 2015). In spiking networks, ST-ERF analysis identifies when theoretical global capacity fails to become an actual global influence field (Zhang et al., 24 Oct 2025).

Several misconceptions recur in the literature. One is to equate all uses of “effective receptive field.” The CNN geometry paper uses ERF in a geometric/combinatorial sense, namely the set of input positions that can influence a neuron (Le et al., 2017). By contrast, gradient-based works and the SNN ST-ERF paper use derivative magnitude to characterize actual influence distributions (Zhang et al., 24 Oct 2025). Another misconception is to treat temporal extent as symmetric around the present. In time-causal formulations, temporal support is one-sided, delayed, and skewed, not Gaussian in the non-causal sense (Lindeberg, 2015). A third is to regard spatial and temporal scale as separable in all cases; motion-adapted kernels explicitly couple them through $R_k$6 and through the joint covariance structure (Lindeberg, 2023).

Across frameworks, ST-ERF serves as a unifying object. In geometric CNN analysis it is the input volume that can affect a unit; in scale-space theory it is the anisotropic, possibly velocity-adapted Gaussian tube over which a receptive field integrates; in motion analysis it is the tilted space-time support that determines direction and speed tuning; in SNNs it is the gradient-defined influence field across past timesteps; and in sensory-neural estimation it is the smooth, sparse spatio-temporal filter recovered from data (Lindeberg, 2017). Taken together, these strands show that ST-ERF is not a single metric but a family of closely related constructs for quantifying the spatial extent, temporal depth, and motion alignment of effective computation in spatio-temporal systems.