Spatial-Temporal Feature Enhancement (STFE)

• STFE is a design paradigm that explicitly enhances video features by jointly refining spatial structures and temporal dynamics using structured priors.
• It leverages techniques such as skeleton-guided refinement, deformable alignment, cross-attention, and multi-scale temporal-spatial pyramids to overcome modality discrepancies and occlusions.
• Applications in visible–infrared person re-identification, video enhancement, and traffic prediction demonstrate measurable gains in metrics like Rank-1 accuracy, mAP, PSNR, and SSIM.

Searching arXiv for the primary paper and a few related STFE papers to ground citations. Spatial-Temporal Feature Enhancement (STFE) denotes a class of methods that explicitly improve feature quality in video or time-sequential data by jointly strengthening spatial structure and temporal dynamics, rather than merely aggregating them. In the most concrete formulation provided here, STFE is exemplified by the STAR framework—“Skeleton-Guided Spatial-Temporal feAture leaRning”—for video-based visible–infrared person re-identification (VVI-ReID), where skeleton information is used as a structured, geometry-based prior to refine frame-level appearance features and sequence-level global representations under strong cross-modality discrepancy, low quality, and occlusion (Jiang et al., 2024). Across the broader literature cited here, STFE appears as a recurring design pattern: multi-scale temporal-spatial pyramids in video saliency detection (Chang et al., 2021), deformable spatio-temporal alignment in low-light video enhancement (Lin et al., 2024), progressive temporal feature alignment in video inpainting (Zou et al., 2021), spatial-temporal feature fusion for HDR video conversion (Xu et al., 2022), and double-layer extraction–evaluation architectures for traffic prediction (Pu et al., 2022). This suggests that STFE is best understood not as a single architecture, but as a general principle for improving the fidelity, discriminativeness, and temporal coherence of learned representations in dynamic visual and sequential systems.

1. Definition and conceptual scope

In the STAR formulation, STFE is described as a framework driven by skeleton information, where skeletons act as an additional modality that is structured, largely invariant to appearance, and more robust to low-quality imagery and occlusions (Jiang et al., 2024). Within that formulation, STFE has two explicit targets: frame-level features, where spatial enhancement is performed per frame, and sequence-level features, where temporal aggregation and body-part contribution learning are guided by a skeleton keypoint graph (Jiang et al., 2024).

The same source states that most previous VVI-ReID works mainly align modalities or aggregate temporal features, but do not explicitly correct or enhance spatial-temporal features themselves, especially in infrared video (Jiang et al., 2024). In that sense, STFE differs from simple temporal pooling, naive feature concatenation, or modality alignment alone. It emphasizes operations such as re-weighting, modulation, deformable alignment, graph-based aggregation, and attention-guided refinement that actively improve feature quality.

Across the other cited works, the term is not always used as a formal module name, yet the underlying idea recurs with consistent structure. In TSFP-Net for video saliency detection, a temporal-spatial feature pyramid enriches shallow feature maps with deeper temporal-spatial semantics through top-down integration (Chang et al., 2021). In STA-SUNet for low-light video enhancement, feature-level alignment across multiple frames precedes transformer-based spatial enhancement (Lin et al., 2024). In DSLNet for SDRTV-to-HDRTV conversion, dynamic multi-frame alignment, spatial-temporal feature modulation, and large-kernel quality enhancement form a staged pipeline of alignment, modulation, and refinement (Xu et al., 2022). A plausible implication is that STFE can be treated as a family of mechanisms that improve feature representations by explicitly coupling spatial organization with temporal context.

2. STFE in visible–infrared person re-identification

In VVI-ReID, the central difficulty is the strong cross-modality discrepancy between RGB and infrared videos. The visible modality offers rich color and texture but is sensitive to illumination and shadows, whereas infrared captures heat signatures and works in low light but is grayscale and weak in texture (Jiang et al., 2024). Spatial-temporal information is, in principle, more modality-robust because a person’s skeleton and motion are similar in RGB and IR, but its accuracy is degraded by low-quality IR videos, occlusions, viewpoint changes, illumination changes in visible videos, and background clutter (Jiang et al., 2024).

STAR addresses this by using skeletons as a modality-invariant structural prior (Jiang et al., 2024). At the frame level, skeleton edge features are extracted from joint coordinates $(x_i(t), y_i(t))$ through pairwise distances

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$

and relative angles

$$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$

which are then passed through a fully-connected layer and an LSTM to obtain temporal skeleton features $T_t$ (Jiang et al., 2024). These features guide visual features $X_t$ through two cross-attention directions: $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$

$H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$

A squeeze-and-excitation module produces channel weights $W_t$, and the refined frame feature is

$$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$

The stated interpretation is that $Y_t$ is a spatially enhanced frame feature in which channels and patterns consistent with skeleton geometry and motion are amplified, while channels likely due to noise, occlusion, or modality-specific artifacts are suppressed (Jiang et al., 2024).

At the sequence level, STAR constructs a spatial-temporal graph $d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$0 whose nodes are skeleton keypoints over time and whose edges include anatomical spatial edges within each frame and temporal edges across consecutive frames (Jiang et al., 2024). A Graph Attention Network updates node features by

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$1

and body-part contribution vectors are aggregated as

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$2

These contributions modulate Generalized Mean pooling,

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$3

to form the global sequence representation

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$4

The explicit claim is that this replaces naive average or max pooling with skeleton-driven, body-part-aware weighting, emphasizing motion patterns and pose configurations that are most discriminative and reliable across modalities (Jiang et al., 2024).

The training objective combines identity loss, triplet loss, SAADG’s style disturbance augmentation loss, and a skeleton consistency loss,

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$5

with

$d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$6

This bidirectional KL divergence aligns edge-based and joint-based skeleton streams and is described as encouraging consistency between two skeleton representations (Jiang et al., 2024).

On HITSZ-VCM, STAR reports I2V Rank-1 $d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$7, mAP $d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$8, and V2I Rank-1 $d_{ij}(t) = \sqrt{(x_j - x_i)^2 + (y_j - y_i)^2}$9, mAP $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$0, outperforming SAADG and CST in mAP and slightly improving Rank-1 (Jiang et al., 2024). Ablation results show that adding frame-level guidance alone yields $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$1 Rank-1 and $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$2 mAP over baseline, sequence-level guidance alone yields $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$3 Rank-1 and $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$4 mAP, and combining both yields $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$5 Rank-1 and $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$6 mAP (Jiang et al., 2024). This establishes a concrete instance in which STFE is operationalized as explicit feature correction and enhancement rather than simple aggregation.

3. Architectural motifs that recur across STFE systems

Several recurrent motifs appear across the cited works. One is multi-scale temporal-spatial fusion. In TSFP-Net, the encoder is fully 3D, so each feature level already encodes spatio-temporal patterns, and a top-down temporal-spatial feature pyramid integrates deep temporal-spatial semantic features into shallow feature maps (Chang et al., 2021). The generic update is presented as

$$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$7

with trilinear upsampling in time and space (Chang et al., 2021). The model’s ablation shows NSS improving from $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$8 with only the final level to $$\theta_{ij}(t) = \tan^{-1}\left(\frac{y_j - y_i}{x_j - x_i}\right),$$9 with only multi-level decoding, and to $T_t$0 with the full temporal-spatial feature pyramid (Chang et al., 2021). The stated interpretation is that shallow maps gain temporal semantics from deeper levels, while deeper maps provide coarse, long-range context.

A second motif is alignment before fusion. STA-SUNet uses EDVR-style Pyramid, Cascading, and Deformable alignment in feature space, with offsets

$T_t$1

and aligned features

$T_t$2

before Swin Transformer enhancement (Lin et al., 2024). The paper explicitly frames low-light video enhancement as requiring recovery of fine spatial details in each frame together with temporal consistency across frames (Lin et al., 2024). On BVI, STA-SUNet reports $T_t$3 against $T_t$4 for PCDUNet, $T_t$5 for UNet, and $T_t$6 for SUNet, and with 5 frames reaches PSNR $T_t$7, SSIM $T_t$8, compared with PSNR $T_t$9, SSIM $X_t$0 for 1 frame (Lin et al., 2024). This indicates that alignment-mediated temporal context is a primary STFE mechanism in restoration.

A third motif is modulation rather than simple fusion. DSLNet uses a dynamic multi-frame alignment module, a spatial-temporal feature modulation module, and a large-kernel quality enhancement module (Xu et al., 2022). In STFM, temporal, current-frame, and spatial modulation are all affine: $X_t$1

$X_t$2

$X_t$3

Removing alignment drops PSNR from $X_t$4 to $X_t$5 and $X_t$6 worsens from $X_t$7 to $X_t$8; removing modulation drops PSNR further to $X_t$9 and $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$0 to $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$1 (Xu et al., 2022). This supports the view that STFE often acts through feature-wise gating or modulation conditioned on spatial-temporal context.

A fourth motif is progressive temporal correction. In video inpainting, the Progressive Temporal Feature Alignment Network inserts 21 Temporal Shift Alignment Modules and achieves a temporal receptive field $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$2 frames (Zou et al., 2021). The aligned-fusion update is summarized as

$H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$3

with warped neighbor features

$H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$4

The paper reports approximately $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$5 increase in PSNR and $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$6 decrease in VFID versus vanilla TSM when alignment is introduced (Zou et al., 2021). This suggests that progressive alignment is another canonical STFE strategy when temporal aggregation alone is spatially misaligned.

4. Feature enhancement strategies: local, global, and evaluative

Some STFE formulations separate local enhancement from global evaluation. In DL-STFEE for traffic prediction, the lower layer performs spatial-temporal feature extraction using four graphs—topological, weighted topological, historical traffic pattern, and attribute similarity—combined with hourly, daily, and weekly temporal resolutions, yielding twelve feature combinations (Pu et al., 2022). The upper layer then applies high-dimensional self-attention across these combinations to evaluate and fuse them (Pu et al., 2022). The resulting attention score matrix serves both as a fusion mechanism and as an explicit measure of feature importance.

The paper reports that weighted topology consistently has the highest spatial importance, followed by historical traffic pattern, then topology, then attribute similarity; for short prediction horizons, hourly resolution dominates, whereas for longer horizons daily resolution becomes more important (Pu et al., 2022). Quantitatively, DL-STFEE achieves accuracy around $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$7 for $H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$8–$H_{t,t} = \mathrm{CrossAttn}(X_t, T_t, T_t),$9 hour horizons and $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$0 for $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$1 hours, with kappa around $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$2–$H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$3 for shorter horizons and $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$4 at $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$5 hours (Pu et al., 2022). This provides a different STFE interpretation: not merely enhancing raw features, but evaluating combinations of extracted spatial-temporal features and amplifying the most predictive ones.

A related but more restoration-oriented formulation appears in “Efficient Video Face Enhancement with Enhanced Spatial-Temporal Consistency” (Wang et al., 2024). There, a 3D-VQGAN learns separate spatial and temporal codebooks,

$H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$6

$H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$7

and fuses quantized spatial and temporal latents by addition (Wang et al., 2024). Temporal latents are defined as

$H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$8

with TA denoting inter-frame temporal attention and Residual denoting motion residuals (Wang et al., 2024). The stated effect is that temporal codes store residual-based motion information and reduce flicker by serving as consistent motion templates (Wang et al., 2024). On VFHQ-Test, the method reports SSIM $H_{o,t} = \mathrm{CrossAttn}(T_t, X_t, X_t).$9, IDS $W_t$0, FVD $W_t$1, Flow-Score $W_t$2, and runtime $W_t$3 seconds for 24 frames (Wang et al., 2024). Here STFE takes the form of discrete codebook priors over appearance and temporal change.

5. Domains and applications

The surveyed works demonstrate that STFE is not restricted to person re-identification. In video saliency detection, TSFP-Net uses a 3D fully convolutional encoder-decoder and temporal-spatial feature pyramid, reporting on DHF1K NSS $W_t$4, CC $W_t$5, and AUC-J $W_t$6, and on Hollywood-2 NSS $W_t$7, CC $W_t$8, AUC-J $W_t$9 (Chang et al., 2021). The paper attributes these gains to multi-scale temporal-spatial enhancement and hierarchical decoding.

In low-light video enhancement, STA-SUNet uses multi-frame spatio-temporal alignment and a Swin Transformer U-Net to recover detail and reduce flicker (Lin et al., 2024). In SDRTV-to-HDRTV conversion, DSLNet reports PSNR $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$0 dB, SR-SIM $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$1, and $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$2, improving over HDRTVNET’s $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$3, $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$4, and $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$5 respectively (Xu et al., 2022). In video inpainting, progressive temporal feature alignment improves both visual quality and temporal consistency (Zou et al., 2021). In video face enhancement, spatial-temporal codebooks and transformers improve both restoration and de-flickering while remaining efficient (Wang et al., 2024).

Beyond visual enhancement, DL-STFEE applies STFE principles to traffic prediction through multi-graph temporal feature combinations (Pu et al., 2022). A plausible implication is that STFE is domain-agnostic so long as the data exhibit structured spatial relations and nontrivial temporal dependence. The data block itself explicitly states, in the STAR context, that such principles are applicable beyond VVI-ReID to pure RGB video ReID under occlusion and viewpoint changes, action recognition, gait recognition, and cross-domain or cross-view video matching where appearance varies but geometry is stable (Jiang et al., 2024).

6. Common misconceptions and methodological boundaries

A common misconception is that temporal aggregation alone constitutes spatial-temporal enhancement. The cited works repeatedly distinguish enhancement from aggregation. STAR explicitly contrasts skeleton-guided correction with methods that only align modalities or aggregate temporal features (Jiang et al., 2024). DSLNet distinguishes alignment, modulation, and quality enhancement from naive multi-frame stacking, showing a multi-frame ResNet with $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$6M parameters reaches PSNR $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$7, whereas DSLNet with $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$8M parameters reaches $$Y_t = \mathrm{FFN}([H_{o,t}; H_{t,t}]) \otimes W_t + X_t + \epsilon.$$9 (Xu et al., 2022). In video inpainting, the critique is that 3D temporal convolution or TSM can suffer from spatial misalignment, so explicit alignment is required (Zou et al., 2021).

Another misconception is that spatial enhancement and temporal enhancement can always be optimized independently. Several papers suggest the opposite. STAR tightly couples pose-based spatial constraints with temporal skeleton patterns (Jiang et al., 2024). STINet explicitly argues that spatial correlation among different resolutions is as important as temporal correlation among frames and that both should interact during interpolation and refinement (Yue et al., 2022). DL-STFEE treats spatial-temporal feature combinations as objects that must themselves be evaluated and fused, rather than assuming one graph or one timescale is sufficient (Pu et al., 2022). This suggests that STFE usually requires mechanisms that bind spatial and temporal cues rather than separate them into isolated branches without interaction.

A further boundary concerns evidentiary scope. Some papers provide explicit ablations, formulas, and quantitative metrics that support concrete STFE claims, while others only offer a conceptual label. A rigorous usage of the term should therefore refer to explicit enhancement operations—such as cross-attention, graph attention, deformable alignment, modulation, codebook-based temporal priors, or evaluative attention over spatial-temporal combinations—rather than using STFE as a generic synonym for any video model.

7. Design principles and outlook

Several design principles emerge consistently from the cited works. First, structured priors improve robustness. STAR uses skeletons as a modality-invariant geometric prior (Jiang et al., 2024); STA-SUNet uses multi-scale feature alignment (Lin et al., 2024); video face enhancement uses spatial and temporal codebooks (Wang et al., 2024). Second, enhancement is often guided rather than uniform. Examples include skeleton-guided GeM pooling (Jiang et al., 2024), SE-style modulation in DSLNet (Xu et al., 2022), and attention-based evaluation of spatial-temporal feature combinations in DL-STFEE (Pu et al., 2022). Third, local and global refinement are frequently separated: local alignment or modulation addresses short-range consistency, while graph-based, recurrent, or transformer-style modules address sequence-level structure.

Fourth, multi-scale treatment is pervasive. TSFP-Net builds a temporal-spatial feature pyramid (Chang et al., 2021), STA-SUNet aligns features across a three-level pyramid (Lin et al., 2024), and STINet’s spatial-temporal frame interpolation and refinement operate jointly across low- and high-resolution streams (Yue et al., 2022). Fifth, temporally coherent supervision matters. STAR adds a bidirectional KL skeleton consistency loss (Jiang et al., 2024), STINet adds a motion consistency loss (Yue et al., 2022), and other systems derive temporal consistency implicitly through sequence modeling and codebook priors (Wang et al., 2024).

Taken together, these sources support a precise encyclopedia-level characterization: STFE is a design paradigm in which spatial organization and temporal evolution are jointly enhanced through explicit mechanisms that correct feature noise, improve alignment, emphasize discriminative regions or channels, and preserve motion continuity. In the STAR formulation, this means skeleton-guided refinement of frame and sequence features for visible–infrared re-identification (Jiang et al., 2024). In adjacent literatures, it means temporal-spatial pyramids (Chang et al., 2021), deformable alignment with transformer backbones (Lin et al., 2024), progressive alignment (Zou et al., 2021), affine feature modulation (Xu et al., 2022), graph-based extraction–evaluation layers (Pu et al., 2022), or spatial–temporal codebooks with transformer lookup (Wang et al., 2024). A plausible implication is that future STFE systems will continue to move away from uniform pooling or purely frame-wise enhancement and toward structured, interaction-driven representations that explicitly model how reliable spatial detail and temporally coherent dynamics should reinforce one another.