Multi-Grained Spatial-Temporal Feature Complementarity

• MGSTC is defined as an architectural design that extracts multi-resolution spatial and temporal features and fuses them to preserve complementary information for downstream tasks.
• The technique employs parallel multi-branch extraction, alignment before fusion, and task-aware complementarity to balance fine- and coarse-grained cues.
• Empirical results across domains such as sound event detection, graph-based action recognition, and video deduplication show improved performance with lower parameter overhead.

Searching arXiv for the cited papers and closely related MGSTC formulations. Multi-Grained Spatial-Temporal Feature Complementarity (MGSTC) denotes an architectural principle in which representations are extracted at multiple spatial and temporal grains and then fused so that fine-scale and coarse-scale cues remain jointly available to the downstream task. In the cited literature, this principle appears in several domain-specific forms: multi-scale spectral-spatial-temporal feature extraction for sound event localization and detection in MFF-EINV2 (Mu et al., 2024), complementary fixed and trainable spatio-temporal graph filtering in ST-GCSN (Cheng et al., 2021), local/global spatial and unified/independent temporal decomposition in MVFN for traffic demand prediction (Zhang et al., 2024), and coarse retrieval combined with fine spatial-temporal matching in MLT-Dedup for online video deduplication (Wang et al., 10 Jun 2026).

1. Conceptual definition and scope

The most explicit definition in the supplied literature appears in MVFN, where MGSTC is written as

$$\text{MGSTC} = \text{“extract local spatial + global spatial features”} \;\;\oplus\;\; \text{“extract unified temporal + independent temporal features,”}$$

with $\oplus$ denoting complementary fusion (Zhang et al., 2024). In that formulation, complementarity is not merely the coexistence of several branches; it is the requirement that distinct grains reinforce one another so that no important fine- or coarse-grained pattern is lost.

The same principle is instantiated differently in other domains. In MFF-EINV2, the relevant grains are spectral resolution, inter-channel spatial cues, and temporal context lengths; the model uses parallel subnetworks and a TF-Convolution Module to extract multi-scale spectral, spatial, and temporal features (Mu et al., 2024). In ST-GCSN, the grains are graph neighborhoods and temporal correlations indexed by scattering scales, while complementarity is realized by combining a mathematically designed scattering branch with a trainable residual branch (Cheng et al., 2021). In MLT-Dedup, a closely related formulation appears under the phrase “Multi-Granular Spatial-Temporal Complementarity,” where coarse clip-level representations are used for high-recall retrieval and fine frame-level plus differential features are used for precise localization and verification (Wang et al., 10 Jun 2026).

This suggests that MGSTC is best understood as a design pattern rather than a single fixed layer. The “spatial” component may refer to mel-frequency structure and inter-channel localization cues, neighborhoods on a skeleton graph, relations among traffic nodes, or framewise video content, depending on the application domain.

2. Recurrent architectural motifs

Across the cited systems, MGSTC is realized through a small set of recurring motifs. The first is parallel or multi-branch extraction. MFF-EINV2 spawns $s=3$ parallel resolution branches by frequency-domain downsampling, ST-GCSN creates fixed and complementary children for each surviving scale index, MVFN separates GCN from cosine re-weighting linear attention in space and MTCN from STCN in time, and MLT-Dedup produces both sparse clip-level and dense frame-level embeddings (Mu et al., 2024, Cheng et al., 2021, Zhang et al., 2024, Wang et al., 10 Jun 2026).

The second motif is alignment before fusion. In MFF-EINV2, cross-branch fusion uses FD or FU operators so that features from different resolutions lie in aligned tensor spaces before being added. In MVFN, local and global spatial streams are concatenated and projected back to a common embedding dimension, while temporal branches are fused by summation. In ST-GCSN, complementarity is index-aligned: for each surviving $(j_1,j_2)$, the model retains both the fixed node and its complementary trainable counterpart. In MLT-Dedup, alignment occurs at the level of a query–candidate similarity tensor $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$, where static and differential cues are stacked channelwise for localization (Mu et al., 2024, Cheng et al., 2021, Zhang et al., 2024, Wang et al., 10 Jun 2026).

The third motif is task-aware complementarity rather than uniform scaling. The grains that are preserved are chosen to match the structure of the task. MFF-EINV2 emphasizes spectral-spatial-temporal coverage for SELD; ST-GCSN uses graph wavelet scales and trainable off-band residual filters; MVFN separates “unified” from “independent” nodewise temporal dynamics; MLT-Dedup separates recall-oriented retrieval from precision-oriented verification (Mu et al., 2024, Cheng et al., 2021, Zhang et al., 2024, Wang et al., 10 Jun 2026).

System	Spatial or feature grains	Temporal grains
MFF-EINV2	high / medium / low spectral resolution with spectral and spatial views	dilated TFCM blocks from short to very long context
ST-GCSN	graph neighborhoods of radius $2^{j_1}$	correlations over $2^{j_2}$ time-steps
MVFN	local GCN and global CLA	unified MTCN and independent STCN
MLT-Dedup	sparse clip-level and fine frame-level embeddings	static and differential frame dynamics

A common misconception is to equate MGSTC with generic multi-scale processing. The cited systems indicate a narrower meaning: complementarity requires explicit mechanisms that preserve and combine nonredundant grains, rather than merely deepening a single branch or enlarging a receptive field.

3. Spectral-spatial-temporal complementarity in MFF-EINV2

In MFF-EINV2, the Multi-scale Feature Fusion module operates on an input tensor $X \in \mathbb R^{C \times T \times F}$ obtained by concatenating log-mel spectrograms and intensity vectors. The module creates three parallel branches by frequency-domain downsampling:

$$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$

For each branch $i$, spectral and spatial features are denoted

$\oplus$0

where the two 2D convolutions cover local versus global spectral patterns and jointly encode inter-channel cues. Temporal modeling is then performed by a TF-Convolution Module with $\oplus$1 consecutive blocks, each of the form PConv→DWConv→PConv→BN→ReLU. The depth-wise convolution uses kernel size $\oplus$2, dilation $\oplus$3 on the time axis, and dilation $\oplus$4 on frequency, giving a temporal receptive field

$\oplus$5

With $\oplus$6, the module simultaneously captures short context at $\oplus$7 with $\oplus$8 and very long context at $\oplus$9 with $s=3$0 (Mu et al., 2024).

Complementarity is reinforced by repeated fusion across branches. With three fusion stages, a prototypical stage-3 update for branch 1 is

$s=3$1

where the upsampling operator first uses a $s=3$2 convolution to align channel dimensions and then upsamples frequency by nearest neighbor. More generally,

$s=3$3

with $s=3$4 selected as an FD or FU operator so that all terms lie in the same tensor space. The walkthrough notes that the paper uses simple addition, although a compact weighted form

$s=3$5

can be written to illustrate the same principle.

The empirical evidence reported for this design is concrete. On the 2022 STARSS22 test set, MFF-EINV2 attains $s=3$6, compared with $s=3$7 for the 2022 baseline, $s=3$8 for ResNet-Conformer, $s=3$9 for CST-former, and $(j_1,j_2)$0 for EINV2. The model uses $(j_1,j_2)$1 M parameters, which is reported as $(j_1,j_2)$2 fewer than EINV2’s $(j_1,j_2)$3 M. On STARSS23 it attains $(j_1,j_2)$4. The ablation studies are equally central to the MGSTC interpretation: varying the number of parallel subnetworks gives $(j_1,j_2)$5 for $(j_1,j_2)$6, $(j_1,j_2)$7 for $(j_1,j_2)$8, $(j_1,j_2)$9 for $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$0, $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$1 for $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$2, and $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$3 for $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$4; varying the number of TFCM blocks gives $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$5 for $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$6, $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$7 for $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$8, $$\mathcal S \in \mathbb R^{T_Q \times T_K \times (2D_s+2)}$$9 for $2^{j_1}$0, $2^{j_1}$1 for $2^{j_1}$2, and $2^{j_1}$3 for $2^{j_1}$4. These results confirm, in the paper’s own summary, that three spectral-spatial resolutions plus six temporal scales yield the best MGST complementarity (Mu et al., 2024).

4. Graph-wavelet complementarity in ST-GCSN

ST-GCSN formulates spatio-temporal complementarity on graphs rather than Euclidean grids. The spatial graph is $2^{j_1}$5 with adjacency $2^{j_1}$6 and degree matrix $2^{j_1}$7, while the temporal graph is the chain graph $2^{j_1}$8 with adjacency $2^{j_1}$9 and degree $2^{j_2}$0. The model defines lazy-random-walk shifts

$2^{j_2}$1

The root signal is $2^{j_2}$2. A scattering layer generates children indexed by spatial scale $2^{j_2}$3 and temporal scale $2^{j_2}$4, applies a pointwise modulus $2^{j_2}$5, and builds an exponentially growing tree of spatio-temporal features. To control this growth, ST-GCSN uses energy-based pruning: for parent node $2^{j_2}$6 and child node $2^{j_2}$7, the subtree at $2^{j_2}$8 is pruned if

$2^{j_2}$9

with $X \in \mathbb R^{C \times T \times F}$0 in practice (Cheng et al., 2021).

The complementary mechanism enters through the “Graph Complementary Scattering Layer.” For each surviving index $X \in \mathbb R^{C \times T \times F}$1, the fixed branch computes

$X \in \mathbb R^{C \times T \times F}$2

while the trainable branch computes

$X \in \mathbb R^{C \times T \times F}$3

The trainable transition matrices $X \in \mathbb R^{C \times T \times F}$4 and $X \in \mathbb R^{C \times T \times F}$5 are initialized from $X \in \mathbb R^{C \times T \times F}$6 and $X \in \mathbb R^{C \times T \times F}$7 and are constrained by row-normalization through an agent parameter $X \in \mathbb R^{C \times T \times F}$8:

$X \in \mathbb R^{C \times T \times F}$9

All trainable branches in the same layer share the same $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$0 and $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$1. The final representation concatenates the fixed and complementary features across all layers and feeds them to a two-layer MLP with cross-entropy loss over 45 classes (Cheng et al., 2021).

The paper’s own interpretation of MGSTC is explicit. Spatial granularity corresponds to neighborhoods of radius $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$2 on the skeleton graph, temporal granularity corresponds to correlations over $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$3 time-steps, the fixed branch covers predefined wavelet bands, and the trainable branch learns residual filters “off the wavelet bands.” The pruning mechanism removes low-energy paths, after which the complementary branch reallocates model capacity on the surviving scale combinations.

The reported validation uses FPHA, with 1175 videos, 45 action categories, 21 hand-joint 3D coordinates per frame, 600 sequences for training, and 575 for testing. Sequence length is clipped or padded to 200 frames and uniformly sampled to 67 frames. With $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$4, $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$5, and $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$6, the model has $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$7 filters per layer. The pruning threshold preserves approximately 2,693 fixed nodes per layer, hence 5,386 total nodes including complementary ones. Quantitatively, ST-GCN achieves 86.32 accuracy, ST-GST 87.19, and ST-GCSN 88.75. The complementarity ablation reports 87.19 for fixed only, 87.82 for trainable only, 86.26 without explicit $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$8 complement, and 88.75 for the full complementary model. A purely trainable GCN with 13.27 M parameters reaches 88.99, whereas ST-GCSN reaches 88.75 with 0.23 M parameters, described as approximately 50 times fewer trainable parameters (Cheng et al., 2021).

5. Local/global and unified/independent decomposition in MVFN

MVFN provides a particularly clear decomposition of MGSTC into separate spatial and temporal complements. Spatial local features are extracted by a two-layer GCN. With adjacency $$X_1 = X,\qquad X_2 = FD_4(X) \in \mathbb R^{2C \times T \times F/4},\qquad X_3 = FD_{16}(X) \in \mathbb R^{4C \times T \times F/16}.$$9, identity $i$0, $i$1, degree matrix $i$2, and hidden representation $i$3, the GCN update is

$i$4

and after $i$5 layers the local spatial representation is $i$6.

Global spatial features are extracted by cosine re-weighting linear attention. From $i$7, the model forms queries, keys, and values by

$i$8

applies $i$9, and uses cosine and sine re-weighting by relative node-index distance. The resulting linearized attention is written as

$\oplus$00

The Graph-Cosine Module then fuses local and global streams through concatenation and projection:

$\oplus$01

This is the spatial half of the complementarity (Zhang et al., 2024).

The temporal half is implemented by the Multi-Channel Separable Temporal Convolutional Network. After transposing $\oplus$02 to a time-first tensor $\oplus$03 with $\oplus$04, MVFN applies four stacked dilated-causal layers. Each layer contains two parallel branches. The Multi-Channel Temporal CNN uses standard 1D convolutions with groups $\oplus$05, kernel size $\oplus$06, and dilations $\oplus$07 to model unified temporal patterns:

$\oplus$08

The Separable TCNN uses groups $\oplus$09 so that each channel is convolved independently:

$\oplus$10

The two temporal branches are fused by simple addition:

$\oplus$11

An ST-Layer is therefore organized as

$\oplus$12

After two ST-Layers, the final embedding $\oplus$13 is collapsed and passed to a two-layer prediction MLP:

$\oplus$14

Training uses MAE loss,

$\oplus$15

with no extra regularizer for complementarity; the complementarity is described as implicit in the architecture. The reported hyperparameters include 2 ST-Layers, 2 GCN layers, hidden dimension $\oplus$16 as an example, 1 CLA head with $\oplus$17 and $\oplus$18, 4 MSTCN layers, kernel size $\oplus$19, dilations $\oplus$20, batch size 64, learning rate $\oplus$21, Adam, and 100 epochs (Zhang et al., 2024).

The key significance of MVFN for MGSTC is conceptual clarity: local/global and unified/independent are treated as orthogonal decompositions, and fusion occurs at both the spatial and temporal stages rather than only at the output layer.

6. Coarse-to-fine complementarity in video deduplication

MLT-Dedup reinterprets MGSTC in a retrieval-and-matching pipeline for large-scale online video deduplication. Its Multi-Level Video Encoder takes a video $\oplus$22 and, in one forward pass, produces fine-grained frame-level embeddings $\oplus$23 and sparse clip-level embeddings $\oplus$24 with $\oplus$25. Using Swin-Large as backbone $\oplus$26 and a Perceiver Resampler $\oplus$27, the extraction is

$\oplus$28

followed by non-overlapping clip aggregation

$\oplus$29

The training objective is a hybrid

$\oplus$30

with details deferred to the paper (Wang et al., 10 Jun 2026).

Complementarity is distributed across the three stages of the deduplication pipeline. Stage 1 indexes only the sparse clip embeddings in an HNSW graph; with approximately 5 times fewer embeddings per video, the system can hold approximately 5 times more videos in the index under fixed memory. For each query clip embedding, the system retrieves top-$\oplus$31 nearest clip indices, maps them to video IDs, thresholds per-clip scores below $\oplus$32, averages the remaining scores, and returns top-$\oplus$33 videos. Stage 2 then loads frame embeddings only for the retrieved candidates and applies DiF-SiM for fine-grained matching (Wang et al., 10 Jun 2026).

DiF-SiM adds a second temporal grain through differential features:

$\oplus$34

For each query-frame–key-frame pair $\oplus$35, it computes learned similarities

$\oplus$36

together with cosine similarities

$\oplus$37

These are stacked into

$\oplus$38

which a YOLOX-s detector treats as an image and regresses into temporal boxes $\oplus$39 with confidence scores. A pair is accepted only if overlap proportions exceed a policy threshold, for example at least 50% coverage, and the score exceeds a second threshold (Wang et al., 10 Jun 2026).

The reported operational outcomes are substantial: MLT-Dedup reduces online repetition rates by 91% at 90% precision, its sparse retrieval design achieves a 5x increase in indexing capacity, and the authors relate the method explicitly to a generic MGSTC framework in which coarse representations support recall and fine representations support precision. This broadens the meaning of complementarity from within-network fusion to pipeline-level division of labor between retrieval and verification (Wang et al., 10 Jun 2026).

7. Interpretive synthesis, misconceptions, and transfer

Taken together, the cited works show that MGSTC does not prescribe a single fusion operator, a single modality, or even a single stage of processing. MFF-EINV2 uses repeated cross-scale addition after FD/FU alignment; ST-GCSN uses explicit fixed-versus-complementary branches indexed by the same scattering paths; MVFN uses concatenation plus projection in space and summation in time; MLT-Dedup separates complementarity across retrieval and matching stages (Mu et al., 2024, Cheng et al., 2021, Zhang et al., 2024, Wang et al., 10 Jun 2026).

A second misconception is that complementarity is equivalent to adding more scales. The MFF-EINV2 ablations argue against that simplification: performance improves up to $\oplus$40 branches and $\oplus$41 TFCM blocks, but degrades at $\oplus$42 and $\oplus$43, with the paper attributing the $\oplus$44 case to “overfitting / wasted capacity” (Mu et al., 2024). In ST-GCSN, the best result does not come from trainability alone; the ablation without explicit $\oplus$45 complement falls to 86.26, below both fixed-only and trainable-only variants (Cheng et al., 2021). In MVFN, complementarity is architectural and needs no extra regularizer, which indicates that the preservation of multiple grains can be encoded directly in network topology (Zhang et al., 2024). In MLT-Dedup, a single scalar similarity is treated as insufficient; Deep-Sim fuses heterogeneous static and differential cues because static appearance and local motion provide different evidence under partial edits or speed changes (Wang et al., 10 Jun 2026).

The supplied literature also points to transfer beyond the original tasks. The MFF-EINV2 walkthrough states that the MGSTC principle can be ported directly to video action recognition, multimodal sensor fusion, and time-series forecasting, specifically through multi-resolution spatial streams, dilated temporal convolutions, and repeated cross-branch residual fusion (Mu et al., 2024). This suggests that the enduring content of MGSTC is not any one module but the disciplined preservation of complementary grains across spatial and temporal structure.