Long-Short Term Temporal Convolution (LSTC)

• Long-Short Term Temporal Convolution (LSTC) is a dual-branch module that preserves both local motion patterns and long-range temporal dependencies via adaptive fusion.
• It replaces conventional stride-2 convolutions by combining a dense short-term branch with a sparse long-term branch to maintain fine-grained action cues.
• LSTC leverages learned cosine similarity for feature alignment, contributing to state-of-the-art results in skeleton-based action recognition benchmarks.

Searching arXiv for the cited LSTC-MDA paper and closely related skeleton-based action recognition work for accurate citation support. Long-Short Term Temporal Convolution (LSTC) is a temporal modeling module for skeleton-based action recognition introduced as part of the unified LSTC-MDA framework, with the specific purpose of preserving both local motion cues and long-range temporal dependencies during temporal downsampling (Ding et al., 18 Sep 2025). It addresses a recurrent weakness of conventional stride-2 temporal convolutions: while such convolutions are effective for local temporal modeling, they tend to discard long-range cues that are important for fine-grained action discrimination. In LSTC, temporal downsampling is reformulated as a two-branch process in which short-term and long-term temporal features are extracted in parallel, aligned in feature space, and fused adaptively through learned similarity weights. The module is presented as a replacement for standard temporal downsampling convolutions, and in the reported framework it contributes to state-of-the-art performance on NTU 60, NTU 120, and NW-UCLA (Ding et al., 18 Sep 2025).

1. Conceptual motivation and problem setting

Skeleton-based action recognition requires models to capture both short-term and long-term temporal dependencies. The short-term component corresponds to local, neighbor-to-neighbor temporal evolution, while the long-term component corresponds to cues distributed across a much larger temporal span. The motivation for LSTC is the claim that conventional temporal downsampling, exemplified by temporal convolutions with kernel size 7 and stride 2, emphasizes the former at the expense of the latter (Ding et al., 18 Sep 2025).

Within the described setting, this trade-off is especially problematic for complex and subtle actions. The motivating example given is the distinction between actions such as “put on shoe” and “take off shoe,” where decisive evidence may be distributed across distant parts of the sequence rather than concentrated in a local temporal neighborhood (Ding et al., 18 Sep 2025). The module is therefore designed not merely to enlarge receptive field in a generic sense, but to preserve long-range cues explicitly during the act of temporal reduction.

This positioning distinguishes LSTC from approaches that treat downsampling as a purely local operation. A plausible implication is that LSTC should be understood less as a general-purpose temporal block and more as a targeted intervention at the point where standard stride-2 temporal convolutions lose information. In the paper’s formulation, the central claim is that long-range temporal evidence should remain available even after the sequence resolution is halved (Ding et al., 18 Sep 2025).

2. Architectural structure

LSTC is defined as a two-branch parallel structure inserted in place of a standard temporal downsampling convolution (Ding et al., 18 Sep 2025). The input is

$X \in \mathbb{R}^{C \times T \times V},$

where $C$ denotes channels, $T$ the temporal length, and $V$ the number of joints. The module produces two temporally downsampled feature maps, one from a short-term branch and one from a long-term branch, each of shape

$\mathbb{R}^{C \times T/2 \times V}.$

The short-term branch uses a standard $7 \times 1$ convolution with stride $(2,1)$ over time and joints. Its purpose is to capture local temporal patterns, and its output is denoted

$F_s \in \mathbb{R}^{C \times T/2 \times V}.$

This branch preserves the conventional inductive bias of local temporal convolution and therefore serves as the module’s dense local descriptor (Ding et al., 18 Sep 2025).

The long-term branch uses a sparse large-kernel convolution. Rather than densely attending across the full temporal window, it covers the entire temporal window while retaining learnable weights only at the first 3 and last 3 positions; all intermediate kernel weights are zero. Its output is

$F_l \in \mathbb{R}^{C \times T/2 \times V}.$

The stated purpose is to capture global, long-range dependencies with minimal added computation (Ding et al., 18 Sep 2025).

The two branches are not simply concatenated or summed. Instead, their outputs are first aligned through linear projections and then fused through similarity-based weighting. This design makes LSTC a structured fusion module rather than a mere multi-branch convolution. In that sense, the module combines architectural asymmetry in temporal coverage with adaptive fusion in feature space.

3. Sparse long-term branch and temporal receptive design

The long-term branch is the distinctive component of LSTC. For each output timestep, the filter spans a temporal window of size $T/2 + 3$, but only six positions are learnable: the first three and the last three. The operation is expressed as

$C$0

where

$C$1

Thus, only the beginning and end of the temporal window are considered (Ding et al., 18 Sep 2025).

The paper characterizes this branch as parameter-efficient. Although the total kernel size is $C$2, the learnable parameter tensor has shape

$C$3

because only six temporal positions are active (Ding et al., 18 Sep 2025). The resulting design couples a very large effective temporal span with a small number of learned weights.

The kernel selection is not arbitrary. The reported ablation studies state that the “first 3 / last 3” sparse kernel achieves the best trade-off between parameter efficiency and accuracy, while denser alternatives such as “every other frame” harm fusion by creating a mismatch with the short-term branch (Ding et al., 18 Sep 2025). This suggests that the long-term branch is not intended to be a generic long-range sampler, but a deliberately structured complement to the local $C$4 branch.

A plausible implication is that the module’s success depends partly on branch complementarity rather than only on branch coverage. The short-term branch models contiguous local evolution; the long-term branch samples distal endpoints of a broad temporal window. Their combination yields a downsampled representation informed by both local continuity and large-span boundary cues.

4. Alignment, similarity estimation, and adaptive fusion

After branch-specific feature extraction, both $C$5 and $C$6 are passed through respective linear projection layers, yielding

$C$7

These are position-wise, channel-mixing linear layers whose role is to align the feature spaces of the two branches for similarity computation (Ding et al., 18 Sep 2025).

The first similarity term is the cosine similarity between aligned short-term and long-term features: $C$8 This term encourages stronger fusion when the local and global branch features are similar (Ding et al., 18 Sep 2025).

A second similarity term introduces a learnable reference matrix

$C$9

and computes

$T$0

The paper states that this component is inspired by LSTFE-Net and acts as an adaptive bias to modulate how much global context should be used at each location (Ding et al., 18 Sep 2025).

The total similarity weight is then

$T$1

Fusion is defined by

$T$2

Here, $T$3 denotes elementwise multiplication. The interpretation given is that the output remains predominantly short-term unless global cues are confidently supported by the similarity structure, in which case long-term features are fused adaptively into the downsampled representation (Ding et al., 18 Sep 2025).

This fusion mechanism is important because LSTC does not treat long-range information as uniformly beneficial. Instead, the long-term branch is gated through branch agreement and the learnable reference term. A plausible implication is that the module is designed to avoid indiscriminate injection of global context, which could otherwise destabilize or blur the local temporal signal during downsampling.

5. Mathematical formulation and comparison with conventional downsampling

The LSTC module is summarized by the following operations: $T$4

$T$5

where only the first 3 and last 3 kernel weights are nonzero,

$T$6

$T$7

and

$T$8

These equations define LSTC as a downsampling operator in which local temporal convolution is augmented by sparse long-range convolution and similarity-based feature fusion (Ding et al., 18 Sep 2025).

The contrast with a conventional stride-2 temporal convolution is explicit. A standard stride-2 convolution provides short temporal context over neighboring 7 frames and discards long-term information during downsampling. By contrast, LSTC preserves both short-term context and long-span cues through the combined effect of its parallel branches and learned fusion (Ding et al., 18 Sep 2025).

The illustrative example given is a sequence of 32 frames. Under conventional stride-2 convolution, the output has 16 frames, each formed from a weighted sum of contiguous 7-frame neighborhoods. Under LSTC, the short-term branch behaves similarly, but the long-term branch computes each output timestep from only the first 3 and last 3 positions within the receptive window, after which fusion weights determine how much long-term content is retained (Ding et al., 18 Sep 2025).

This comparison clarifies that LSTC is not simply a wider convolution. Its long branch is sparse rather than dense, and the final output is modulated by similarity. The design therefore couples receptive-field extension, sparsity, and adaptive weighting in a single downsampling module.

6. Empirical role within LSTC-MDA

LSTC is introduced within LSTC-MDA, a unified framework that also incorporates mixed data augmentation. The framework addresses two stated challenges in skeleton-based action recognition: scarcity of labeled training samples and difficulty modeling short- and long-range temporal dependencies (Ding et al., 18 Sep 2025). LSTC targets the second of these directly, while the broader framework also extends Joint Mixing Data Augmentation with an Additive Mixup at the input level and restricts mixup operations to the same camera view to avoid distribution shifts (Ding et al., 18 Sep 2025).

The reported ablation results quantify the contribution of LSTC within the SkateFormer-R backbone. The baseline “Raw SkateFormer-R (w/o LSTC)” achieves 92.42%. Adding “LSTC only” yields 92.58%, corresponding to a gain of $T$9. Adding “LSTC + JMDA” yields 93.24% ($V$0), and the “Full LSTC-MDA” reaches 93.36% ($V$1) (Ding et al., 18 Sep 2025). These figures are presented as evidence that each component contributes.

At the full framework level, LSTC-MDA achieves 94.1% and 97.5% on NTU 60 (X-Sub and X-View), 90.4% and 92.0% on NTU 120 (X-Sub and X-Set), and 97.2% on NW-UCLA (Ding et al., 18 Sep 2025). Because these results belong to the integrated framework rather than the isolated LSTC module, they should be interpreted as system-level outcomes rather than direct measurements of LSTC alone.

The paper also summarizes LSTC’s empirical effect in qualitative terms: it preserves context, better retrieves information from distant frames, boosts accuracy particularly in fine-grained, long-lasting actions where short-term cues are ambiguous, and adds only a very modest parameter count and computation due to the sparse kernel (Ding et al., 18 Sep 2025). This suggests that the module is intended as a low-overhead temporal substitute for standard downsampling rather than as a heavy long-range reasoning block.

7. Scope, related uses of the acronym, and interpretive considerations

Within the present topic, LSTC specifically denotes Long-Short Term Temporal Convolution as defined in the skeleton-based action recognition framework LSTC-MDA (Ding et al., 18 Sep 2025). The acronym is not unique in the literature. It has also been used for “Long and Short-Term Constraints” in safe reinforcement learning (Hu et al., 2024), “Long-Short Temporal Co-teaching” in weakly supervised video anomaly detection (Sun et al., 2023), and “Long-Short Term Context” in atomic action detection (Li et al., 2021). These are distinct formulations that share the long/short temporal motif but not the same mechanism.

This multiplicity of usage can create terminological confusion. In the temporal-convolution sense, LSTC refers to a module with parallel short-term and long-term convolutional branches, sparse large-kernel design, linear alignment, cosine-similarity weighting, and adaptive fusion during temporal downsampling (Ding et al., 18 Sep 2025). It should therefore not be conflated with methods that use separate temporal networks, co-teaching schemes, constraint formulations, or long/short context decomposition under different architectures.

A common misconception would be to treat LSTC as simply a larger temporal kernel. The formulation shows otherwise: the long-term branch is sparse rather than dense, the short-term branch remains explicitly local, and the final fusion depends on learned similarity rather than unconditional aggregation (Ding et al., 18 Sep 2025). Another possible misconception would be to interpret the module as a standalone recognition architecture. In the paper, it is a module inserted into a larger backbone and evaluated as part of a unified framework.

Taken as a design pattern, LSTC embodies a specific claim about temporal downsampling in skeleton-based action recognition: downsampling should preserve explicitly modeled long-range cues rather than rely solely on local stride-2 convolution. The reported ablations and benchmark results support that claim within the LSTC-MDA setting (Ding et al., 18 Sep 2025). A plausible implication is that the module’s broader relevance lies in rethinking temporal reduction as a selective fusion problem rather than as a purely local filtering step.