---
title: Long-Short Term Temporal Convolution (LSTC)
url: https://www.emergentmind.com/topics/long-short-term-temporal-convolution-lstc
type: topic
---

# Long-Short Term Temporal Convolution (LSTC)

Searching arXiv for the cited LSTC-MDA paper and closely related skeleton-based action recognition work for accurate citation support.
Long-Short Term Temporal Convolution (LSTC) is a temporal modeling module for skeleton-based action recognition introduced as part of the unified LSTC-MDA framework, with the specific purpose of preserving both local motion cues and long-range temporal dependencies during temporal downsampling [2509.14619]. It addresses a recurrent weakness of conventional stride-2 temporal convolutions: while such convolutions are effective for local temporal modeling, they tend to discard long-range cues that are important for fine-grained action discrimination. In LSTC, temporal downsampling is reformulated as a two-branch process in which short-term and long-term temporal features are extracted in parallel, aligned in feature space, and fused adaptively through learned similarity weights. The module is presented as a replacement for standard temporal downsampling convolutions, and in the reported framework it contributes to state-of-the-art performance on NTU 60, NTU 120, and NW-UCLA [2509.14619].

## 1. Conceptual motivation and problem setting

Skeleton-based action recognition requires models to capture both short-term and long-term temporal dependencies. The short-term component corresponds to local, neighbor-to-neighbor temporal evolution, while the long-term component corresponds to cues distributed across a much larger temporal span. The motivation for LSTC is the claim that conventional temporal downsampling, exemplified by temporal convolutions with kernel size 7 and stride 2, emphasizes the former at the expense of the latter [2509.14619].

Within the described setting, this trade-off is especially problematic for complex and subtle actions. The motivating example given is the distinction between actions such as “put on shoe” and “take off shoe,” where decisive evidence may be distributed across distant parts of the sequence rather than concentrated in a local temporal neighborhood [2509.14619]. The module is therefore designed not merely to enlarge receptive field in a generic sense, but to preserve long-range cues explicitly during the act of temporal reduction.

This positioning distinguishes LSTC from approaches that treat downsampling as a purely local operation. A plausible implication is that LSTC should be understood less as a general-purpose temporal block and more as a targeted intervention at the point where standard stride-2 temporal convolutions lose information. In the paper’s formulation, the central claim is that long-range temporal evidence should remain available even after the sequence resolution is halved [2509.14619].

## 2. Architectural structure

LSTC is defined as a two-branch parallel structure inserted in place of a standard temporal downsampling convolution [2509.14619]. The input is
\[
X \in \mathbb{R}^{C \times T \times V},
\]
where \(C\) denotes channels, \(T\) the temporal length, and \(V\) the number of joints. The module produces two temporally downsampled feature maps, one from a short-term branch and one from a long-term branch, each of shape
\[
\mathbb{R}^{C \times T/2 \times V}.
\]

The short-term branch uses a standard \(7 \times 1\) convolution with stride \((2,1)\) over time and joints. Its purpose is to capture local temporal patterns, and its output is denoted
\[
F_s \in \mathbb{R}^{C \times T/2 \times V}.
\]
This branch preserves the conventional inductive bias of local temporal convolution and therefore serves as the module’s dense local descriptor [2509.14619].

The long-term branch uses a sparse large-kernel convolution. Rather than densely attending across the full temporal window, it covers the entire temporal window while retaining learnable weights only at the first 3 and last 3 positions; all intermediate kernel weights are zero. Its output is
\[
F_l \in \mathbb{R}^{C \times T/2 \times V}.
\]
The stated purpose is to capture global, long-range dependencies with minimal added computation [2509.14619].

The two branches are not simply concatenated or summed. Instead, their outputs are first aligned through linear projections and then fused through similarity-based weighting. This design makes LSTC a structured fusion module rather than a mere multi-branch convolution. In that sense, the module combines architectural asymmetry in temporal coverage with adaptive fusion in feature space.

## 3. Sparse long-term branch and temporal receptive design

The long-term branch is the distinctive component of LSTC. For each output timestep, the filter spans a temporal window of size \(T/2 + 3\), but only six positions are learnable: the first three and the last three. The operation is expressed as
\[
y[n] = \sum_{c=1}^{C}\sum_{i \in I} w_{c,i} \cdot x[n + i],
\]
where
\[
I = \{0,1,2,T/2,T/2+1,T/2+2\}.
\]
Thus, only the beginning and end of the temporal window are considered [2509.14619].

The paper characterizes this branch as parameter-efficient. Although the total kernel size is \(T/2 + 3\), the learnable parameter tensor has shape
\[
(C, C, 6, 1),
\]
because only six temporal positions are active [2509.14619]. The resulting design couples a very large effective temporal span with a small number of learned weights.

The kernel selection is not arbitrary. The reported ablation studies state that the “first 3 / last 3” sparse kernel achieves the best trade-off between parameter efficiency and accuracy, while denser alternatives such as “every other frame” harm fusion by creating a mismatch with the short-term branch [2509.14619]. This suggests that the long-term branch is not intended to be a generic long-range sampler, but a deliberately structured complement to the local \(7 \times 1\) branch.

A plausible implication is that the module’s success depends partly on branch complementarity rather than only on branch coverage. The short-term branch models contiguous local evolution; the long-term branch samples distal endpoints of a broad temporal window. Their combination yields a downsampled representation informed by both local continuity and large-span boundary cues.

## 4. Alignment, similarity estimation, and adaptive fusion

After branch-specific feature extraction, both \(F_s\) and \(F_l\) are passed through respective linear projection layers, yielding
\[
\tilde{F}_s \quad \text{and} \quad \tilde{F}_l.
\]
These are position-wise, channel-mixing linear layers whose role is to align the feature spaces of the two branches for similarity computation [2509.14619].

The first similarity term is the cosine similarity between aligned short-term and long-term features:
\[
S_{sl} = \text{cosine}(\tilde{F}_s, \tilde{F}_l).
\]
This term encourages stronger fusion when the local and global branch features are similar [2509.14619].

A second similarity term introduces a learnable reference matrix
\[
\mu \in \mathbb{R}^{C \times T/2 \times V},
\]
and computes
\[
S_{\mu l} = \text{cosine}(\mu, \tilde{F}_l).
\]
The paper states that this component is inspired by LSTFE-Net and acts as an adaptive bias to modulate how much global context should be used at each location [2509.14619].

The total similarity weight is then
\[
S = S_{sl} + S_{\mu l}.
\]
Fusion is defined by
\[
x = F_s + S \odot F_l. \tag{1}
\]
Here, \(\odot\) denotes elementwise multiplication. The interpretation given is that the output remains predominantly short-term unless global cues are confidently supported by the similarity structure, in which case long-term features are fused adaptively into the downsampled representation [2509.14619].

This fusion mechanism is important because LSTC does not treat long-range information as uniformly beneficial. Instead, the long-term branch is gated through branch agreement and the learnable reference term. A plausible implication is that the module is designed to avoid indiscriminate injection of global context, which could otherwise destabilize or blur the local temporal signal during downsampling.

## 5. Mathematical formulation and comparison with conventional downsampling

The LSTC module is summarized by the following operations:
\[
F_s = \text{Conv}_{7,1}^{(s=2)}(X),
\]
\[
F_l = \text{SparseConv}_{T/2+3}(X),
\]
where only the first 3 and last 3 kernel weights are nonzero,
\[
\tilde{F}_s = \text{Linear}(F_s), \quad \tilde{F}_l = \text{Linear}(F_l),
\]
\[
S_{sl} = \text{cosine}(\tilde{F}_s, \tilde{F}_l), \quad S_{\mu l} = \text{cosine}(\mu, \tilde{F}_l),
\]
and
\[
x = F_s + (S_{sl} + S_{\mu l}) \cdot F_l.
\]
These equations define LSTC as a downsampling operator in which local temporal convolution is augmented by sparse long-range convolution and similarity-based feature fusion [2509.14619].

The contrast with a conventional stride-2 temporal convolution is explicit. A standard stride-2 convolution provides short temporal context over neighboring 7 frames and discards long-term information during downsampling. By contrast, LSTC preserves both short-term context and long-span cues through the combined effect of its parallel branches and learned fusion [2509.14619].

The illustrative example given is a sequence of 32 frames. Under conventional stride-2 convolution, the output has 16 frames, each formed from a weighted sum of contiguous 7-frame neighborhoods. Under LSTC, the short-term branch behaves similarly, but the long-term branch computes each output timestep from only the first 3 and last 3 positions within the receptive window, after which fusion weights determine how much long-term content is retained [2509.14619].

This comparison clarifies that LSTC is not simply a wider convolution. Its long branch is sparse rather than dense, and the final output is modulated by similarity. The design therefore couples receptive-field extension, sparsity, and adaptive weighting in a single downsampling module.

## 6. Empirical role within LSTC-MDA

LSTC is introduced within LSTC-MDA, a unified framework that also incorporates mixed data augmentation. The framework addresses two stated challenges in skeleton-based action recognition: scarcity of labeled training samples and difficulty modeling short- and long-range temporal dependencies [2509.14619]. LSTC targets the second of these directly, while the broader framework also extends Joint Mixing Data Augmentation with an Additive Mixup at the input level and restricts mixup operations to the same camera view to avoid distribution shifts [2509.14619].

The reported ablation results quantify the contribution of LSTC within the SkateFormer-R backbone. The baseline “Raw SkateFormer-R (w/o LSTC)” achieves 92.42%. Adding “LSTC only” yields 92.58%, corresponding to a gain of \(+0.16\%\). Adding “LSTC + JMDA” yields 93.24% (\(+0.82\%\)), and the “Full LSTC-MDA” reaches 93.36% (\(+0.94\%\)) [2509.14619]. These figures are presented as evidence that each component contributes.

At the full framework level, LSTC-MDA achieves 94.1% and 97.5% on NTU 60 (X-Sub and X-View), 90.4% and 92.0% on NTU 120 (X-Sub and X-Set), and 97.2% on NW-UCLA [2509.14619]. Because these results belong to the integrated framework rather than the isolated LSTC module, they should be interpreted as system-level outcomes rather than direct measurements of LSTC alone.

The paper also summarizes LSTC’s empirical effect in qualitative terms: it preserves context, better retrieves information from distant frames, boosts accuracy particularly in fine-grained, long-lasting actions where short-term cues are ambiguous, and adds only a very modest parameter count and computation due to the sparse kernel [2509.14619]. This suggests that the module is intended as a low-overhead temporal substitute for standard downsampling rather than as a heavy long-range reasoning block.

## 7. Scope, related uses of the acronym, and interpretive considerations

Within the present topic, LSTC specifically denotes Long-Short Term Temporal Convolution as defined in the skeleton-based action recognition framework LSTC-MDA [2509.14619]. The acronym is not unique in the literature. It has also been used for “Long and Short-Term Constraints” in safe reinforcement learning [2403.18209], “Long-Short Temporal Co-teaching” in weakly supervised video anomaly detection [2303.18044], and “Long-Short Term Context” in atomic action detection [2110.09819]. These are distinct formulations that share the long/short temporal motif but not the same mechanism.

This multiplicity of usage can create terminological confusion. In the temporal-convolution sense, LSTC refers to a module with parallel short-term and long-term convolutional branches, sparse large-kernel design, linear alignment, cosine-similarity weighting, and adaptive fusion during temporal downsampling [2509.14619]. It should therefore not be conflated with methods that use separate temporal networks, co-teaching schemes, constraint formulations, or long/short context decomposition under different architectures.

A common misconception would be to treat LSTC as simply a larger temporal kernel. The formulation shows otherwise: the long-term branch is sparse rather than dense, the short-term branch remains explicitly local, and the final fusion depends on learned similarity rather than unconditional aggregation [2509.14619]. Another possible misconception would be to interpret the module as a standalone recognition architecture. In the paper, it is a module inserted into a larger backbone and evaluated as part of a unified framework.

Taken as a design pattern, LSTC embodies a specific claim about temporal downsampling in skeleton-based action recognition: downsampling should preserve explicitly modeled long-range cues rather than rely solely on local stride-2 convolution. The reported ablations and benchmark results support that claim within the LSTC-MDA setting [2509.14619]. A plausible implication is that the module’s broader relevance lies in rethinking temporal reduction as a selective fusion problem rather than as a purely local filtering step.

Source: https://www.emergentmind.com/topics/long-short-term-temporal-convolution-lstc