STAR: Semantic Temporal Adaptive Representation

• STAR is a unified framework for few-shot action recognition that integrates per-frame semantic alignment with temporal dynamics to overcome misalignment and smoothing issues.
• It employs Temporal Semantic Attention and Semantic Temporal Prototype Refiner, leveraging LLM-derived cues to enhance both short-term motions and long-range dependencies.
• STAR achieves consistent improvements across benchmarks (e.g., up to 8.1% gain on SSv2-Full) by coupling vision-language cues with Mamba-based state-space modeling.

Semantic Temporal Adaptive Representation Learning (STAR) is a unified framework for few-shot action recognition (FSAR) that addresses two failure modes identified in prior vision-language approaches: semantic-temporal misalignment, in which static textual prompts do not capture decisive visual cues that appear sparsely across sequences, and inadequate modeling of multi-scale temporal dynamics, in which short-term discriminative cues and long-range dependencies are oversmoothed or fragmented. STAR combines a semantic-alignment component with a temporal-aware component, introduces frame-level cross-modal alignment through Temporal Semantic Attention (TSA), incorporates a Semantic Temporal Prototype Refiner (STPR) built on semantic-guided Mamba blocks, and uses temporally dependent class descriptors derived from LLMs to provide long-range semantic guidance. On five FSAR benchmarks, the method reports consistent improvements over state-of-the-art baselines, including gains of up to $8.1\%$ on SSv2-Full, $6.7\%$ on SSv2-Small under 1-shot, and $7.3\%$ on HMDB51 (Liu et al., 13 May 2026).

1. Problem setting and design rationale

FSAR requires models to generalize to novel action categories from only a handful of annotated samples. Within this setting, STAR is motivated by the observation that existing vision-language methods still exhibit semantic-temporal misalignment and weak multi-scale temporal modeling. The framework is therefore organized around two complementary objectives: explicit semantic alignment at the frame level and temporally structured prototype refinement across multiple frequencies (Liu et al., 13 May 2026).

The architecture is described as a unified framework consisting of a semantic-alignment component and a temporal-aware component. The semantic-alignment component is instantiated by TSA, which aligns per-frame video features with class semantics. The temporal-aware component is instantiated by STPR, which integrates semantic-guided Mamba blocks, multi-frequency temporal sampling, and bidirectional state-space refinement. This design is intended to bridge both the semantic gap and the temporal gap while transferring the sequence modeling capability of Mamba into FSAR.

A central claim of the method is that semantic alignment should occur before or during temporal modeling rather than after global aggregation. The paper explicitly states that TSA avoids the over-smoothing of global pooling and ensures that downstream temporal refinements focus on semantically relevant frames. In parallel, STPR is designed to capture short-term motions at multiple frequencies while enforcing long-range coherence through bidirectional state-space modeling. This suggests a deliberate coupling between semantic selectivity and temporal structure rather than a pipeline in which language supervision is used only at the final classification stage.

2. Temporal Semantic Attention

TSA operates on per-frame visual embeddings and temporally dependent class descriptors. The inputs are support-set and query per-frame embeddings $V \in \mathbb{R}^{F \times D}$ from CLIP-Vision and class descriptors $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$ from the frozen CLIP text encoder. The class descriptors are temporally dependent rather than static label tokens, which allows TSA to condition alignment on richer action semantics.

The module begins by layer-normalizing each class token:

$$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$

It then applies multi-head cross-attention with queries $Q=T_{\text{norm}}$ and keys/values given by the frame sequence:

$$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$

For each class $i$, this produces a class-conditioned video response $$V_{\text{enhanced}}^{(i)} \in \mathbb{R}^{F\times D}$$. The operative point is that alignment is performed at frame granularity, not only after temporal pooling.

The semantic supervision term is a contrastive alignment loss. STAR encourages each class-conditioned video response to be close to its own text embedding and far from the others by an InfoNCE objective:

$6.7\%$0

Here $6.7\%$1 is cosine similarity and $6.7\%$2 is a learnable temperature. In the paper’s interpretation, this mechanism enforces fine-grained semantic-temporal consistency and prepares the sequence for downstream temporal refinement (Liu et al., 13 May 2026).

3. Semantic Temporal Prototype Refiner

STPR refines each support video’s frame features into a single prototype through a sequence of state-space and semantic-guided operations. Its foundation is a Temporal State-Space Module (TSSM), described as based on Mamba/S4. The continuous-time state-space model is

$6.7\%$3

and, after Zero-Order Hold discretization with step $6.7\%$4,

$6.7\%$5

$6.7\%$6

Mamba’s SelectiveScan parametrizes $6.7\%$7 and gates inputs for efficiency. This is the mechanism by which STAR imports state-space sequence modeling into FSAR.

Semantic-Guided Focus (SGF) injects class semantics at the frame level. For support video $6.7\%$8 with frame features $6.7\%$9 and class token $7.3\%$0, STPR computes per-frame relevance scores

$7.3\%$1

yielding $7.3\%$2. The support features are then reweighted through a residual addition:

$7.3\%$3

This directly couples semantic relevance and temporal feature refinement.

Action-Specific Dynamic Temporal (ASD) introduces multi-frequency temporal sampling with strides $7.3\%$4. For each stride $7.3\%$5, the sequence is decomposed into $7.3\%$6 interleaved subsequences indexed by offset $7.3\%$7:

$7.3\%$8

with an analogous definition for the query. A unidirectional TSSM is applied independently to each subsequence, producing $7.3\%$9, where $V \in \mathbb{R}^{F \times D}$0. Each subsequence output is then upsampled back to $V \in \mathbb{R}^{F \times D}$1 frames and averaged over offsets:

$V \in \mathbb{R}^{F \times D}$2

$V \in \mathbb{R}^{F \times D}$3

Support features are fused with the semantic-focused frames:

$V \in \mathbb{R}^{F \times D}$4

The stated purpose is to capture short-term motions at multiple frequencies.

Action-Centric Unified Temporal (ACU) adds bidirectional state-space refinement at each scale to capture long-range dependencies. For support features,

$V \in \mathbb{R}^{F \times D}$5

followed by reverse-and-sum fusion:

$V \in \mathbb{R}^{F \times D}$6

The same operation is applied to the query to obtain $V \in \mathbb{R}^{F \times D}$7. Multi-scale fusion then uses uniform weights $V \in \mathbb{R}^{F \times D}$8:

$V \in \mathbb{R}^{F \times D}$9

A final channel-attention step, denoted ECA, uses global pooling and a $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$0D convolution:

$T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$1

with the same construction for the query. The paper characterizes the resulting prototypes as semantically aligned and temporally consistent (Liu et al., 13 May 2026).

4. LLM-derived descriptors and optimization

STAR uses temporally dependent class descriptors derived offline from LLMs. An offline LLM, such as GPT-4o or Gemini 2.5-Pro, is prompted to expand each class label into a short paragraph describing its key temporal phases. These descriptions are collected as $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$2 and encoded by the frozen CLIP text encoder $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$3:

$T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$4

These descriptors serve two roles: they act as queries in TSA and provide semantic modulation in SGF. The paper presents them as a source of long-range semantic guidance (Liu et al., 13 May 2026).

The few-shot classification stage is based on refined query features and class prototypes. If $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$5 denotes the refined query feature and

$T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$6

denotes the class prototype, then classification uses an OTAM-style temporal distance $T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$7:

$T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$8

The few-shot cross-entropy loss is

$T_c=\{c_1,\dots,c_M\}\in\mathbb{R}^{M\times D}$9

Training jointly optimizes semantic alignment and few-shot classification:

$$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$0

where $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$1 is set to $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$2. This makes the optimization explicitly bi-objective: one term enforces cross-modal discriminability, while the other preserves episode-level classification performance.

The paper’s summary of semantic-temporal synergy is precise. TSA aligns each frame to class semantics and avoids the over-smoothing associated with global pooling; SGF injects semantic priors at the frame level; ASD captures short-term motions at multiple frequencies; and ACU’s bidirectional state-space modeling guarantees long-range coherence. The stated novelty is the integration of vision-language alignment with Mamba-style state-space modeling in a principled multi-scale temporal hierarchy.

5. Reported benchmark results and ablations

The reported evaluation uses standard 5-way 1-shot / 5-shot protocols with a ResNet-50 backbone. Across SSv2-Small, SSv2-Full, HMDB51, UCF101, and Kinetics, STAR is reported to outperform the listed baselines consistently (Liu et al., 13 May 2026).

Dataset	STAR (1-shot / 5-shot)	Comparator
SSv2-Small	55.6% / 59.2%	TSAM: 53.1% / 58.0%
SSv2-Full	63.5% / 67.1%	TSAM: 60.2% / 64.3%
HMDB51	74.9% / 85.0%	CLIP-FSAR: 69.4% / 80.7%
UCF101	94.1% / 98.2%	CLIP-FSAR: 92.4% / 97.0%
Kinetics	94.0% / 95.6%	CLIP-FSAR: 90.1% / 92.0%

The reported deltas are $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$3 on SSv2-Small, $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$4 on SSv2-Full, $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$5 on HMDB51, $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$6 on UCF101, and $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$7 on Kinetics. The paper also states that STAR achieves “up to $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$8 and $$T_{\text{norm}} = \operatorname{LayerNorm}(T_c)\in\mathbb{R}^{M\times D}.$$9 gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting” when compared to older vision-only baselines, and $Q=T_{\text{norm}}$0 on HMDB51.

Ablations on SSv2-Full clarify the relative contributions of the semantic and temporal components. Under the 1-shot / 5-shot setting, the additions labeled $Q=T_{\text{norm}}$1TCR only yield $Q=T_{\text{norm}}$2, $Q=T_{\text{norm}}$3TSA only yields $Q=T_{\text{norm}}$4, and $Q=T_{\text{norm}}$5STPR only yields $Q=T_{\text{norm}}$6. Combined variants produce $Q=T_{\text{norm}}$7 for TCR+TSA, $Q=T_{\text{norm}}$8 for TSA+STPR, $Q=T_{\text{norm}}$9 for TCR+STPR, and $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$0 when all three are used. Within STPR itself, the 1-shot SSv2-Full ablation reports $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$1 for ACU only, $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$2 for ASD only, $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$3 for ASD+ACU $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$4, and $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$5 for the addition of SGF $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$6. For multi-frequency sampling, the best setting is $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$7, which yields $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$8, outperforming any single-stride or dual-stride alternative.

These ablations support the paper’s interpretation that STPR is the dominant source of improvement, while the semantic modules still contribute complementary gains. A plausible implication is that the strongest performance arises not from semantic alignment or temporal modeling in isolation, but from their explicit coupling.

6. Scope, limitations, and subsequent directions

The paper identifies several limitations. STAR relies on offline LLM-generated prompts, which adds a preprocessing step and may carry LLM biases. The text encoder remains frozen, and the authors note that end-to-end fine-tuning of text and vision modalities may yield further gains. The strides $$\operatorname{CrossAttention}(Q,K,V) = \operatorname{Softmax}\!\left(\frac{QK^\top}{\sqrt{D}}\right)V.$$9 and fusion weights $i$0 are fixed rather than learned dynamically. Although the state-space backbone scales linearly, extending the method to very long sequences, specified as $i$1 frames, may require additional memory and time optimizations (Liu et al., 13 May 2026).

The stated future directions follow directly from these constraints: joint prompt adaptation, dynamic stride selection, and real-time deployment on edge devices. These directions are consistent with the method’s two organizing principles. Joint prompt adaptation would address the fixed nature of the current descriptors; dynamic stride selection would generalize the hand-specified multi-frequency hierarchy; and edge deployment would test whether the linear-scaling SSM formulation can be translated into operational efficiency under stricter latency and memory constraints.

A recurrent misconception in reading the method would be to treat STAR as a purely temporal refinement model or, conversely, as a prompt-engineering variant of CLIP-based FSAR. The paper’s formulation does not support either reduction. STAR is defined by the interaction of frame-wise vision-language alignment, semantic-guided multi-scale temporal refinement, and LLM-derived temporal cues. The code is available at https://github.com/HongliLiu1/STAR-main, which positions the work for direct comparison and reproduction within the FSAR literature.