STAR: Semantic-Temporal Adaptive Representation Learning for Few-Shot Action Recognition

Abstract: Few-shot action recognition (FSAR) requires models to generalize to novel action categories from only a handful of annotated samples. Despite progress with vision-LLMs, existing approaches still suffer from semantic-temporal misalignment, where static textual prompts fail to capture decisive visual cues that appear sparsely across sequences, and from inadequate modeling of multi-scale temporal dynamics, as short-term discriminative cues and long-range dependencies are often either oversmoothed or fragmented. To address these challenges, we propose Semantic Temporal Adaptive Representation Learning (STAR), a unified framework, consisting of a semantic-alignment component and a temporal-aware component, effectively bridging the semantic and temporal gaps and transferring the sequence modeling capability of Mamba into the FSAR. The semantic alignment module introduces a Temporal Semantic Attention (TSA) mechanism, which performs frame-level cross-modal alignment with textual cues, ensuring fine-grained semantic-temporal consistency. The temporal-aware module incorporates a Semantic Temporal Prototype Refiner (STPR) that integrates semantic-guided Mamba blocks with multi-frequency temporal sampling and bidirectional state-space refinement, yielding semantically aligned prototypes with enhanced discriminative fidelity and temporal consistency. Furthermore, temporally dependent class descriptors derived from LLMs provide long-range semantic guidance. Extensive experiments on five FSAR benchmarks demonstrate the consistent superiority of STAR over state-of-the-art methods. For instance, STAR achieves up to 8.1% and 6.7% gains on the SSv2-Full and SSv2-Small datasets under the 1-shot setting, and 7.3% on HMDB51, validating its effectiveness under limited supervision. The code is available at https://github.com/HongliLiu1/STAR-main.

• The paper introduces STAR, a framework that integrates large-scale vision–language pretraining with temporal optimization to address semantic–temporal misalignment in few-shot action recognition.
• It employs modules such as Temporal Semantic Attention and Semantic Temporal Prototype Refiner to refine frame-level features and capture multi-scale dynamics, resulting in significant accuracy improvements.
• Empirical evaluations demonstrate that STAR outperforms prior techniques with enhanced efficiency, making it highly promising for low-resource and rare event detection scenarios.

Semantic–Temporal Adaptive Representation Learning for Few-Shot Action Recognition: An In-Depth Analysis

Introduction

The STAR framework addresses persistent challenges in Few-Shot Action Recognition (FSAR), namely semantic–temporal misalignment and the lack of precise modeling for multi-scale temporal dynamics. The model integrates large-scale vision–language pretraining with temporal optimization, exploiting Mamba-based State Space Models (SSMs) for efficient long-range sequence modeling. Its innovation lies in explicitly coupling semantic alignment with hierarchical temporal refinement, using LLMs to provide class-conditioned guidance and formulating modules that ensure fine-grained, frame-level semantic–temporal correspondence. The empirical evaluations on standard FSAR benchmarks demonstrate significant accuracy improvements over recent state-of-the-art approaches, supporting the efficacy and scalability of this design.

Figure 1: The motivation demonstrates misalignment in prior work and the benefit of explicit frame-level semantic–temporal modeling by STAR.

STAR Architecture and Core Modules

STAR constructs video representations by integrating frame-level visual features and temporally dependent semantic descriptors. The architecture consists of two primary modules: Temporal Semantic Attention (TSA) and the Semantic Temporal Prototype Refiner (STPR).

The pipeline begins with encoding support and query videos into per-frame embeddings via a CLIP-based backbone. Class semantics are augmented by LLM-generated, temporally aware descriptions, which are then embedded using a frozen CLIP text encoder. The TSA module performs cross-modal, frame-level alignment using multi-head attention, ensuring that decisive action frames in the video align with relevant temporal phases in the class semantics.

STPR follows, refining these features through three submodules:

• Semantic-Guided Focus (SGF): Assigns high weights to frames exhibiting strong semantic relevance, down-weighting background or contextually weak segments.
• Action-Specific Dynamic Temporal (ASD): Captures short-term, high-frequency motion patterns using multi-stride temporal decomposition and causal SSMs.
• Action-Centric Unified Temporal (ACU): Fuses bidirectional SSM features to model global, long-range dependencies, merging across multiple temporal resolutions.

Through this pipeline, prototypes are not only temporally robust but also semantically discriminative, inherently aligning dynamic action cues with their textual counterparts.

Figure 2: The STAR framework overview, revealing modular design with explicit semantico-temporal pathways.

Figure 3: Temporal State Space Module (TSSM) structure, leveraging the Mamba framework for sequence modeling.

Figure 4: STPR module architecture, decomposing refinement into SGF, ASD, and ACU for hierarchical and semantic-temporal modeling.

Temporal–Semantic Alignment and Prototype Matching

The TSA module is central to STAR’s success. By implementing cross-attention between temporally expanded class embeddings and video frame features, it facilitates the anchoring of framewise dynamics in semantic space. An InfoNCE-style contrastive loss encourages feature correspondence, strengthening fine-grained semantic–temporal alignment.

STPR’s multi-frequency design averts the over-smoothing observed in basic SSM models. ASD isolates phase-specific, local dynamics by varying the temporal stride, and ACU’s bidirectional passes guarantee context completeness. Importantly, semantic guidance is maintained throughout, with the SGF mechanism modulating each processing stage. Metric-based matching using refined prototypes enables discrimination of hard-negative actions, especially where only short or ambiguous motion cues exist.

Empirical Results and Ablation Analysis

STAR consistently outperforms recent CLIP-based and temporal alignment methods under 1-shot and 5-shot protocols.

• On SSv2-Full, accuracy reaches 66.1–75.8% (ViT-B backbone), translating to absolute gains of 5–8% compared to CLIP-FSAR and TSAM baselines.
• Notable improvements on HMDB51 (+7.1% over previous SOTA in 1-shot), and competitive results on Kinetics and UCF101.

  Figure 5: STAR’s top-1 accuracy is robust across various frame lengths, outperforming baselines as the sequence increases.

Ablation studies reveal:

• All STAR components (LLM-based Temporal Class Representations, TSA, and STPR) contribute additively; their removal leads to measurable performance drops.
• Multi-frequency temporal sampling ($W = \{1,2,4\}$) is critical; restricting to a single scale reduces discriminative capacity.
• STAR’s effectiveness is orthogonal to the underlying frame alignment mechanism, enhancing both OTAM and Bi-MHM pipelines.
• LLM-generated prompts outperform manual templates, showing the benefit of detailed, temporally contextual action semantics.

  Figure 6: STAR’s N-way 1-shot accuracy exhibits superior scaling on SSv2-Small and Kinetics, even as class count increases.

Qualitative Visualization and Model Analysis

Frame–class attention maps generated by TSA highlight the model’s ability to localize decisive action events, distinguishing between transient and background frames. STAR’s frame importance analysis on HMDB51 demonstrates focusing on critical motion phases, mitigating prototype pollution from context/apparatus cues.

Figure 7: TSA module frame-to-class attention maps show focused semantic alignment, with key action frames strongly activated.

Figure 8: LLM-driven generation of temporal class representations, translating brief labels into sequence-aware descriptions.

Figure 9: Frame-level importance score distribution, with highlighted peaks denoting accurate localization of salient dynamics.

T-SNE analysis visualizes the compactness and separability of prototypes in embedding space, with STAR yielding dense intra-class manifolds and minimal cross-class intrusion, unlike attention-based baselines.

Figure 10: T-SNE visualization reveals denser clustering and improved inter-class boundaries with STAR prototypes.

Computational analysis indicates STAR is more efficient than attention-based competitors, reducing both parameter count and FLOPs, and sustaining higher inference throughput while maintaining accuracy superiority.

Implications, Limitations, and Future Directions

The STAR framework demonstrates the importance of explicit semantic–temporal synergy for generalization in few-shot video understanding. By bridging the representational gap using sophisticated LLM-guided semantics and scalable SSM-based temporal modeling, the approach paves the way for robust recognition in highly data-scarce, context-ambiguous scenarios.

Practically, STAR is relevant for low-resource deployment, task transfer, and rare event detection, with favorable computational scaling and extensibility to other modalities (e.g., skeleton, audio).

Theoretically, the findings suggest vision–LLMs require not only better prompts but also temporally distributed, contextual representations to match human-level discriminability in video tasks. The STPR design may serve as a blueprint for integrating semantic grounding into sequence models beyond classical attention or SSMs.

Potential directions include:

• Online or adaptive LLM-semantic augmentation to further enhance prompt flexibility.
• Investigations extending the semantic–temporal coupling principles to multi-agent, interactive, or compositional scenarios.
• Applications to OOD detection and open-vocabulary action segmentation leveraging STAR’s frame-specific prototypes.

Conclusion

STAR achieves state-of-the-art results in FSAR through a synergy of semantically aware temporal modeling and efficient SSM-based sequence processing. Its ability to localize, align, and refine action prototypes at the frame level underpins its generalization and scalability. The framework substantiates that hierarchical, explicit semantic–temporal integration, grounded in modern vision–language and sequence modeling techniques, is a robust paradigm for low-shot video understanding.