Few-Shot Action Recognition Benchmarks

• Few-shot action recognition benchmarks are evaluation protocols and datasets that test algorithms' ability to classify novel actions from minimal examples.
• They incorporate techniques such as prototype-based matching, generative augmentation, and contrastive learning to enhance representation and temporal modeling.
• These benchmarks standardize evaluation with well-defined train/test splits, rigorous statistical sampling, and comprehensive metrics for seen and unseen classes.

Few-shot action recognition benchmarks are established evaluation protocols and datasets designed to assess algorithms that must recognize novel action categories with only a handful of labeled examples per class. These benchmarks enable rigorous comparison of models in the challenging regime where annotated video data is scarce and inductive generalization from limited supervision is required. The benchmarks have evolved to reflect advances in representation learning, sequence modeling, multi-modal fusion, and generative augmentation, and they increasingly incorporate fine-grained action distinctions and generalization to both seen and unseen classes.

1. Foundational Problem Setting

Few-shot action recognition (FSAR) aims to learn to classify video actions from sparse annotated examples, often specified as $N$-way $K$-shot episodic tasks. Each benchmark episode samples $N$ novel classes, each with $K$ support videos, and typically measures accuracy on a query set disjoint from the support. This problem is further extended in the Generalized Few-Shot Learning (G-FSL) setting, in which both previously seen (base) and newly encountered (novel) classes are present at test time, posing additional challenges such as overcoming classifier bias toward base categories (Dwivedi et al., 2019).

The need for such benchmarks arises due to the prohibitive cost of large-scale video annotation and the practical importance of rapid adaptation in domains such as surveillance or robotics. Standard evaluation protocols require careful construction of class disjoint splits, with random task instantiation to measure statistical significance across multiple episodes (Dwivedi et al., 2019, Ben-Ari et al., 2020).

2. Benchmarks and Datasets

State-of-the-art FSAR methods are evaluated on several standardized datasets:

Dataset	Nature	Distinctions
UCF101	Scene-centric	Appearance-focused
HMDB51	Scene-centric	Varied views
Kinetics / Kinetics-400	Large, diverse	Significant variation
Something-Something V2 (SSv2)	Motion-centric	Fine action granularity
Olympic-Sports	Sports domain	Complex motion
FineGym, HAA500	Fine-grained	Hierarchical annotation

Recent benchmarks mandate disjoint splits for base/novel class evaluation, careful selection of k-shot instances, and often average over 6,000 or more episodes for statistical robustness (2108.06647). For fine-grained action recognition, split protocols are precisely controlled to disentangle the effects of intra-class and inter-class variance (Tang et al., 2023, 2108.06647).

Advanced protocols also assess temporal detection performance (e.g., few-shot temporal action detection on ActivityNet 1.2 (Ben-Ari et al., 2020)) and untrimmed video sequences.

3. Methodological Advances Reflected in Benchmarks

Benchmarks have driven methodological innovation along several axes:

• Prototype-based Matching and Temporal Modeling: Early approaches rely on episodic metric learning (e.g., ProtoGAN (Dwivedi et al., 2019), TAEN (Ben-Ari et al., 2020)), in which prototypes summarize support instances (using arithmetic mean or learned aggregators), and query videos are classified by nearest neighbor or trajectory distance metrics.
• Generative Augmentation: GAN-based generation of synthetic features for novel classes, conditioned on learned class prototypes, enables strong generalization benchmarks in the G-FSL regime (Dwivedi et al., 2019).
• Attention and Contrastive Learning: Hybrid prototype-centered contrastive loss and attention mechanisms address outlier suppression, class overlapping, and data underutilization (PAL (Zhu et al., 2021)); bidirectional attention and contrastive meta-learning significantly improve fine-grained action discrimination (2108.06647).
• Multi-view and Relational Representation: Multi-view encoding and matching (M³Net (Tang et al., 2023)), hierarchical spatio-temporal enrichment (STRM (Thatipelli et al., 2021)), and relation-guided set matching (HyRSM (Wang et al., 2022)) reflect the progression from isolated frame/clip representations to more holistic, task-adaptive embeddings.
• Unsupervised and Multimodal Pretraining: Recent unsupervised approaches (MetaUVFS (Patravali et al., 2021)) leverage over half a million unlabeled videos via contrastive self-supervision, with action-appearance alignment surpassing several supervised baselines.

4. Evaluation Criteria and Metrics

The primary evaluation metric is average classification accuracy across episodes, with secondary metrics as appropriate to the task (e.g., mean average precision for detection). In the G-FSL setting, explicit reporting of accuracy on both seen (base) and novel classes, and their harmonic mean

$$\text{Acc}_H = \frac{2 \cdot \text{Acc}_S \cdot \text{Acc}_N}{\text{Acc}_S + \text{Acc}_N}$$

is mandated to quantify models’ ability to generalize without strong bias (Dwivedi et al., 2019).

Ablation studies on the effect of prototype construction, feature pruning, temporal tuple cardinality, and auxiliary losses are standard, with performance improvements of 3–10 percentage points being typical for strong innovations (Thatipelli et al., 2021, Huang et al., 2022, Zhu et al., 2021).

5. Representative Numerical Results and Advances

Benchmarks consistently report improvements in performance as methods evolve. Examples include:

• ProtoGAN achieves absolute harmonic mean improvements of up to 5.7% over baselines on HMDB51 (G-FSL setting) (Dwivedi et al., 2019).
• TAEN matches or outperforms prior alignment and metric-learning based models on both Kinetics (video classification, e.g., 67.27% in 1-shot, 83.12% in 5-shot) and ActivityNet (temporal detection, e.g., mAP 33.64% at 0.5 tIoU in 5-shot) (Ben-Ari et al., 2020).
• Compound Prototype Matching outperforms prior methods by 7–10 percentage points on challenging datasets (SSv2, Kinetics), particularly where object interactions or temporal variations dominate (Huang et al., 2022).
• PAL demonstrates >10% absolute gain in fine-grained action recognition scenarios such as Sth-Sth-100 (Zhu et al., 2021), and multi-view methods such as M³Net and bidirectional attention further push the boundary for fine-grained datasets such as Gym99 and Gym288 (Tang et al., 2023, 2108.06647).
• Unsupervised methods (MetaUVFS) outperform supervised baselines on standard datasets without reliance on labeled base classes (Patravali et al., 2021).

6. Benchmark Contributions and Implications

The development of FSAR benchmarks has catalyzed several significant trends:

• Strong Protocols for Generalization: Introduction of G-FSL as a benchmark protocol rigorously quantifies the ability to distinguish between seen and novel actions with few examples. Subsequent datasets and papers require explicit reporting of base/novel class performance (Dwivedi et al., 2019).
• Diversity of Evaluation Tasks: FSAR benchmarks now encompass not only trimmed video classification but also temporal detection, fine-grained categorization, and temporal localization—necessitating richer representation and generalization strategies (Ben-Ari et al., 2020, 2108.06647).
• Standardization and Reproducibility: Careful definition of train/test splits, repeated random episode sampling, and aggregation of results with confidence intervals are now standard practice (2108.06647, Tang et al., 2023).
• Future Directions: Benchmarks increasingly encourage multimodal fusion, generative synthesis, unsupervised/self-supervised learning, and the integration of textual/semantic guidance. They also highlight the importance of robust handling of outliers, overlap, and motion dynamics under extreme data scarcity.

7. Challenges and Open Problems

Despite continuous progress, several benchmarking challenges persist:

• Fine-grained action distinction remains difficult, especially for actions with subtle intra-class variations or overlapping attribute distributions (2108.06647).
• Temporal misalignment and sparse annotation complicate the adaptation of alignment-based methods to real-world video, particularly for datasets with significant temporal clutter (SSv2, ActivityNet).
• The transferability of methods between appearance-centric and motion-centric benchmarks is not uniform—models excelling on UCF101 or Kinetics may underperform on SSv2 or fine-grained splits (Huang et al., 2022, 2108.06647).
• Standardization of evaluation protocols across diverse datasets and task variants continues to be refined, with new protocols emphasizing more comprehensive reporting and cross-dataset evaluation.

---

Few-shot action recognition benchmarks have served as a driving force for conceptual and methodological advances in the field. Through explicit structuring of evaluation scenarios and datasets, these benchmarks have provided an objective basis for the comparison and development of architectures that effectively generalize in the low-shot regime across spatial, temporal, and semantic dimensions.