NTU RGB+D 120 Dataset Overview

• NTU RGB+D 120 dataset is a large-scale multimodal benchmark offering synchronized RGB, depth, 3D skeleton, and IR data for detailed human activity analysis.
• It comprises 114,480 video samples across 120 action classes from diverse performers and camera setups, ensuring rich variability and practical evaluation protocols.
• The dataset has spurred methodological innovations in graph convolution, attention mechanisms, and vision-language integration for enhanced single- and multi-modal action recognition.

The NTU RGB+D 120 dataset is a large-scale multimodal benchmark for human activity understanding, widely utilized in research on video-based action recognition. Designed to address the limitations of earlier action recognition datasets—including insufficient scale, class diversity, view and background variability, and a lack of realistic 3D data—NTU RGB+D 120 provides an extensive collection of RGB videos, depth maps, 3D skeletal joint sequences, and infrared (IR) frames. Spanning 120 action classes, 106 distinct performers, and 32 camera setups, it serves as a foundation for the paper and evaluation of both classical and deep learning approaches in single- and multi-modal action recognition (Liu et al., 2019).

1. Composition and Data Modalities

NTU RGB+D 120 comprises 114,480 video samples encompassing approximately 8 million frames. The actions span 82 daily activities (e.g., eating, writing, sitting), 26 mutual interactions (e.g., handshaking, hugging), and 12 health-related actions (e.g., falling, staggering). Data were acquired using Microsoft Kinect v2 sensors at a native resolution of 1920×1080 for RGB and 512×424 for depth and IR modalities. For each sample, four data streams are provided:

• RGB Video: Color recordings at 1920×1080 resolution.
• Depth Maps: High-precision, losslessly-compressed maps matched per-frame with RGB.
• 3D Skeletons: 25-joint coordinates per frame, including corresponding pixel indices in the RGB and depth planes.
• Infrared (IR) Frames: 512×424 resolution, enabling analysis under challenging illumination.

Data collection involved 106 subjects (ages 10–57, representing 15 nationalities) across up to 96 distinct environmental backgrounds and substantial illumination variations. Each subject performed every action at least twice for each configuration.

2. Experimental Protocols and Statistical Properties

The dataset is structured for multiple evaluation scenarios, with three standard protocols:

• Cross-Subject (X-Sub): Splits samples by subject IDs, training on half of the subjects (IDs specified) and testing on the remainder. This accentuates generalization to unseen subjects.
• Cross-Setup (X-Set): Samples are partitioned by camera setup ID (train on even, test on odd). This measures robustness to viewpoint, distance, and background variation across 32 configurations (camera heights 0.5–2.7 m, distances 2.0–4.5 m, 155 total viewpoints).
• One-Shot Recognition: Classes are separated into auxiliary and novel subsets; only one exemplar per novel class is given at test time—the rest must be labeled by nearest-neighbor assignment after training a generator on auxiliary labels.

Each action class contains roughly 900 video samples, with sequences averaging 4–5 seconds in duration. The dataset’s folder structure encodes subject, setup, action class, and camera ID, and each frame contains synchronized modalities and comprehensive metadata.

3. Baseline Evaluations and Comparative Results

NTU RGB+D 120 has established itself as the reference dataset for both skeleton-based and multimodal action recognition. Baseline performances using classical methods (e.g., Part-Aware LSTM, Soft RNN, ST-LSTM) and recent deep architectures (GCA-LSTM, FSNet, ST-GCN, STC-Net) are reported predominantly on skeleton data but also on RGB, Depth, and modality fusion:

Method	X-Sub (%)	X-Set (%)
Part-Aware LSTM	25.5	26.3
GCA-LSTM	58.3	59.2
FSNet (skeleton)	59.9	62.4
Body Pose Evolution Map	64.6	66.9
RGB only (best baseline by fusion)	58.5	54.8
RGB+Depth+Skeleton	64.0	66.1

Recent advances—such as LVLM-VAR, which leverages a Video-to-Semantic-Tokens module and LoRA-fine-tuned Vision-Language Large Models (LVLMs)—achieve 86.5% (X-Sub) and 90.0% (X-Set) on RGB-only benchmarks, showing notable improvement over prior state-of-the-art models including PoseC3D and STC-Net (Peng et al., 6 Sep 2025).

4. Methodological Advances and Benchmarks

The dataset has catalyzed the development of specialized architectures and protocols for robust action recognition. Three prominent directions are:

• Graph Convolutional Models: Extensively benchmarked (e.g., ST-GCN, InfoGCN), these exploit 3D skeleton connectivity.
• Multimodal Fusion and Attention Mechanisms: Fusing RGB, depth, and skeleton streams, and employing attention to focus on discriminative joints, parts, or frames.
• Vision-Language Integration: LVLM-VAR introduces semantic tokenization of video via a VST module and action reasoning using LVLMs such as LLaVA-13B. Discrete semantic tokens ($K=512$) compress spatio-temporal features, and LoRA fine-tuning adapts the LVLM for classification and explanation generation. This framework systematically outperforms previous methods on both X-Sub and X-Set, as shown in the following comparative table:

Method	X-Sub (%)	X-Set (%)
ST-GCN	82.1	84.5
InfoGCN	85.1	86.3
PoseC3D	85.9	89.7
STC-Net	86.2	88.0
LVLM-VAR	86.5	90.0

5. One-Shot and Low-Resource Evaluation

One-shot action recognition is directly addressed in NTU RGB+D 120, partitioning 100 classes for auxiliary feature training and 20 novel classes for evaluation. The Action-Part Semantic Relevance-aware (APSR) framework achieves 45.3% accuracy on novel classes using weighted part pooling based on semantic similarity between class descriptions and body parts. One-shot accuracy increases with auxiliary set size, ranging from 29.1% (20 classes) up to 45.3% (100 classes) (Liu et al., 2019).

6. Dataset-Specific Challenges and Mitigation Strategies

NTU RGB+D 120 exposes significant challenges for algorithmic robustness and generalization:

• Fine-Grained Class Ambiguity: Subtle object and pose distinctions (e.g., “drinking water” vs. “drinking tea”) necessitate models with semantic discrimination. VST tokens in LVLM-VAR encode object and relation states, directly addressing this.
• Cross-Setup and View Variation: The 32 camera setups introduce nontrivial viewpoint shifts and background diversity. Temporal self-attention is employed to extract invariant action cues, leveraging world knowledge from LVLMs for context reasoning.
• Long-Tailed Distribution: With 120 classes and sample imbalance, models are prone to overfitting common actions. Quantization and adaptive fine-tuning (e.g., LoRA) aid in regularizing representations and preventing catastrophic forgetting.
• Interpretability: Traditional approaches yield only class scores. LVLM-based approaches generate human-interpretable natural language rationales for predictions, facilitating diagnosis and model trust (Peng et al., 6 Sep 2025).

7. Impact and Ongoing Research Directions

NTU RGB+D 120 has become the de facto benchmark for evaluating scalable, generalizable, and interpretable action recognition models. It has stimulated advances in graph-based skeleton modeling, attention mechanisms, multimodal learning, and—more recently—the integration of large vision-LLMs for both accuracy and interpretability. The dataset’s standardized protocols, extensive evaluation splits, and modality richness continue to drive innovations in fine-grained behavior analysis, robust recognition under view and context change, and few-shot learning for novel action discovery (Liu et al., 2019, Peng et al., 6 Sep 2025).