ETRI-Activity3D Dataset

• ETRI-Activity3D dataset is a realistic, large-scale RGB-D benchmark capturing elderly daily activities from robot viewpoints in apartment-like environments.
• It offers synchronized modalities including high-resolution RGB videos, depth maps, and 3D skeleton sequences to facilitate diverse recognition pipelines.
• Benchmark studies using models like FSA-CNN and ConViViT highlight its challenges, especially in addressing domain gaps between elderly and young subjects.

to=arxiv_search 天天中彩票的 ＿一本道 code: {"query":"ETRI-Activity3D dataset elderly daily activities robot-view arXiv", "max_results": 10} ETRI-Activity3D is a large-scale RGB-D daily activity recognition dataset introduced specifically for robots to recognize elderly people’s activities from a robot-view. It contains 112,620 samples collected from 100 subjects performing 55 activity classes, and provides three synchronized modalities: RGB videos, depth maps, and 3D skeleton sequences. The dataset was collected in an apartment-like environment using Kinect v2 sensors and was designed to reflect the working environment and service situations of human-care robots rather than idealized laboratory capture conditions (Jang et al., 2020). Subsequent work has used it as a benchmark for skeleton-guided gesture recognition (Kim et al., 2021), hybrid CNN-transformer activity recognition from RGB video (Dokkar et al., 2023), and knowledge-augmented vision-language modeling for robotic action recognition (Nguyen et al., 19 Sep 2025).

1. Origin and design rationale

ETRI-Activity3D was created to address three gaps in earlier 3D activity benchmarks: application mismatch, lack of realism, and insufficient scale and diversity. The motivating observation was that existing datasets were often not focused on the daily activities that elderly people actually perform, were not collected from viewpoints that robots would typically have in home-care settings, and were frequently limited by controlled laboratory conditions, fixed cameras, constrained backgrounds, and action categories with weak relevance to elderly daily life (Jang et al., 2020).

The dataset therefore centers on practical action categories selected from close observation of the daily lives of the elderly, realistic data collection reflecting robot working environments and service situations, and a scale intended to overcome limitations of prior 3D activity analysis benchmarks. The underlying premise is that deep learning for action recognition is data-hungry and that elderly users are likely to be primary beneficiaries of human-care robots, so elderly-centered data are not merely a demographic extension but a deployment-relevant requirement (Jang et al., 2020).

A common simplification is to treat ETRI-Activity3D as just another RGB-D action benchmark. The source paper positions it more narrowly and more practically: as a robot-centered, elderly-centered benchmark for daily living activity recognition. Later work further characterizes it as the first large-scale RGB-D dataset of daily activities of elderly people and uses it as a challenging real-world benchmark for elderly daily activity recognition rather than as a generic action-recognition corpus (Dokkar et al., 2023).

2. Capture configuration and the meaning of “robot-view”

The dataset was collected using Kinect v2 sensors in an apartment-like environment intended to resemble elderly living conditions rather than a laboratory. Four capturing platforms were arranged to simultaneously record multiple views. Sensors were placed at 70 cm and 120 cm heights, corresponding to typical human-care robot viewpoints, and the distance from sensor to subject varied from 1.5 m to 3.5 m. The apartment included realistic spaces such as the living room, bedroom, and kitchen (Jang et al., 2020).

Subjects were instructed to perform actions naturally, to ignore the cameras, and to vary posture and hand usage when appropriate. The collection protocol also intentionally included situations in which a robot would likely observe people from behind, especially in small spaces such as kitchens and bathrooms. In later descriptions of the dataset, this same capture regime is summarized as involving multiple viewpoints, heights, and distances, with action instances performed in varying contexts (Nguyen et al., 19 Sep 2025).

Within the dataset literature, “robot-view” does not denote a single canonical camera pose. It denotes the kind of perspective a service robot would plausibly have in a home-care environment: low or mid-height viewpoints, oblique or rear views, distance variation, and imperfect observation conditions. The intended target is realistic robot perception rather than ideal action capture (Jang et al., 2020).

3. Modalities, class structure, and participant composition

ETRI-Activity3D contains 55 classes in total. Of these, 52 classes were derived from observing the daily life of elderly people, and 3 classes were added specifically for human-robot interaction: waving, beckoning, and pointing. The paper also notes five mutual actions, such as handshaking and hugging. This yields a class design that mixes ordinary elderly daily activities, interaction actions relevant to robots, and socially meaningful mutual actions (Jang et al., 2020).

The participant pool consists of 100 subjects with a balanced age composition: 50 elderly subjects and 50 young adults. The elderly group is aged 64–88 years with average age 77, and the younger group is in their 20s with average age 23. The elderly group consisted of 17 men and 33 women, while the younger group had 25 men and 25 women. This age split was included specifically to enable analysis of domain differences between elderly and young adults (Jang et al., 2020).

The three synchronized modalities are RGB video, depth map, and skeleton sequence. RGB streams are captured at $1920 \times 1080$, depth maps at $512 \times 424$, and skeletons provide the 3D coordinates of 25 body joints tracked by Kinect v2. These characteristics make the dataset usable for RGB-only, depth-based, skeleton-based, and multimodal recognition pipelines (Jang et al., 2020).

Attribute	Value
Subjects	100
Activity classes	55
Samples	112,620
Modalities	RGB video, depth map, skeleton sequence
RGB resolution	$1920 \times 1080$
Depth resolution	$512 \times 424$
Skeleton joints	25

The dataset’s scale and composition were presented as comparable in scope to NTU RGB+D 120 while being more realistic for elder-care scenarios. A plausible implication is that its value lies not only in the number of samples, but also in the coupling of scale with a deployment-oriented action taxonomy and capture geometry (Jang et al., 2020).

4. Protocols, variability, and difficulty

The main evaluation protocol is cross-subject: 67 subjects for training and 33 subjects for testing. The test subject IDs are $\{3, 6, 9, 12, \ldots, 99\}$, with the remaining subjects used for training. The original paper also describes a separate cross-age analysis in which the dataset was split into elderly and adult subsets and then further divided into train/test portions to test domain transfer across age groups (Jang et al., 2020).

The original preprocessing and augmentation pipeline includes 3D rotation, body-shape variation, noise injection, and random sampling over varying sequence lengths. For ETRI-Activity3D, input length during training and testing ranged from 32 to 200 frames. The dataset was explicitly shown to exhibit strong temporal variability: a model trained with variable-length inputs remained above 83% accuracy for input lengths from 32 to 96 frames, but accuracy dropped sharply for 16-frame clips, indicating that short clips may not capture enough discriminative motion (Jang et al., 2020).

A central difficulty result concerns the domain difference between elderly and adult subjects. Statistics over normalized skeletons showed that elderly and adult groups differ significantly, with elderly actions having longer average frame length and different motion differential statistics. The cross-domain evaluation makes this concrete:

Train	Elderly test	Adults test
Elderly	87.7	69.0
Adults	74.9	85.0
Mixed	84.8	82.1

These results show that networks trained on elderly data perform much better on elderly test data, and likewise for adults, confirming a real domain gap between age groups. This is one of the dataset’s defining empirical properties: age-related motion differences are not a marginal nuisance variable but a measurable source of distribution shift (Jang et al., 2020).

5. Initial benchmarks and multimodal baselines

The dataset paper introduced Four-Stream Adaptive CNN (FSA-CNN) as a benchmark model rather than merely as an auxiliary baseline. FSA-CNN was designed to cope with spatio-temporal variations, input-specific nonlinear adaptation, and the extension of the conventional two-stream approach. Its four streams are action sequence, short-term temporal differential, long-term temporal differential, and spatial differential. The model uses global max pooling to support variable-length inputs and an activation network to adapt nonlinearly to input-specific variations (Jang et al., 2020).

In the cross-subject setting, the main skeleton-only benchmark results reported on ETRI-Activity3D are: IndRNN 73.9%, Beyond Joint 79.1%, SK-CNN 83.6%, ST-GCN 86.8%, Motif ST-GCN 89.9%, Ensem-NN 83.0%, MANs 82.4%, HCN 88.0%, and FSA-CNN 90.6%. The spread of these results is itself informative: the dataset is sufficiently difficult to separate methods meaningfully rather than collapsing performance into a narrow band (Jang et al., 2020).

The multimodal experiments show that the dataset supports RGB-only, skeleton-only, and fused recognition settings. Reported results include RGB only at 90.1%, skeleton only at 90.6%, and RGB + skeleton at 93.7% for FSA-CNN. Comparative multimodal baselines include Deep Bilinear Learning at 88.4%, Evolution Pose Map at 93.6%, and c-ConvNet at 91.3%. The source paper interprets the improvement from combining 2D skeletons extracted from RGB with 3D skeletons from Kinect as evidence that the modalities are complementary, while also noting that 2D skeletons are attractive for deployment because they do not require depth cameras (Jang et al., 2020).

One misconception that later work helps clarify is that multimodality is always necessary for strong performance on ETRI-Activity3D. The original benchmarks established multimodal complementarity, but later RGB-only methods exceeded earlier multimodal baselines on the same dataset. Empirically, this makes ETRI-Activity3D a benchmark not only for multimodal fusion but also for testing whether stronger temporal or semantic modeling can reduce dependence on additional sensors.

6. Later use in gesture recognition, RGB modeling, and vision-language learning

Subsequent research has used ETRI-Activity3D to probe different aspects of activity understanding beyond the original FSA-CNN setting. In "Gesture Recognition with a Skeleton-Based Keyframe Selection Module" (Kim et al., 2021), the dataset is used in a 14-class configuration for gesture recognition. Each labeled gesture clip is paired with its skeleton sequence, and if a video contains multiple gestures, it is split into multiple clips before training and evaluation. The proposed bidirectional consecutively connected two-pathway network (BCCN) does not treat the skeleton as a direct recognition stream; instead, the skeleton sequence is fed into an LSTM-based keyframe selection module that selects informative RGB keyframes, while a temporal-attention pathway models denser video input. On ETRI-Activity3D, the reported fusion-method results are C2D at 80.338%, I3D at 82.673%, SlowFast at 85.349%, SlowFast + skeleton feature at 85.354%, SlowFast + keyframe selection module at 86.151%, and full BCCN at 87.775%. The associated interpretation in the paper is that, for this dataset, skeletons can be more useful as a temporal selector than as a directly fused modality, particularly when classes differ by subtle motions and execution speed varies across subjects.

In "ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition" (Dokkar et al., 2023), ETRI-Activity3D is used as a challenging real-world benchmark for elderly daily activity recognition, but only in the final comparison against prior work rather than in the ablation study. The model uses RGB only. Its CNN front-end converts raw input from a 3-channel video to a 128-channel video, after which a transformer applies patch embedding with $16 \times 16$ patches and factorized self-attention to learn spatiotemporal dependencies. On ETRI-Activity3D, the reported table gives FSA-CNN at 90.1% for RGB, FSA-CNN at 93.7% for RGB + S, and ConViViT at 95.09% for RGB. This means the RGB-only hybrid CNN-transformer model exceeds both the prior RGB-only and RGB + skeleton baselines reported in that comparison.

In "KRAST: Knowledge-Augmented Robotic Action Recognition with Structured Text for Vision-LLMs" (Nguyen et al., 19 Sep 2025), ETRI-Activity3D becomes the central benchmark for a prompt-learning approach built on frozen CLIP-style text and vision encoders. The paper describes the dataset as a large-scale RGB-D benchmark for video-based action recognition in robot-assisted elderly care and uses the official cross-subject split with 67 subjects for training and 33 for testing. RGB frames are resized from $1920 \times 1080$ to $456 \times 256$; a YOLOv11 detector localizes the main person; a fixed $224 \times 224$ crop centered on the person is taken; and 32 frames per video are selected by uniform frame sampling. Using only RGB video at test time, the best prompt strategy, SegKPT with $S+H+D$, reaches top-1 accuracy 95.22%, F1-score 0.946, and weighted F1-score 0.952. The comparison table in the paper lists Motif ST-GCN at 89.9%, HCN at 88.0%, Deep Bilinear Learning at 88.4%, Evolution Pose Map at 93.6%, c-ConvNet at 91.3%, FSA-CNN at 90.1% for RGB, FSA-CNN at 90.6% for skeleton, FSA-CNN at 93.7% for RGB + skeleton, and KRAST at 95.22% for RGB.

Taken together, these later studies show that ETRI-Activity3D is not tied to a single modeling tradition. It has supported skeleton-only recognition, RGB-only recognition, skeleton-guided keyframe selection, hybrid CNN-transformer architectures, and frozen vision-LLMs with structured prompts. The consistent role of the dataset across these works is to expose precisely the kinds of conditions that matter for robot perception in indoor elder-care scenarios: subtle motion differences, intra-class variation, realistic viewpoint changes, and cross-subject generalization under non-laboratory capture conditions (Kim et al., 2021, Dokkar et al., 2023, Nguyen et al., 19 Sep 2025).