JRDB-Social Dataset

Updated 15 April 2026

JRDB-Social is a comprehensive dataset that formally defines social groups through multimodal annotations and scenario-driven contexts for human–robot and social interaction research.
It provides detailed per-frame annotations of individual attributes, intra-group interactions, and group-level context using synchronized sensors like 360° video, LiDAR, and audio.
Recent graph-based models show significant advances in group detection accuracy and speed, highlighting the dataset's potential to benchmark novel social perception techniques.

JRDB-Social is an extension and specialized annotation track within the JRDB family of datasets, designed to enable analysis and benchmarking of human social grouping, intra-group interactions, group-level context, and group-aware multi-agent prediction in crowded, robotic–egocentric video settings. Its layered annotation schema and scenario-driven context support research in social perception for embodied agents, human–robot interaction, and socially aware navigation.

1. Definition, Scope, and Annotation Protocol

JRDB-Social formalizes "social group" detection as the identification of sets of people in a scene sharing a common activity or interaction. A group is defined as any subset of two or more people in a frame engaged in a common real-world interaction (e.g. talking, walking together); singleton groups (size=1) are also annotated to identify isolated individuals (Ehsanpour et al., 2021). Annotations are provided per frame, with group assignment determined by event boundaries—annotators track group continuity as members join or leave. Confidence ("difficulty level") is attached to every group label: “Easy” (direct visual evidence), “Moderate” (partial/temporal cues), “Difficult” (occlusion or distance), and “Impossible” (rarely).

JRDB-Social is built upon multimodal data: 360° cylindrical RGB video, synchronized dual Velodyne-16 LiDAR, stereo depth, audio, GPS, and IMU, acquired from a mobile social robot traversing both indoor and outdoor university campus environments (Jahangard et al., 2024). The core JRDB video spans 54 sequences of approximately 64 minutes, etc.

The annotation schema comprises:

Individual attributes: gender (male/female), age bracket (5 bins), race (Caucasian, Black, Asian), atomic action (20 classes, e.g. walking, waving), and body-pose keypoints (Jahangard et al., 2024).
Intra-group interactions: 20 dyadic labels (e.g. “Walking Together,” “Conversation,” “Shaking Hand,” “Holding Something Together”) exhaustively annotated per detected pair-per-frame with confidence (Jahangard et al., 2024).
Group-level context: structured captions decomposed into body-position connection (e.g. “chair,” “floor”), salient scene content (“pillar,” “table”), venue, and inferred group purpose (“studying,” “ordering food,” etc.).

All individual and group annotations undergo two-stage verification and spot-checking (Jahangard et al., 2024).

JRDB-Social contains approximately 1.8 million person-frames annotated with demographic and activity attributes, and ≈200k pair-frames with dyadic interaction tags (Jahangard et al., 2024). For social grouping specifically (as formalized in JRDB-Act—a core foundation of JRDB-Social), the dataset includes approximately 27,000 frames (7 fps, ~64 min), averaging 30 people per frame, totaling ≈810,000 person group-labels (Ehsanpour et al., 2021).

Empirical group-size distribution (all frames) is highly skewed:

Group Size	Proportion (%)	Notes
1	75.5	singleton
2	16.6
3	5.0
4	1.2
5+	1.0 (max=29)	rare, large

The modal group size by group-level context annotation is typically 2–3 persons, with maximum up to 8 persons in context-aware labels (Jahangard et al., 2024).

Temporal grouping is densely annotated at 7 fps; however, evaluation (for benchmarking comparability) proceeds on 1 Hz "key frames," with group continuity naturally spanning multiple seconds (Ehsanpour et al., 2021).

In intra-group interactions, the most common labels are "Walking Together," "Standing Together," "Sitting Together," and "Conversation," each representing 15–25% of frames (on a log-scale), while gestures such as "Hugging," "Shaking Hand" occur in <1% of frames (Jahangard et al., 2024).

JRDB-Social employs a dual taxonomy:

Human–Human interactions (3 classes): Talking to someone, Listening to someone, Greeting gestures.
Human–Object interactions (12 classes, examples): Holding something, Eating something, Reading, Typing, Pushing an object, Interacting with doors, Pointing at something, etc. See Figure 1 in (Ehsanpour et al., 2021) for a complete listing.

All social group assignments are informed by these interaction-based cues and annotators are instructed to synchronize group boundaries with changes in detected shared activities.

At the full-group context level, free-form textual descriptions are parsed into four fields (body-position connection, salient scene content, venue, purpose), instantiating socially relevant scene semantics for downstream analysis (Jahangard et al., 2024).

4. Evaluation Metrics and Benchmarks

Social group detection in JRDB-Social is evaluated via detection and group assignment. Let $D$ be detected boxes with predicted group IDs, $G$ the ground-truth boxes and group IDs. Assignment proceeds by IoU ≥ 0.5 matching.

True Positives (TP): Correct box-group assignments under optimal bipartite mapping.
False Positives (FP): Matched boxes with incorrect group ID, or unmatched detections.
False Negatives (FN): Ground-truth boxes missed or with incorrect group ID.

Metrics:

$\mathrm{Precision} = \frac{|\mathrm{TP}|}{|\mathrm{TP}| + |\mathrm{FP}|} \quad \mathrm{Recall} = \frac{|\mathrm{TP}|}{|\mathrm{TP}| + |\mathrm{FN}|}$

$F_1 = 2 \frac{\mathrm{Precision}\;\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$

AP per group-size bin ( $G_k$ AP): Compute AP restricted to ground-truth groups of size $k$ .
Mean group mAP: Averaged over $k=1,2,3,4,5+$ : $\mathrm{mAP}_{\mathrm{groups}} = \frac{1}{5}\sum_{k=1}^{5+} \mathrm{AP}(G_k)$

Detection and tracking evaluation also utilize CLEAR MOT metrics (MOTA, MOTP, IDF1, and ID switches) extended to 3D IoU for tracking-by-detection settings (Shenoi et al., 2020).

5. End-to-End Group Detection Models and Key Results

Recent methods for social group detection on JRDB-Social are predominantly trajectory-based, often formulated as a graph neural network clustering problem. A leading approach encodes each track's 15-timestep trajectory via a single-layer LSTM (32 hidden units), aggregates tracks as nodes, and constructs a fully connected graph where edge features capture time-averaged pairwise distance (GIoU-based) (Jahangard et al., 2023). Five stacked graph Transformer layers with 8 heads each operate on this graph. Loss is supervised via BCE adjacency, an eigenvalue-based clustering term, and an MSE group-count loss.

Results on JRDB-Act validation set (with ground-truth tracks):

Method	G1 AP	G2 AP	G3 AP	G4 AP	G5+ AP	mAP
Ehsanpour et al. 2022	81.4	64.8	49.1	63.2	37.2	59.2
Ours (traj + Graph-Trf)	71.9	73.6	64.0	71.2	48.6	65.9

This model achieves a +6.7 point mAP improvement and 12× faster inference speed (0.7 M params, 29.4 fps) versus prior work (Jahangard et al., 2023). Test set improvements are also observed (+0.9 mAP).

Key ablations confirm that grouping accuracy is highest for pairs (G2), declines for larger groups, and is highly sensitive to the detection/tracking backbone. Bottlenecks include ambiguity with parallel motion (distinct people on similar trajectories) and occlusion-driven errors in crowded scenes (Jahangard et al., 2023, Ehsanpour et al., 2021).

6. Downstream Tasks and Application Benchmarks

JRDB-Social supports a spectrum of research tasks:

Individual-level recognition: Gender, age, race, atomic action (Jahangard et al., 2024).
Interaction detection: Multi-label classification per dyadic pair per frame.
Group-level scene understanding: Multilabel extraction of context (body-position connection, salient content, venue, purpose).

Evaluation commonly uses per-label F1, with baseline results reported for multi-modal vision–LLMs (Video-LLaMA, MiniGPT-4, InstructBLIP). State-of-the-art holistic models reach F1 ≈ 0.30 on attribute classification under guided conditions, with much lower performance (F1 ≈ 0.17–0.20) for open-scene counting and context, indicating substantial headroom (Jahangard et al., 2024).

Detection and group-centric models (e.g., graph Transformer (Jahangard et al., 2023)) currently define the frontier for end-to-end scalable social group detection.

7. Relevance, Limitations, and Open Challenges

JRDB-Social establishes the first comprehensive, robot-centric dataset enabling joint study of group membership, intra-group activity, and high-order social context with rigorous benchmarking. It presents densely annotated, multimodal, temporal data capturing real group formation/dissolution under naturalistic conditions.

Salient limitations include:

Model performance degrades for group context classification (BPC, SSC, venue, purpose), especially for language-based models (Jahangard et al., 2024).
Robustness to detector/tracker errors and heavy occlusion remains a challenge.
Counting group cardinality in open scenes is difficult for current models.

A plausible implication is that cross-modal training and graph-based temporal models are critical for advancing robust, real-time social group understanding in robotics and surveillance.

JRDB-Social is freely available (non-commercial, no-derivatives) to the research community, with exhaustive code and preprocessed annotation resources (Ehsanpour et al., 2021, Jahangard et al., 2024), and is expected to remain a cornerstone reference for the study of social dynamics in built environments.