---
title: JRDB-Social Dataset
url: https://www.emergentmind.com/topics/jrdb-social
type: topic
---

# JRDB-Social Dataset

JRDB-Social is an extension and specialized annotation track within the JRDB family of datasets, designed to enable analysis and benchmarking of human social grouping, intra-group interactions, group-level context, and group-aware multi-agent prediction in crowded, robotic–egocentric video settings. Its layered annotation schema and scenario-driven context support research in social perception for embodied agents, human–robot interaction, and socially aware navigation.

## 1. Definition, Scope, and Annotation Protocol

JRDB-Social formalizes "social group" detection as the identification of sets of people in a scene sharing a common activity or interaction. A group is defined as any subset of two or more people in a frame engaged in a common real-world interaction (e.g. talking, walking together); singleton groups (size=1) are also annotated to identify isolated individuals [2106.08827]. Annotations are provided per frame, with group assignment determined by event boundaries—annotators track group continuity as members join or leave. Confidence ("difficulty level") is attached to every group label: “Easy” (direct visual evidence), “Moderate” (partial/temporal cues), “Difficult” (occlusion or distance), and “Impossible” (rarely).

JRDB-Social is built upon multimodal data: 360° cylindrical RGB video, synchronized dual Velodyne-16 LiDAR, stereo depth, audio, GPS, and IMU, acquired from a mobile social robot traversing both indoor and outdoor university campus environments [2404.04458]. The core JRDB video spans 54 sequences of approximately 64 minutes, etc.

The annotation schema comprises:
- **Individual attributes**: gender (male/female), age bracket (5 bins), race (Caucasian, Black, Asian), atomic action (20 classes, e.g. walking, waving), and body-pose keypoints [2404.04458].
- **Intra-group interactions**: 20 dyadic labels (e.g. “Walking Together,” “Conversation,” “Shaking Hand,” “Holding Something Together”) exhaustively annotated per detected pair-per-frame with confidence [2404.04458].
- **Group-level context**: structured captions decomposed into body-position connection (e.g. “chair,” “floor”), salient scene content (“pillar,” “table”), venue, and inferred group purpose (“studying,” “ordering food,” etc.).

All individual and group annotations undergo two-stage verification and spot-checking [2404.04458].

## 2. Dataset Statistics and Social Group Distribution

JRDB-Social contains approximately 1.8 million person-frames annotated with demographic and activity attributes, and ≈200k pair-frames with dyadic interaction tags [2404.04458]. For social grouping specifically (as formalized in JRDB-Act—a core foundation of JRDB-Social), the dataset includes approximately 27,000 frames (7 fps, ~64 min), averaging 30 people per frame, totaling ≈810,000 person group-labels [2106.08827].

Empirical group-size distribution (all frames) is highly skewed:

| Group Size | Proportion (%) | Notes         |
|------------|---------------|--------------|
| 1          | 75.5          | singleton    |
| 2          | 16.6          |              |
| 3          | 5.0           |              |
| 4          | 1.2           |              |
| 5+         | 1.0 (max=29)  | rare, large  |

The modal group size by group-level context annotation is typically 2–3 persons, with maximum up to 8 persons in context-aware labels [2404.04458].

Temporal grouping is densely annotated at 7 fps; however, evaluation (for benchmarking comparability) proceeds on 1 Hz "key frames," with group continuity naturally spanning multiple seconds [2106.08827].

In intra-group interactions, the most common labels are "Walking Together," "Standing Together," "Sitting Together," and "Conversation," each representing 15–25% of frames (on a log-scale), while gestures such as "Hugging," "Shaking Hand" occur in <1% of frames [2404.04458].

## 3. Social Group Taxonomy and Interaction Labels

JRDB-Social employs a dual taxonomy:
- **Human–Human interactions** (3 classes): Talking to someone, Listening to someone, Greeting gestures.
- **Human–Object interactions** (12 classes, examples): Holding something, Eating something, Reading, Typing, Pushing an object, Interacting with doors, Pointing at something, etc. See Figure 2 in [2106.08827] for a complete listing.

All social group assignments are informed by these interaction-based cues and annotators are instructed to synchronize group boundaries with changes in detected shared activities.

At the full-group context level, free-form textual descriptions are parsed into four fields (body-position connection, salient scene content, venue, purpose), instantiating socially relevant scene semantics for downstream analysis [2404.04458].

## 4. Evaluation Metrics and Benchmarks

Social group detection in JRDB-Social is evaluated via detection and group assignment. Let $D$ be detected boxes with predicted group IDs, $G$ the ground-truth boxes and group IDs. Assignment proceeds by IoU ≥ 0.5 matching.

- **True Positives (TP)**: Correct box-group assignments under optimal bipartite mapping.
- **False Positives (FP)**: Matched boxes with incorrect group ID, or unmatched detections.
- **False Negatives (FN)**: Ground-truth boxes missed or with incorrect group ID.

Metrics:

\[
\mathrm{Precision} = \frac{|\mathrm{TP}|}{|\mathrm{TP}| + |\mathrm{FP}|}
\quad
\mathrm{Recall} = \frac{|\mathrm{TP}|}{|\mathrm{TP}| + |\mathrm{FN}|}
\]
\[
F_1 = 2 \frac{\mathrm{Precision}\;\mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}
\]

- **AP per group-size bin** ($G_k$ AP): Compute AP restricted to ground-truth groups of size $k$.
- **Mean group mAP**: Averaged over $k=1,2,3,4,5+$:
\[
\mathrm{mAP}_{\mathrm{groups}} = \frac{1}{5}\sum_{k=1}^{5+} \mathrm{AP}(G_k)
\]

Detection and tracking evaluation also utilize CLEAR MOT metrics (MOTA, MOTP, IDF1, and ID switches) extended to 3D IoU for tracking-by-detection settings [2002.08397].

## 5. End-to-End Group Detection Models and Key Results

Recent methods for social group detection on JRDB-Social are predominantly trajectory-based, often formulated as a graph neural network clustering problem. A leading approach encodes each track's 15-timestep trajectory via a single-layer LSTM (32 hidden units), aggregates tracks as nodes, and constructs a fully connected graph where edge features capture time-averaged pairwise distance (GIoU-based) [2304.05678]. Five stacked graph Transformer layers with 8 heads each operate on this graph. Loss is supervised via BCE adjacency, an eigenvalue-based clustering term, and an MSE group-count loss.

Results on JRDB-Act validation set (with ground-truth tracks):

| Method                       | G1 AP | G2 AP | G3 AP | G4 AP | G5+ AP | mAP  |
|------------------------------|-------|-------|-------|-------|--------|------|
| Ehsanpour et al. 2022        | 81.4  | 64.8  | 49.1  | 63.2  | 37.2   | 59.2 |
| Ours (traj + Graph-Trf)      | 71.9  | 73.6  | 64.0  | 71.2  | 48.6   | 65.9 |

This model achieves a +6.7 point mAP improvement and 12× faster inference speed (0.7 M params, 29.4 fps) versus prior work [2304.05678]. Test set improvements are also observed (+0.9 mAP).

Key ablations confirm that grouping accuracy is highest for pairs (G2), declines for larger groups, and is highly sensitive to the detection/tracking backbone. Bottlenecks include ambiguity with parallel motion (distinct people on similar trajectories) and occlusion-driven errors in crowded scenes [2304.05678, 2106.08827].

## 6. Downstream Tasks and Application Benchmarks

JRDB-Social supports a spectrum of research tasks:
- **Individual-level recognition**: Gender, age, race, atomic action [2404.04458].
- **Interaction detection**: Multi-label classification per dyadic pair per frame.
- **Group-level scene understanding**: Multilabel extraction of context (body-position connection, salient content, venue, purpose).

Evaluation commonly uses per-label F1, with baseline results reported for multi-modal vision–language models (Video-LLaMA, MiniGPT-4, InstructBLIP). State-of-the-art holistic models reach F1 ≈ 0.30 on attribute classification under guided conditions, with much lower performance (F1 ≈ 0.17–0.20) for open-scene counting and context, indicating substantial headroom [2404.04458].

Detection and group-centric models (e.g., graph Transformer [2304.05678]) currently define the frontier for end-to-end scalable social group detection.

## 7. Relevance, Limitations, and Open Challenges

JRDB-Social establishes the first comprehensive, robot-centric dataset enabling joint study of group membership, intra-group activity, and high-order social context with rigorous benchmarking. It presents densely annotated, multimodal, temporal data capturing real group formation/dissolution under naturalistic conditions.

Salient limitations include:
- Model performance degrades for group context classification (BPC, SSC, venue, purpose), especially for language-based models [2404.04458].
- Robustness to detector/tracker errors and heavy occlusion remains a challenge.
- Counting group cardinality in open scenes is difficult for current models.

A plausible implication is that cross-modal training and graph-based temporal models are critical for advancing robust, real-time social group understanding in robotics and surveillance.

JRDB-Social is freely available (non-commercial, no-derivatives) to the research community, with exhaustive code and preprocessed annotation resources [2106.08827, 2404.04458], and is expected to remain a cornerstone reference for the study of social dynamics in built environments.

Source: https://www.emergentmind.com/topics/jrdb-social