JRDB-Reasoning Benchmark

• JRDB-Reasoning is a structured benchmark that quantifies reasoning complexity in embodied AI via a spatio-temporal graph model.
• It features an adaptive query generation engine that creates customizable visual grounding and VQA queries with annotated multi-step workflows.
• The benchmark extends JRDB with rich human-object interaction and geometric relationship annotations to enhance multi-entity and spatial analysis.

JRDB-Reasoning is a structured benchmark built on the JRDB egocentric perception dataset, designed to rigorously evaluate visual reasoning in human-crowded, robot-inhabited environments. By introducing a formalized notion of reasoning complexity, an adaptive query generation engine, and extended human-object interaction and geometric relationship annotations, JRDB-Reasoning enables dynamic, difficulty-controlled, and workflow-annotated evaluation of visual-LLMs (VLMs) for spatial, attribute, and logical reasoning tasks in embodied AI contexts (Jahangard et al., 14 Aug 2025).

1. Formalization of Reasoning Complexity

JRDB-Reasoning introduces an explicit metric for reasoning complexity based on the dynamic spatio-temporal graph (STG) underlying each scene. Reasoning complexity $D$ is formally defined as:

$D = S + R + T$

where $S$ is the number of entities (nodes), $R$ is the number of spatial relationships (edges), and $T$ is the number of temporal relationships (time-slot edges). This triad quantitatively grades the difficulty of a visual reasoning query.

Difficulty bins are defined as: | Difficulty | $S$ | $R$ | $T$ | Description | |-----------------|-----|----------|-----|-------------------------------------| | $\mathcal{D}_1$ | 1 | 0 | 1 | Single entity, no relation | | $\mathcal{D}_2$ | 2 | $D = S + R + T$0 1 | 1 | Two entities, at least one relation | | $D = S + R + T$1 | 3 | $D = S + R + T$2 2 | 1 | Three entities, at least two edges |

This principled grading distinguishes JRDB-Reasoning from prior benchmarks by supporting fine-grained, progressive evaluation as queries require more compositional and multi-entity logic (Jahangard et al., 14 Aug 2025).

2. Adaptive Query Generation Engine and Workflow Annotation

The core of JRDB-Reasoning is a generative query engine capable of producing customizable queries—Visual Grounding (VG) and Visual Question Answering (VQA)—at specific difficulty levels. This engine operates algorithmically:

• Instantiates an STG from synchronized multi-modal sensor data.
• Enumerates possible $D = S + R + T$3-compliant entity-relation-time combinations.
• Randomly samples a compliant combination.
• Generates the natural-language query, solution, and—uniquely—a full intermediate workflow: every step in the reasoning chain is annotated with the subset of entities and sub-question at that stage.

Pseudocode overview: $R$4

Each question is thus linked to a record of all intermediate sets, filters, and relations used, enabling stepwise assessment of model reasoning depth (Jahangard et al., 14 Aug 2025).

3. Dataset Extensions for Human-Object Interaction and Geometric Reasoning

JRDB-Reasoning augments core JRDB annotations with two types of fine-grained reasoning cues:

A. Human–Object Interaction (HOI) Annotations:

Frame-level, multi-label tags describe actions involving humans and objects in one of several categories: - Pose-based (e.g., walking, standing) - Observational (e.g., looking at robot, reading) - Physical (e.g., carrying, touching) - Manipulative (e.g., operating, using)

B. Geometric Relationship Annotations:

For each entity pair $D = S + R + T$4:

• Angle $D = S + R + T$5 in the horizontal plane, relative to $D = S + R + T$6's forward direction:

$D = S + R + T$7

• Distance $D = S + R + T$8 (meters), quantized into 5 bins.
• Each pair maps into one of 16 direction categories (e.g., front, left, back-right) and 5 distance categories (very close to very far).

These annotations are formatted in standardized JSON for programmatic access by query engines (Jahangard et al., 14 Aug 2025).

4. Customization and Stepwise Workflow Support

JRDB-Reasoning allows users to tune five high-level sliders to generate query-answer pairs:

• Modality: image or video
• Task type: Visual Grounding, VQA-Wh, or VQA-Counting
• Subject type: human, object, or both
• Spatial and temporal scope: single, pair, or clique

The engine instantiates the STG on selected frames, enumerates candidate entity-relation combinations per the target $D = S + R + T$9, samples one, and outputs:

• The final query (Q)
• The answer (A)
• A stepwise workflow $S$0

Example:

Query: Count the number of female persons looking at the robot. Workflow: 1. $S$1: "Identify all humans." $S$2: "There are 7 humans." 2. $S$3: "Filter those by gender = female." $S$4: "3 females remain." 3. $S$5: "Among these, who look at the robot?" $S$6: "2 females." Final answer: 2.

All intermediate steps are preserved for granular benchmarking of compositional reasoning stages (Jahangard et al., 14 Aug 2025).

5. Evaluation Protocols and Baseline Results

JRDB-Reasoning defines two supervised evaluation tasks:

• VG (Visual Grounding): Predict the bounding box of the target; match is correct if mIoU $S$7.
• VQA: Standard accuracy (exact answer or accepted synonym) over generated queries.

These tasks are evaluated over the stratified difficulty bins $S$8.

Table: Image-based VQA Accuracy

Model	$S$9	$R$0	$R$1
InternVL 2.5	49.9	49.2	44.6
Paligemma	48.6	47.5	44.6
Qwen2.5-VL	36.3	33.0	27.7
Human eval	96.8	88.5	87.1

Evaluation reveals significant drops in performance as $R$2 increases, reflecting the increased compositional complexity of queries. Humans outperform all VLMs by a large margin, particularly for multi-entity, multi-relation reasoning (Jahangard et al., 14 Aug 2025).

6. Context, Impact, and Relation to Adjacent Benchmarks

JRDB-Reasoning provides, for the first time in robot perception benchmarking:

• A mathematically formal, task-agnostic metric for reasoning difficulty.
• Structured, dynamic query generation with stepwise workflow annotations.
• Rich fusion of pose, HOI, and geometric context for compositional reasoning in human-rich environments.

This is distinct from previous JRDB derivatives such as JRDB-Act (Ehsanpour et al., 2021), which focuses on spatio-temporal action/social group detection, and from JRDB-Social (Jahangard et al., 2024), which annotates social interactions and group contexts but does not grade or chain reasoning steps. Neuro-symbolic approaches evaluated on JRDB-Reasoning demonstrate that explicit scene graph construction and symbolic predicate filtering significantly enhance spatial query performance over correlation-driven VLMs (Jahangard et al., 30 Oct 2025).

JRDB-Reasoning’s engine facilitates not only model benchmarking but also research into chain-of-thought, interpretable, and hierarchical visual reasoning—key challenges for future embodied AI systems.

7. Future Directions and Limitations

Key open issues highlighted by experimental analyses:

• Visual-LLMs struggle on higher-complexity queries, especially those involving more entities or relationships.
• Stepwise, interpretable evaluation enables pinpointing model failure modes, such as lost context in multi-stage filtering.
• While JRDB-Reasoning enables automated, large-scale benchmark creation, it currently fixes $R$3 (single time slot) per query; extension to multi-frame, temporally compositional queries remains a promising frontier (Jahangard et al., 14 Aug 2025).

A plausible implication is that benchmarks with explicit workflow annotations and difficulty grading can serve as effective tools for accelerating VLM research, especially towards robust, interpretable spatial and social reasoning in challenging multi-agent, embodied settings.