HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation

Abstract: Large-scale demonstration datasets have been central to recent progress in general-purpose robot policies. However, existing datasets are collected in human-absent settings, and policies trained on such data may perform tasks competently in isolation but fail to exhibit human-aware behaviors. To address this gap, we introduce HABIT, a large-scale robot demonstration dataset for human-present environments. We organize tasks into three roles capturing distinct modes of human-robot interaction: Collaborator, where human and robot jointly accomplish a task; Coworker, where they pursue separate tasks in a shared space; and Supervisor, where the human directs the robot. The dataset comprises over 10K episodes and over 160 hours across 60 tasks. Our experiments show that training on human-present data elicits human-aware behaviors that robot-only data fails to produce: spatiotemporal synchronization in Collaborator tasks, yielding in Coworker tasks, and gesture grounding in Supervisor tasks. Moreover, training on HABIT enables rapid adaptation to new human-robot interaction tasks. By introducing human presence as a new axis of dataset diversity, HABIT extends robot policies to environments shared with humans.

• The paper presents a comprehensive human-aware interaction dataset with over 10,000 episodes capturing diverse real-world robot manipulation tasks.
• The methodology leverages structured task workflows and synchronized multi-camera recordings to ensure fine-grained behavioral data and safe human-robot coordination.
• Empirical results demonstrate that models fine-tuned on HABIT achieve faster adaptation and significantly reduced failure rates in complex human-present scenarios.

HABIT: A Large-Scale Human-Aware Behavior and Interaction Dataset for Robot Manipulation

Motivation and Dataset Scope

Traditional robot manipulation datasets privilege human-absent environments, resulting in policies that, while competent at manipulation, fail to internalize the fundamentally social and interactive nature of real-world deployment environments. Robots deployed in shared human environments must coordinate, yield, interpret cues, and avoid conflict with humans—behaviors missing from datasets curated in isolation. The "HABIT: Human-Aware Behavior and Interaction Training Dataset for Robot Manipulation" (2606.31682) directly addresses this deficit, presenting a large-scale, high-diversity demonstration dataset for bimanual robots operating in human-present scenes.

The HABIT dataset comprises over 10,000 episodes and 164 hours across 60 manipulation tasks, carefully structured to capture the critical axes of human-robot interaction:

• Collaborator: Shared goals, physical and temporal coordination, e.g., joint object manipulation or tool handover.
• Coworker: Parallel activities in a shared physical workspace, with safety-yielding and collision avoidance paramount.
• Supervisor: Robot responds to high-level, often ambiguous, human signals such as pointing, requiring intent inference.

  Figure 1: The dataset spans 164 hours of human-robot interaction demonstrations over 60 tasks categorized by interaction mode: Collaborator, Coworker, Supervisor.

Task Structure and Data Collection Protocol

Tasks in HABIT are formalized via explicit "task workflows," each defined as a directed graph over human ($H_i$) and robot ($R_i$) subtasks. Edges encode strict ordering, enforcing the temporal logic underpinning realistic interactions. This formalization supports highly reproducible, structured collection, overcoming ambiguity problems common in high-level language instructions alone.

Figure 2: Sample task workflows and subtask breakdowns, with corresponding human and robot egocentric views.

The workspace comprises two Franka Research 3 arms with Robotiq grippers, teleoperated using a Meta Quest 3 headset. Five synchronized RGB cameras (three robot-side, two human-side) record multimodal video, supporting both fine-grained behavioral annotation and diverse observation modeling.

Figure 3: Workspace setup includes dual-arm robots, separate human and robot work tables, and extensive camera coverage.

Human safety and learnability drive collection protocol design: robots yield by default, human movement speed is varied per episode to foster adaptive timing policies, and human gestures (for Supervisor tasks) are signaled reactively and with randomized onset, ensuring the necessity of real-time visual cue interpretation by the robot.

Dataset Diversity and Role Taxonomy

Dataset statistics confirm substantial per-role heterogeneity. Collaborator and Supervisor roles are dominated by cross-agent edges—demanding fine-grained inter-agent coupling—while Coworker scenarios are dominated by independently ordered but spatially overlapping activities.

Figure 4: The interaction structure is role-dependent, with Collaborator and Supervisor favoring dense cross-agent sequencing (left), yielding high scenario diversity (center), and broad duration spectra (right).

Task and subtask diversity is maximized: HABIT includes 157 unique robot subtasks, 182 unique human subtasks, forming 308 distinct human-robot subtask pairs. Episode lengths and complexity are highly variable, reflecting real-world task demands rather than synthetic regularity.

Evaluation Regime

Success is not confined to object state achievement. Each trial is evaluated on:

• Manipulation success
• Workflow compliance: Temporal and spatial value alignment with human action, including spatiotemporal synchronization or compliance with explicit cues.
• Human safety: Zero-tolerance for human-robot collisions or unintended contact.

Six representative tasks are selected to stress the unique failure modes and coordination requirements intrinsic to each interaction role—e.g., "Table Serving" and "Shelf Cleaning" (Collaborator), "Waste Sorting" and "Box Packaging" (Coworker), "Donut Serving" and "Food Storage" (Supervisor).

Figure 5: Representative tasks evaluated per role: Table Serving & Shelf Cleaning (Collaborator), Waste Sorting & Box Packaging (Coworker), Donut Serving & Food Storage (Supervisor).

Empirical Results

Fine-tuning $\pi_{0.5}$ and GR00T N1.6 VLA models on HABIT induces a consistent improvement in task success rates, especially on tasks where socially competent or intent-driven behavior is essential. The largest empirical gains are observed on Coworker tasks, where yielding to human motion and safe operation in shared workspaces are critical. Supervisor tasks yield gains predominantly where language disambiguation is insufficient and gesture grounding is necessary.

Role-specific failure analysis demonstrates that models trained on robot-only data exhibit precondition failures in Collaborator mode (pre-execution), collisions in Coworker mode, and gesture-misinterpretation in Supervisor mode.

Figure 6: Distinct failure modes per role: Collaborator (precondition violation), Coworker (collision), Supervisor (gesture following failures).

Critically, models trained with HABIT data show substantially reduced rates in all role-specific failure regimes, indicating learned internalization of social context, yielding behavior, and visual intent grounding.

Figure 7: HABIT-trained models dramatically reduce precondition, collision, and gesture-following failures relative to robot-only baselines.

Mid-training on the general HABIT dataset prior to fine-tuning on new tasks enables faster adaptation (sample efficiency): policies reach higher success with fewer demonstrations, highlighting transfer via internalization of human-aware prior knowledge.

Figure 8: Policies with HABIT mid-training consistently reach higher accuracy at lower demonstration counts compared to direct fine-tuning.

Implications and Future Directions

The introduction of the human presence axis as a primary factor in dataset diversity represents a shift in the modeling and evaluation of robot capabilities. HABIT demonstrates that human-aware, socially compatible robot behaviors can be learned from demonstration, not simply engineered or simulated. The strong sample-efficiency results have direct implications for continual learning, few-shot adaptation, and broad generalization in real-world deployments.

Two clear future directions emerge:

1. Incorporating all camera streams: Only robot-side egocentric/exocentric views are used in training, despite richer human-side data that would improve intent grounding, particularly in complex gesture interpretation.
2. Investigating transfer to unseen human-robot interaction roles and multi-agent extensions, including generalization to broader human body and behavior diversity, and adaptation to multi-human, multi-robot scenes.

The robust OOD analysis confirms that the dataset's built-in diversity confers resistance to moderate human-centric shifts in body silhouette and attire; however, continued expansion of the diversity envelope is critical for broader deployment.

Conclusion

HABIT is a comprehensive human-aware behavioral dataset for robot manipulation, addressing fundamental gaps in state-of-the-art policy learning by emphasizing active, co-present human-robot interaction. It provides a robust foundation for research in safe, generalist robotics operating in real, social, and dynamic environments. The results validate the hypothesis that human presence is a crucial, learnable, and transferable axis of data diversity, essential for building genuinely socially compatible robotic foundation models.