OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction

Abstract: Human-robot collaboration has been studied primarily in dyadic or sequential settings. However, real homes require multiadic collaboration, where multiple humans and robots share a workspace, acting concurrently on interleaved subtasks with tight spatial and temporal coupling. This regime remains underexplored because close-proximity interaction between humans, robots, and objects creates persistent occlusion and rapid state changes, making reliable real-time 3D tracking the central bottleneck. No existing platform provides the real-time, occlusion-robust, room-scale perception needed to make this regime experimentally tractable. We present OmniRobotHome, the first room-scale residential platform that unifies wide-area real-time 3D human and object perception with coordinated multi-robot actuation in a shared world frame. The system instruments a natural home environment with 48 hardware-synchronized RGB cameras for markerless, occlusion-robust tracking of multiple humans and objects, temporally aligned with two Franka arms that act on live scene state. Continuous capture within this consistent frame further supports long-horizon human behavior modeling from accumulated trajectories. The platform makes the multiadic collaboration regime experimentally tractable. We focus on two central problems: safety in shared human-robot environments and human-anticipatory robotic assistance, and show that real-time perception and accumulated behavior memory each yield measurable gains in both.

• The paper introduces a unified real-time multi-camera system for markerless 3D human and object perception to enable complex multiadic interactions.
• It employs advanced calibration with 48 synchronized cameras and distributed edge inference for robust 3D pose reconstruction and coordinated multi-robot actuation.
• Experimental results highlight enhanced safety, anticipatory assistance, and rapid behavior memory integration, significantly reducing collisions and improving cycle times.

OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction

Motivation and Problem Statement

Most research in human-robot collaboration has remained confined to dyadic or sequential interaction models. However, domestic environments often necessitate complex multiadic collaboration, involving simultaneous, spatially and temporally coupled interactions among multiple humans and robots—resulting in persistent occlusion, rapid state changes, and high coordination requirements. Previous robotic and vision platforms lack scalable, real-time, markerless, room-scale systems for unified perception of humans, robots, and objects, making experimental study of such regimes essentially intractable.

System Architecture

OmniRobotHome introduces the first residential platform with wide-area real-time 3D human and object perception, synchronized with multi-robot actuation, all aligned within a shared world frame. The system is instrumented with 48 hardware-synchronized RGB cameras, distributed across 12 edge nodes running parallel 2D pose pipelines, and a central server triangulating temporally aligned full-body human poses and 6D object poses. Two Franka Research 3 arms support real-time actuation based on live world state. This architecture also enables longitudinal collection of behavioral data, facilitating long-range modeling.

Figure 1: System overview of OmniRobotHome, with 48 hardware-synchronized cameras distributed over 12 edge nodes, enabling unified 3D perception of humans, objects, and robots in a shared world frame.

Key engineering advances include:

• Large Sensing Coverage: The 23.1 m² environment is densely covered to eliminate occlusion-induced dead zones.
• Real-Time Multi-Human/Object Perception: A distributed pipeline with YOLO26/RTMPose supports markerless, low-latency 3D pose reconstruction for multiple humans and objects.
• Unified Frame Alignment: Calibration procedures (multi-stage ChArUco/intrinsics/extrinsics, bundle adjustment, and hand-eye algorithms) achieve 1.21 px mean reprojection error, outperforming COLMAP.
• Multi-Robot Actuation: Robot controllers and camera streams are temporally synchronized for fully closed-loop safety and interaction.
• Long-Horizon Behavior Memory: Continuous trajectory capture accumulates personalized, actionable priors for human routine modeling.

Real-Time Perception Pipeline

Robust perception at room scale is supported via distributed edge inference and multi-view, multi-person 3D tracking.

Figure 2: (a) Distributed real-time human pose: YOLO-based detection and RTMPose per camera; fusion via RANSAC triangulation. (b) Object 6D pose with FoundationStereo and FoundationPose; meshes precomputed using MV-SAM3D.

• Human Pose: Each edge node computes INT8 YOLO26 for person detection and FP16 RTMPose for full-body 2D keypoints. All results are fused centrally via RANSAC for 3D keypoints, filtered for temporal stability, and relayed to robot controllers.
• Object Pose: Calibrated RGB stereo pairs are used for learned depth (FoundationStereo), mask-based instance segmentation (YOLO-E), and template-mesh 6D pose estimation (FoundationPose), with mesh precomputation minimzing runtime delay.
• Throughput and Latency: The architecture sustains high throughput even with multiple tracked entities, supporting real-time control in multi-agent settings.

Experimental Scenarios and Benchmarks

Two canonical human-robot challenges were evaluated:

• Safety in Shared Workspaces: Two arms sort objects in a kitchen while a human moves freely. Policies leverage real-time 3D human state for adaptive yielding, rerouting, or pausing operations.

  Figure 3: (a) Safety-aware multi-arm sorting in a shared kitchen; (b) human-anticipatory robotic assistance inferring per-item placement rules from partial demonstrations.

• Anticipatory Assistance: Given partial human demonstrations, the robot predicts the destination for remaining objects, requiring live intent inference and generalization to unseen categories.

Behavior Learning and Memory

Behavior memory facilitates both reactive safety and proactive assistance.

Figure 4: (a) Safety improves as behavior memory accumulates; non-monotonic transitions mark heterogeneous early routines. (b) Intent prediction accuracy shows distinct failure at low demonstration counts, with recovery as examples increase. (c) Individual top-down occupancy evolves as personalized routines solidify.

• Safety: Collision counts drop sharply with minimal recorded data, reflecting early acquisition of dominant approach patterns. There is a non-monotonic phase due to rare atypical trajectories, after which performance rapidly recovers.
• Intent Prediction: Both standard LLM and chain-of-thought models achieve 100% placement accuracy from partial demonstrations, whereas lookup baselines require full demonstration coverage to converge.
• Personalization: As data accrues, occupancy distributions stabilize into individualized spatial routines, enabling subject-specific adaptive policies.

Intention-Aware Robotic Transfer

The system enables intention-aware multi-agent object transfer, adapting to real-time cues in 3D pose and scene context.

Figure 5: (a) Robot delivers a watering kettle upon human attention to a dry plant. (b) Handover of mustard as human prepares a hot dog. (c) Drink delivered upon gesture, with trajectory adapting in real time to human movement.

• Pipeline: The robot detects human pose, uses a VLM to infer intention from pose and object arrangement, and executes a dynamically updated handover trajectory using real-time hand keypoints.
• Robust Adaptivity: The multi-view infrastructure enables closed-loop adaptation for temporally coordinated handover, robust to occlusion and sudden pose change.

Camera Coverage Analysis and System Calibration

Quantitative ablations confirm the necessity and scalability of dense multi-view coverage.

Figure 6: (a)-(b) Insufficiently observed joints decrease with camera count; nearly all joints are reconstructable above 40 cameras. (c)-(d) Per-joint visibility and triangulation error bounds improve and stabilize with coverage; (e)-(h) room volumes rapidly lose unobserved areas with more cameras.

• Calibration: Custom ChArUco-based pipeline achieves lower error and higher consistency than COLMAP, critical for artifact-free 3D state estimation in unpatterned room regions.
• Camera Count: Reducing cameras via farthest-point removal leads to rapid increase in occluded regions and unreconstructable joints, especially for extremities; 48 cameras achieve over 80% 4-view voxel coverage, eliminating practical dead zones.

Quantitative Results

• Safety-Throughput Tradeoff: Adaptive policies conditioned on real-time 3D human state achieve a 2.6× reduction in collisions at improved average cycle time relative to static-radius policies.
• Anticipatory Assistance: LLM-based models reach perfect placement accuracy with only 50% of the demonstration sequence, demonstrating rapid intent generalization.
• Behavior Memory: Both safety and intent inference benefit from early behavior memory, though modes of convergence differ—safety saturates after dominant patterns are learned, while intent demands category-spanning demonstrations.

Implications and Future Directions

Practical Implications:

OmniRobotHome constitutes an enabling infrastructure for systematic, reproducible experiments in multiadic HRI under naturalistic, unstructured conditions. Its real-time, unified world-frame perception, integration of multi-robot actuation, and longitudinal behavior modeling break through the perception-safety bottleneck impeding progress towards domestically viable collaborative assistance.

Theoretical Implications:

The results empirically validate that continuous, wide-area 3D state estimation and behavioral priors are not merely performance optimizers but categorical enablers for task regimes beyond traditional dyadic settings. Individualized behavior memory allows for scalable personalization and adaptive safety not achievable through single-episode or population-level policies, indicating new benchmarks for study in long-horizon interactive autonomy.

Future Research Directions:

• Extension to dynamic or structural environment variation (e.g., home layout changes, additional robotic/mobile agents).
• Deployment in larger, less instrumented settings using partial camera arrays informed by coverage analysis.
• Public release of multi-modal, multi-agent datasets for training and evaluating vision-language-action models and human-aware planning algorithms.
• Integration with open-vocabulary and foundation model pipelines for more generalizable, semantic intent reasoning across tasks.

Conclusion

OmniRobotHome provides the first platform for experimental tractability of multiadic human-robot interaction at-room scale, with robust real-time perception, multi-robot actuation, and behavioral memory. Quantitative and qualitative findings show that dense markerless 3D tracking, unified actuation, and trajectory-accumulated priors enable adaptive and anticipatory collaborative behaviors not possible in prior systems. Limitations include current focus on a single room and fixed-base robots; future work will target mobile robots, more diverse settings, and large-scale data release for the HRI, CV, and robotics communities.

Citation:

For full technical details, see "OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction" (2604.28197).