OmniRobotHome: A Multiadic Home-Robotics Platform

• OmniRobotHome is a room-scale multiadic home-robotics platform that integrates real-time wide-area human and object perception with synchronized robot actuation.
• It employs 48 hardware-synchronized RGB cameras, distributed inference, and dual Franka arms to achieve robust markerless 3D scene reconstruction in domestic environments.
• Experimental evaluations show measurable improvements in safety and throughput through dynamic safety policies and behavior memory that anticipates human actions.

Searching arXiv for the primary paper and a few closely related home-robotics works to ground the article in the cited literature. OmniRobotHome is a room-scale residential human-robot interaction platform for multiadic collaboration: a regime in which multiple humans and robots share a workspace, act concurrently, and coordinate over interleaved subtasks with tight spatial and temporal coupling. It is presented as the first room-scale residential platform that unifies wide-area real-time 3D human and object perception with coordinated multi-robot actuation in a shared world frame, using 48 hardware-synchronized RGB cameras, distributed inference, and two Franka arms acting on live scene state. Its central thesis is that, in real homes, reliable closed-loop collaboration is limited primarily by occlusion-robust, room-scale, markerless perception rather than by planning or control alone; the reported experiments therefore focus on two downstream problems—safety in shared human-robot environments and human-anticipatory robotic assistance—and report measurable gains from both real-time perception and accumulated behavior memory (Lee et al., 30 Apr 2026).

1. Research setting and conceptual scope

The platform is motivated by a gap between conventional human-robot collaboration research and domestic reality. Much prior work is framed either as dyadic interaction, involving one human and one robot, or as sequential interaction, in which a robot observes a human and acts later. OmniRobotHome instead targets multiadic collaboration, where several humans and robots may work simultaneously in the same room while acting on partially shared objects and subtasks. In this setting, persistent occlusion, rapid object state changes, and tight timing constraints make live 3D scene estimation a prerequisite for experimentally meaningful study (Lee et al., 30 Apr 2026).

This design target distinguishes OmniRobotHome from several adjacent lines of home-robotics research. "HomeRobot: Open-Vocabulary Mobile Manipulation" formulates open-vocabulary mobile manipulation as finding arbitrary objects in unseen homes and placing them on commanded receptacles, emphasizing perception, language, navigation, manipulation, and sim-to-real reproducibility on Hello Robot Stretch (Yenamandra et al., 2023). "Household navigation and manipulation for everyday object rearrangement tasks" studies modular room-to-room object rearrangement using semantic mapping, DETIC, collaborative filtering, Contact-GraspNet, and behavior trees on Fetch (Iyer et al., 2023). "Omni-Manip" addresses a different bottleneck—beyond-FOV humanoid manipulation under viewpoint-limited egocentric sensing—by replacing narrow-FOV RGB-D with head-mounted panoramic LiDAR and a diffusion-based upper-body imitation policy (Qu et al., 5 Mar 2026). OmniRobotHome operates at another level of abstraction: it is not an egocentric manipulation policy and not an OVMM benchmark, but an infrastructure platform for real-time, room-scale, multi-human, multi-robot interaction (Lee et al., 30 Apr 2026).

A common misconception is that room-scale multiadic interaction can be studied adequately with either a single egocentric camera stack or a small number of third-view cameras. The platform is built around the opposite claim. In close-proximity household interaction, humans occlude one another, robot arms occlude people, furniture creates dead zones, and objects disappear during manipulation. This suggests that the central systems problem is not merely wider sensing coverage in the abstract, but sufficiently dense multi-view coverage to make shared-world closed-loop control experimentally tractable (Lee et al., 30 Apr 2026).

2. Room-scale platform architecture

OmniRobotHome instruments a natural home environment as a 23.1 m² living space equipped with 48 RGB cameras operating at 30 Hz, 12 edge nodes for distributed inference, 2 Franka Research 3 robot arms with parallel-jaw grippers, and a central server that aggregates perception into a shared world frame (Lee et al., 30 Apr 2026). The camera subsystem is heterogeneous by design: 24 wide-angle lenses at 3 mm provide broad coverage, while 24 telephoto lenses at 6 mm provide detail. Of the telephoto cameras, 8 are arranged as 4 stereo pairs for object 6D pose tracking. Illumination is provided by 15 LED panels (Lee et al., 30 Apr 2026).

The 48 cameras are distributed across 12 capture nodes. Ten nodes are dedicated to human pose estimation and process 40 cameras, while 2 nodes process the remaining 8 cameras for object tracking through stereo. The raw sensing load is about 13.5 GB/s, which makes centralized raw-video processing impractical and motivates the platform’s distributed design (Lee et al., 30 Apr 2026).

The sensing and actuation stack is temporally aligned. All cameras are synchronized by an external hardware trigger consisting of a 10 Vpp square wave with 50% duty cycle, distributed through hierarchical 8-port GPIO signal distributors. Timestamps from a reference camera are then used to trigger robot data reception, allowing perception outputs and robot control to remain aligned in time rather than merely co-logged post hoc (Lee et al., 30 Apr 2026).

The two Franka Research 3 arms are fixed-base manipulators, not mobile platforms. This is an important architectural fact. OmniRobotHome studies multiadic shared-workspace interaction in a residential room, but it does not present a mobile domestic assistant in the sense of Marvin’s compact 4-mecanum-wheel assistive base for elderly monitoring, remote presence, and night assistance (Eirale et al., 2021). A plausible implication is that the platform is intended first as observability and control infrastructure for tightly coupled interaction, rather than as a complete autonomous whole-home service robot.

3. Calibration and real-time 3D perception

The shared world frame is constructed through a three-stage calibration pipeline. Intrinsic calibration uses ChArUco boards and Zhang’s calibration method. Extrinsic calibration uses 16 distributed ChArUco boards placed in the room: for each camera-board pair the system solves PnP, builds a bipartite pose graph linking cameras and boards, traverses that graph from the most connected camera using BFS, chains the resulting $SE(3)$ transforms to a common origin, and refines the result with bundle adjustment (Lee et al., 30 Apr 2026). The supplement gives the optimization objective as

$$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$

where $T_b \in SE(3)$ is the board pose, $T_c \in SE(3)$ is the camera pose, $\mathbf{K}_c$ are optional intrinsics, $\mathbf{p}_i^{\text{loc}}$ are board corner 3D positions, $\mathbf{u}_i$ are observed image points, and $\Pi(\cdot)$ is the projection function (Lee et al., 30 Apr 2026).

Robot-to-world registration is performed separately through hand-eye calibration. Each Franka base is registered into the same world frame using 20 perturbed robot configurations, Tsai-Lenz initialization, and nonlinear least-squares refinement. The reported chain is

$T^{world}_{board,i}\, X = Z\, T^{base}_{flange,i},$

where $X \in SE(3)$ is the flange-to-board transform and $$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$0 is the robot-base-to-world transform (Lee et al., 30 Apr 2026). A fixed anchor ChArUco board is placed near each robot base so that the base can later be re-localized without repeating full calibration (Lee et al., 30 Apr 2026).

The paper reports quantitative calibration gains over COLMAP in this setting. Mean reprojection error decreases from 1.42 px to 1.21 px, standard deviation decreases from 1.09 px to 0.80 px, reprojections within 1 px increase from 43.9% to 48.9%, and reprojections within 2 px increase from 77.3% to 86.8% (Lee et al., 30 Apr 2026). The stated reason is that the custom pipeline explicitly uses distributed calibration-board placements to cover the relevant workspace, whereas COLMAP degrades where calibration patterns are absent.

Human perception is performed through a distributed markerless multi-human 3D pose pipeline. On each edge node, YOLO26 performs person detection and RTMPose estimates 2D whole-body keypoints; only keypoints and confidence scores are streamed to the central server (Lee et al., 30 Apr 2026). The model outputs 133 keypoints per person in COCO WholeBody format, including body, feet, face, and hands, while the room-scale 3D system reconstructs only the 65 non-face keypoints (Lee et al., 30 Apr 2026). Multi-view aggregation on the server uses RANSAC-based reconstruction for 3D triangulation and a One Euro Filter for temporal smoothing, with $$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$1 Hz and $$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$2; the filter is reset if a joint is missing for more than 10 frames (Lee et al., 30 Apr 2026).

Identity management is handled in two stages. For confirmed persons, torso keypoints are projected into each camera and matched to detections with a cost matrix solved by the Hungarian algorithm. A velocity-corrected prediction bridges brief occlusions, and unmatched 2D detections are later triangulated using DLT to initialize new tracks (Lee et al., 30 Apr 2026). Track confirmation requires 3 consecutive frames, tracks are pruned after 60 missed frames, and a re-identification cache retains pruned identities for 90 frames, or about 3 s (Lee et al., 30 Apr 2026).

Object perception is treated separately through a markerless object 6D pose pipeline built on a calibrated stereo pair. FoundationStereo estimates dense metric depth, YOLO-E provides segmentation masks, and FoundationPose performs marker-free 6D tracking using a template mesh (Lee et al., 30 Apr 2026). When no CAD model is available, the system reconstructs a mesh offline using MV-SAM3D and aligns it to the world frame using silhouette reprojection (Lee et al., 30 Apr 2026). Object tracking runs at about 6 Hz, slower than human tracking but considered sufficient because manipulated objects move more slowly than humans (Lee et al., 30 Apr 2026).

The camera-count study is a core part of the systems argument. Above 40 cameras, unobserved joints become negligible; triangulation error stabilizes beyond about 35 cameras; and at 48 cameras more than 80% of room voxels have at least 4 views, eliminating dead zones across the tested room (Lee et al., 30 Apr 2026). This is the empirical basis for the high-density sensing design.

4. Coordinated actuation and safety in shared workspaces

The perception stack is coupled to two fixed-base Franka Research 3 arms acting in the same shared world frame (Lee et al., 30 Apr 2026). Because both arms and all cameras are registered together, manipulation can be conditioned on live human and object state. The platform therefore supports concurrent manipulation by two arms, cooperative handovers, spatially distributed task execution, and human-aware safety control (Lee et al., 30 Apr 2026).

The principal safety study uses a kitchen organization task in which food items are scattered across a table and kitchen surfaces, two robot arms must sort all items to their correct destinations, and a human moves freely through the workspace (Lee et al., 30 Apr 2026). The arms may pick from the table, hand over between arms, place items at destination locations, and stop or reroute when the human approaches (Lee et al., 30 Apr 2026). The system is therefore evaluated not just on collision avoidance in isolation, but on the throughput–safety tradeoff under live task execution.

The compared safety policies are non-aware, static cylinder, dynamic, and dynamic plus behavior learning (Lee et al., 30 Apr 2026). The static policy uses a fixed safety cylinder around the TCP, with radii of $$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$3 m and $$\underset{T_b, T_c, (\mathbf{K}_c)}{\arg\min} \sum_i \left\lVert \Pi\left(T_c, T_b, \mathbf{p}_i^{\text{loc}}\right) - \mathbf{u}_i \right\rVert^2,$$4 m tested. The dynamic policy instead uses a velocity-adaptive cylinder centered at the handover midpoint and expanded toward the human as approach velocity increases. The dynamic plus behavior-learning policy additionally uses history-based intrusion statistics to enlarge the safety region in directions from which the human historically approaches (Lee et al., 30 Apr 2026).

The reported metrics are average cycle time, human hits, triggers, and fallback time (Lee et al., 30 Apr 2026). Non-aware control is fastest at 19.97 s average cycle time but yields 387.5 human hits. Static control at 0.5 m reduces human hits to 73.5 but increases cycle time to 89.33 s and fallback time to 618.64 s. Static control at 2.0 m nearly eliminates hits at 0.5, but average cycle time rises to 432.53 s and fallback time to 1229.54 s. The dynamic policy reduces collisions to 28.5 with 63.12 s average cycle time, and dynamic plus behavior learning reduces collisions further to 21.5 with essentially unchanged cycle time at 62.81 s (Lee et al., 30 Apr 2026).

These results are used to argue that dynamic safety breaks the standard safety-throughput tradeoff and that behavior memory improves safety further without a meaningful throughput penalty (Lee et al., 30 Apr 2026). The paper summarizes the effect by noting that the dynamic policy reduces collisions to about 38% of the tight static radius, and that behavior memory anticipates the human before entry into the robot’s operating region (Lee et al., 30 Apr 2026).

The safety simulator runs physics at 1000 Hz, renders at 60 Hz, replays 30 Hz human skeleton recordings, and checks collision at 50 Hz (Lee et al., 30 Apr 2026). For collision modeling, the robot TCP is approximated as a sphere of radius 0.08 m and the human body as a vertical cylinder of radius 0.3 m and height 1.8 m (Lee et al., 30 Apr 2026). The platform therefore studies safety as a measurable systems property under replayable, room-scale, temporally aligned interaction rather than as an informal qualitative demonstration.

5. Behavior memory and human-anticipatory assistance

A defining property of OmniRobotHome is that it does not stop at instantaneous 3D scene reconstruction. Because humans, objects, and both robots are continuously tracked in a consistent world frame, the platform accumulates trajectories over time and turns them into behavior memory (Lee et al., 30 Apr 2026). This memory is used in two distinct ways: for anticipatory safety and for intent modeling.

In the safety-memory formulation, the system stores approach episodes, bins human approach bearing into 12 angular bins, tracks intrusion versus passthrough frequency per bin, and applies exponential forgetting to discount old behavior (Lee et al., 30 Apr 2026). The resulting memory modulates the safety cylinder based on historically likely approach directions. The paper reports that early observations already capture dominant approach patterns, that collision count drops sharply as memory accumulates, that returns diminish after around 20 episodes, and that cycle time barely changes, making the improvement nearly free in throughput terms (Lee et al., 30 Apr 2026).

The second memory study concerns human-anticipatory robotic assistance through placement prediction. In the task, a human sorts 12 objects, with 6 fruits going to the sink and 6 packaged goods going to the shelf (Lee et al., 30 Apr 2026). The robot must predict where remaining objects should go after observing only part of the demonstration sequence. The compared methods are Lookup, which copies the last observed placement for each object or category; LLM, which uses Claude Sonnet 4.5 on text descriptions of the demonstrations without explicit category hints; and LLM + Reasoning, which uses the same model with explicit chain-of-thought style reasoning instruction (Lee et al., 30 Apr 2026).

The metric is correct placement ratio as a function of observed demonstration fraction (Lee et al., 30 Apr 2026). At 0% observation, Lookup achieves 50.0 while both LLM variants achieve 58.3. At 25%, Lookup reaches 62.5, LLM reaches 88.9, and LLM + Reasoning reaches 78.5. At 50%, both LLM variants reach 100.0, whereas Lookup reaches 75.0. Lookup reaches 100.0 only at full history (Lee et al., 30 Apr 2026). The reported interpretation is that accumulated behavioral evidence allows the model to infer the latent rule earlier than a direct lookup baseline, although at very low demonstration counts reasoning can overfit misleading patterns (Lee et al., 30 Apr 2026).

The paper also includes a qualitative intention-aware transfer demonstration. A VLM receives a 2×3 mosaic of six camera views, 3D body keypoints, and a candidate object list to infer human intent, after which the robot selects a target object, executes pick-handover-place, and continuously updates the handover target using the human’s real-time 3D hand keypoints (Lee et al., 30 Apr 2026). The examples shown involve a watering kettle for a dry plant, a mustard bottle for a hot dog, and a soft drink for a person eating chips (Lee et al., 30 Apr 2026). This demonstration is presented not as a general-purpose domestic assistant, but as evidence that the platform supports timing-sensitive anticipatory assistance grounded in live multi-view state estimation.

6. Relation to home-robotics literature, limitations, and implications

OmniRobotHome occupies a specific position within home robotics. Its contribution is infrastructural and experimental rather than that of a complete household autonomy stack. HomeRobot standardizes open-vocabulary mobile manipulation across simulation and real Stretch hardware and reports real-world baseline success rates of 15% for a heuristic method and 20% for an RL method, emphasizing that perception is a dominant bottleneck in OVMM (Yenamandra et al., 2023). The household rearrangement system of (Iyer et al., 2023) integrates semantic mapping, object-rearrangement reasoning, manipulation, and semantic navigation through behavior trees for multi-room tidy-up. The one-shot teaching system of (Bajracharya et al., 2019) instead emphasizes task-specific household autonomy through VR demonstration, parameterized behaviors, dense visual embeddings, and task graphs, reporting 85% overall success on three long tasks in two homes. Omni-Manip shows that omnidirectional 3D perception can unlock beyond-FOV humanoid manipulation in viewpoint-limited environments, with 82/120 real-world successes against 18/120, 22/120, and 25/120 for diffusion-policy baselines using narrower perception (Qu et al., 5 Mar 2026). Compared with these systems, OmniRobotHome is primarily a room-scale observability and actuation substrate for multiadic interaction (Lee et al., 30 Apr 2026).

This distinction matters for interpreting the platform’s capabilities. OmniRobotHome is not a mobile robot, does not present a full home navigation stack, and does not claim general domestic autonomy. The paper explicitly points toward future directions involving mobile robots, more varied room layouts, broader embodiment diversity, and releasing synchronized multiadic data to support human-aware vision-language-action models (Lee et al., 30 Apr 2026). A plausible implication is that the current platform should be read as a controlled, instrumented intermediate step: it makes a previously underexplored regime observable and reproducible before attempting broader household generality.

The limitations and assumptions follow from that role. The platform depends on heavy environmental instrumentation: 48 hardware-synchronized RGB cameras, distributed edge computation, illumination infrastructure, and fixed-base robot integration (Lee et al., 30 Apr 2026). Its principal empirical domain is a single 23.1 m² living space rather than many homes. The object pipeline runs at about 6 Hz, slower than the human pipeline (Lee et al., 30 Apr 2026). The assistance experiments use structured sorting rules and few-shot inference from accumulated demonstrations rather than open-ended language-grounded domestic planning (Lee et al., 30 Apr 2026). None of these are hidden constraints; they define the platform’s purpose.

The broader significance of OmniRobotHome lies in making the multiadic home-collaboration regime experimentally manageable. Its three enabling ideas are continuous room-scale perception, a shared world frame that registers humans, objects, and both robots together, and accumulated behavior memory that supports both reactive safety and anticipatory assistance (Lee et al., 30 Apr 2026). This suggests a research trajectory in which home robotics increasingly depends on long-horizon, shared-world representations of human routine and intent, rather than on isolated dyadic interaction loops or purely local robot-centered sensing.