RoboOmni: Dual-Mode Robotics Systems

• RoboOmni is a term for two distinct robotics systems: a retail omni-directional dual-arm platform and an omni-modal framework for proactive manipulation.
• Both systems integrate advanced sensor fusion, combining vision, audio, and text to enable shared-control teleoperation and intention-based action execution.
• Experimental results demonstrate improved task completion, lower collision rates, and reduced operator workload, validating each system's practical robustness.

RoboOmni is a name used in recent robotics literature for two distinct systems with different problem formulations and technical stacks. In one usage, it denotes an omni-directional dual-arm mobile manipulation robot for retail environments, teleoperated through shared control with a human operator and equipped with heterogeneous grippers for single-arm and coordinated bi-manual handling. In another usage, it denotes an end-to-end omni-modal framework for proactive robot manipulation that unifies intention recognition, interaction confirmation, and action execution from visual, auditory, and textual context rather than from explicit commands (Lima et al., 27 Feb 2026, Wang et al., 27 Oct 2025).

1. Disambiguation and problem setting

The retail-store RoboOmni is presented as a teleoperated omni-directional dual-arm mobile robot specifically tailored for use in retail environments. Its central premise is that autonomous robots in retail encounter difficulty adapting to the dynamic nature of retail products and often struggle to operate autonomously in novel situations. The proposed response is a shared-control tele-operation method in which a Virtual Reality motion capture system captures operator commands and transmits them to a remotely located robot, while the robot’s own controller enforces collision and kinematic constraints and supports both single-arm and dual-arm coordinated manipulation (Lima et al., 27 Feb 2026).

The omni-modal RoboOmni is presented in a different research line, motivated by the claim that current Vision-Language-Action approaches largely rely on explicit instructions even though humans rarely issue instructions directly in real-world interactions. Its formulation is therefore proactive rather than reactive: intent is inferred from spoken dialogue, environmental sounds, and visual cues under what the paper calls cross-modal contextual instructions. The framework is organized as Perceiver, Thinker, Talker, and Executor modules built on a single joint token space for vision, audio, text, and robotic actions (Wang et al., 27 Oct 2025).

A common source of confusion is the shared name. The two systems address different levels of the robotics stack: one is a physical mobile manipulation platform centered on shared-control teleoperation in a mock retail setting, while the other is an end-to-end autoregressive model for omni-modal intention recognition and manipulation across simulation and real-world settings. This suggests that “RoboOmni” functions more as a thematic label for omni-directional or omni-modal robotic capability than as a single canonical system.

2. Retail-store RoboOmni: platform architecture and mechanical design

The retail platform uses an omni-directional mobile base with four mecanum wheels of radius $R_w$ placed at the corners of a rectangular chassis with half-lengths $l_x$ and $l_y$, with roller axes at $\pm 45^\circ$ to the chassis axes. The resulting base is holonomic, with full planar degrees of freedom in $[x,y,\theta_z]$ and no steering joints. Onboard computation is provided by an NVIDIA Jetson Xavier TX2, and the sensing suite includes a 2D LiDAR, Intel RealSense D415 RGB-D camera, T265 tracking cameras, an IMU, and wheel encoders (Lima et al., 27 Feb 2026).

The base kinematics are specified explicitly. Let the chassis velocity in the base frame be $v = [v_x, v_y, \omega_z]^T$ and the wheel angular rates be $\omega = [\omega_1,\omega_2,\omega_3,\omega_4]^T$. The inverse-kinematic mapping from body velocity to wheel rates is

$\begin{bmatrix}\omega_1\\omega_2\\omega_3\\omega_4\end{bmatrix} = \frac{1}{R_w} \begin{bmatrix} 1 & -1 & -(l_x+l_y)\ 1 & 1 & (l_x+l_y)\ 1 & 1 & -(l_x+l_y)\ 1 & -1 & (l_x+l_y) \end{bmatrix} \begin{bmatrix}v_x\v_y\\omega_z\end{bmatrix},$

and the forward model is the pseudo-inverse of that matrix. This formulation fixes the platform squarely within standard mecanum-wheel holonomic base control while making the kinematic assumptions explicit.

Manipulation is provided by two UR5e cobots, each with 6 revolute joints, joint torque sensing, maximum payload of 5 kg, and repeatability of $\pm 0.03$ mm. The reported standard UR5e link parameters are approximately $d_1=89$ mm, $l_x$0 mm, $l_x$1 mm, $l_x$2 mm, $l_x$3 mm, and $l_x$4 mm. The workspace is described as a near-spherical envelope of radius approximately 850 mm, with the arms mounted symmetrically at $l_x$5 about the platform centerline. This geometry is paired with heterogeneous end effectors: the left arm carries an OnRobot RG2 two-finger rigid parallel gripper with position and force control and maximum opening of approximately 110 mm, while the right arm carries a custom three-finger soft gripper with a PLA palm, TPU-95A fingers, and three tendon actuators driven by servo motors. One finger base is fixed and two are movable via a meshed gear, allowing reconfiguration between cylindrical and spherical grasp.

3. Shared-control teleoperation, optimization, and grasp strategy

Human input is captured with an HTC Vive system that tracks each hand controller in 6-DOF at approximately 90 Hz. The control assignment is asymmetric: the touchpad axes on the left controller are mapped to base $l_x$6, the right controller touchpad controls yaw $l_x$7, and trigger buttons control gripper open and close. All data are published over ROS topics to the onboard MPC controller. This establishes a human-in-the-loop architecture in which the operator specifies motion intent while the robot handles constraint satisfaction and terminal goal convergence (Lima et al., 27 Feb 2026).

The shared-control layer is formulated as an MPC that blends operator commands and autonomous goal-seeking. Its state and input are $l_x$8, with $l_x$9 denoting end-effector positions, and $l_y$0, with $l_y$1 denoting end-effector velocities. Over horizon $l_y$2, the cost is

$l_y$3

Here, $l_y$4 and $l_y$5 are references obtained from the operator’s recent motion via a constant-velocity Kalman filter; $l_y$6 and $l_y$7 are tracking-weight matrices; and $l_y$8 is a goal-alignment weight. Blending is adapted by

$l_y$9

The dynamics use a point-mass approximation,

$\pm 45^\circ$0

The constrained optimization includes linear floor-and-wall planes $\pm 45^\circ$1, ellipsoidal obstacle avoidance

$\pm 45^\circ$2

dual-arm coupling constraints for bi-manual grasps, and velocity bounds $\pm 45^\circ$3. For a top-down bi-manual approach, the coupling is written as

$\pm 45^\circ$4

with analogous side-grasp constraints involving a $\pm 45^\circ$5 rotation about $\pm 45^\circ$6.

The solver stack is AL-iLQR for constrained MPC, followed by extraction of the first-step Cartesian trajectory, then TracIK for eight-solution joint inverse kinematics, and finally a weighted-least-squares choice of the configuration closest to the current posture while favoring an elbow-away posture:

$\pm 45^\circ$7

The high-level loop runs at 10 Hz: read operator poses to obtain $\pm 45^\circ$8; detect shelf goals $\pm 45^\circ$9 via vision; solve MPC; propagate one step to $[x,y,\theta_z]$0; compute TracIK solutions; select $[x,y,\theta_z]$1; then send joint commands to the UR5e arms and motor voltages to the base. Safety layers include LiDAR-based dynamic speed reduction when obstacles are detected in the commanded direction and the factory-default on-board admittance/impedance control of the UR5e for compliant contact.

Object handling is delegated across the heterogeneous grippers using perception from YOLOv6 plus depth to estimate object size and orientation. If an object is rigid and cuboidal, the RG2 is used with either a side or top grasp depending on shelf space. If an object is compliant or irregular, such as produce or packets, the soft gripper is used, with top approach if width is at most 18 cm and side approach only if depth exceeds 7 cm to form closure. Gripper opening is set by object diameter, and the approach vector is chosen to avoid occlusions. In single-arm tasks, the MPC is decoupled for one end effector and provides collision-free Cartesian paths with autonomous terminal “snap-to-goal” once within approximately 5 cm. In dual-arm tasks, the joint cost tracks both hands while enforcing constant relative pose during long-item extraction.

4. Retail validation, quantitative performance, and failure modes

Validation is conducted in a mock retail environment consisting of a 1.5 m wide three-tier shelf and a drop-box at the robot flank. The operator station is 15 m away, with VR-only visual feedback provided through stereo cameras and depth. The evaluation therefore focuses on remote manipulation under constrained visual telepresence rather than direct line-of-sight operation (Lima et al., 27 Feb 2026).

For single-arm picking with the soft gripper and RG2, the mean completion times are 12 s and 10 s respectively, with a reported success rate of 95%. For dual-arm long-item picking, the mean time is 18 s and the inter-arm-distance drift is at most 5 cm, attributed to the 10 Hz MPC loop. The shared-control design is also compared with pure teleoperation: completion time is reduced by 30%, collisions are reduced from 2 minor bumps in pure teleop to 0 collisions out of 20 trials, and operator workload measured by NASA-TLX is 20% lower with shared control.

The observed failure modes are specific and operationally informative. Grasp failure occurs if object width is less than 7 cm in side approach; the proposed mitigation is custom finger sizes or adaptive finger inserts. Occasional collision with shelf lips under pure teleoperation is reported, and this is prevented by the ellipsoidal MPC constraints. MPC latency at 10 Hz causes minor path jitter, with the stated plan being either to increase loop rate or to off-load the MPC to GPU. These observations delimit the system’s current bottlenecks: grasp geometry at small scales, teleoperation without constraint-aware assistance, and compute-induced temporal smoothness limits.

5. Omni-modal RoboOmni: architecture, contextual instructions, and datasets

The omni-modal RoboOmni is built as a single end-to-end autoregressive model with four logical modules—Perceiver, Thinker, Talker, and Executor—that share one joint token space for vision, audio, text, and robotic actions. The Perceiver ingests visual frames, raw audio, and preceding text turns, producing the unified multimodal representation

$[x,y,\theta_z]$2

where $[x,y,\theta_z]$3 and $[x,y,\theta_z]$4. The Thinker is the omni-modal reasoning backbone, a transformer-style LLM that reasons over $[x,y,\theta_z]$5 and autoregressively emits text tokens, speech tokens, and discrete action tokens. The Talker synthesizes speech from Thinker outputs, and the Executor converts discrete action tokens into continuous robot controls. Action tokenization uses the FAST+ scheme to map 7-D $[x,y,\theta_z]$6-control vectors $[x,y,\theta_z]$7 into short sequences of 2048 discrete symbols (Wang et al., 27 Oct 2025).

Cross-modal fusion follows the Qwen2.5-Omni pipeline for spatiotemporal fusion. Text generation and action generation share the same attention blocks:

$[x,y,\theta_z]$8

$[x,y,\theta_z]$9

After emission of action tokens $v = [v_x, v_y, \omega_z]^T$0, the Executor applies the inverse FAST+ transform to recover continuous controls $v = [v_x, v_y, \omega_z]^T$1. The architectural premise is that direct modeling of speech, environmental audio, dialogue history, vision, and actions in a unified autoregressive stream preserves cross-modal dependencies that are otherwise weakened by cascaded ASR-plus-control pipelines.

The framework’s distinctive task setting is defined through six contextual instruction types. These are sentiment cues, overlapping voices, non-verbal cues, identity cues, dyadic dialogue, and triadic dialogue. In each case, intent is not given by an explicit imperative; rather, it must be inferred from combinations of prosody, temporal overlap, environmental sounds, speaker attributes, and multi-party conversational structure. Visual context is aligned frame by frame with audio and dialogue so that $v = [v_x, v_y, \omega_z]^T$2 and $v = [v_x, v_y, \omega_z]^T$3 co-occur in time, enabling spatiotemporal fusion.

To support this setting, the work introduces OmniAction, comprising 141,162 episodes derived from 74,645 base trajectories. It includes 5,096 distinct speaker timbres across six demographic groups, 2,482 non-verbal audio events, 640 ambient recordings mixed at varied SNRs, 112 manipulation skills, 748 object categories, and roughly equal splits across the six instruction phenomena at approximately 23–24k episodes each. Scenes are sourced from Open-X-Embodiment and then augmented with GPT-4o to rewrite explicit commands into multi-turn dialogues. The generation pipeline has three stages: textual scripting, auditory realization using MOSS-TTS, CosyVoice, and Gemini with CTC-based crosstalk and event insertion, and manual verification, which yields 98.7% intent recoverability. The released simulation benchmark OmniAction-LIBERO contains 240 tasks spanning four LIBERO suites and all six instruction types, plus a “real” split with 10 volunteer recordings.

6. Optimization, benchmarks, deployment, and interpretive significance

Training uses a single autoregressive maximum-likelihood objective over a unified token space $v = [v_x, v_y, \omega_z]^T$4. The conversational and action terms are

$v = [v_x, v_y, \omega_z]^T$5

$v = [v_x, v_y, \omega_z]^T$6

with combined loss

$v = [v_x, v_y, \omega_z]^T$7

Pretraining runs for 10 epochs on approximately 15,360 A100-GPU hours with batch size 512 and learning rate $v = [v_x, v_y, \omega_z]^T$8 with 1k-step warmup. Supervised fine-tuning uses learning rate $v = [v_x, v_y, \omega_z]^T$9 for 10–30k steps on 8 A100 GPUs, and task mixing interleaves conversational and action episodes so that dialogue and manipulation are optimized jointly (Wang et al., 27 Oct 2025).

The simulation benchmarks are quantitatively separated from prior baselines. On OmniAction-LIBERO-TTS, RoboOmni achieves 85.6% overall success, while the best cascaded textual baseline, ASR $\omega = [\omega_1,\omega_2,\omega_3,\omega_4]^T$0 NORA, reaches 25.9%; OpenVLA, OFT, and $\omega = [\omega_1,\omega_2,\omega_3,\omega_4]^T$1 are all below 10%. The suite-level results are 93.0% on Spatial versus best 56.5%, 85.8% on Goal versus 16.3%, 84.0% on Object versus 13.8%, and 79.5% on Long-Horizon versus 51.0%. On OmniAction-LIBERO-Real, RoboOmni reaches 76.6%, compared with 73.8% for $\omega = [\omega_1,\omega_2,\omega_3,\omega_4]^T$2, 40.1% for OpenVLA, 17.4% for NORA, and 6.5% for OFT. In proactive assistance metrics, intent recognition accuracy is 88.9% for RoboOmni, compared with 50% for Qwen2.5-Omni-7B, 55.6% for ASR+GPT-4o, and 27.8% for Qwen2.5-Omni-3B. Inference speed is reported as 0.49× on an RTX 4090, versus 1.0× latency for cascaded ASR+VLA pipelines, with the reduction attributed to elimination of the ASR step. On Spatial tasks, OmniAction-pretrained plus SFT reaches approximately 90% success in 2k steps, whereas from-scratch SFT peaks at approximately 30% even after 20k steps. A cascaded planner-controller using a Qwen2.5-Omni planner feeding text-based VLAs underperforms due to semantic drift and loss of paralinguistic cues.

Real-world deployment uses a WidowX 250S robot arm with onboard GPU inference and real-environment speech from 10 volunteers mixed with background noise. The reported scenarios include lid placement, drawer opening, and pick-and-place among distractors in everyday kitchen tasks. The system is described as robust to varied accents, noise, and overlapping speech, and as capable of proactive clarification exemplified by the query, “Should I put the egg dumpling into the hot pot?” A plausible implication is that the key contribution is not only action prediction but preservation of tone, speaker identity, and environmental context without an ASR bottleneck.

Taken together, the two RoboOmni systems illustrate two non-overlapping but technically adjacent interpretations of “omni” in robotics. The retail platform operationalizes omni-directional mobility, dual-arm coordination, and shared-control MPC for remote manipulation in a structured store mockup. The omni-modal framework operationalizes joint modeling of vision, speech, environmental audio, dialogue, and action for proactive manipulation under latent-intent conditions. The shared name should therefore not be read as indicating a single architecture or research lineage; instead, it identifies two distinct contributions at different layers of robotic autonomy.