Point Cloud Teleoperation: Model-Mediated Control
- The paper demonstrates that point cloud mediation substitutes delayed sensor streams with a structured 3D model, enabling enhanced spatial awareness and control.
- A variety of system architectures—ranging from XR manipulation to tele-ultrasound—use point clouds to integrate visual, haptic, and learned policy cues for robust operation.
- Empirical findings reveal that hybrid interfaces combining point cloud data with localized RGB detail reduce errors and workload while improving task performance.
Point cloud-based model-mediated teleoperation denotes teleoperation architectures in which the remote environment is represented through a mediated $3$D model derived from depth sensing, and the operator interacts with that model visually, haptically, or through a shared-autonomy controller rather than relying solely on raw $2$D video or delayed bilateral feedback. In recent work, point clouds have been used as fused XR scene models for robot manipulation, as environmental perception layers in passthrough AR, as compact patient models for tele-ultrasound force rendering, as live geometry in decoupled-viewpoint humanoid telepresence, as multi-view VR scene reconstructions on mobile devices, and as geometric inputs to learned shared-autonomy policies for dynamic catching (Dincer et al., 4 Apr 2026, Gong et al., 15 Sep 2025, Yeung et al., 18 Sep 2025, Chen et al., 2022, Werner et al., 2024, Zhao et al., 30 Mar 2026).
1. Conceptual foundations
In classical model-mediated teleoperation, the remote environment is modeled locally so that the leader or haptic side can interact with the model instead of the delayed physical system. The tele-ultrasound formulation makes this explicit: the sonographer manipulates a virtual transducer, while force and torque are rendered from a local patient model derived from a point cloud and a volumetric potential field, avoiding continuous low-latency force transmission from the remote follower (Yeung et al., 18 Sep 2025). In visual teleoperation, the same mediating principle appears in a different form. A multi-view XR manipulation system frames the central perceptual trade-off as follows: RGB streams provide sharp local detail but weak depth cues, whereas point clouds provide global $3$D structure but often lack the resolution needed for precise grasping and contact-rich actions (Dincer et al., 4 Apr 2026). An AR demonstration-collection system similarly treats the point cloud not as the control signal itself but as a representation of the remote physical environment and a situational awareness aid that helps the operator localize objects, judge spatial constraints, and operate without entering the robot workspace (Gong et al., 15 Sep 2025). In immersive humanoid teleoperation, mediation is extended further by combining a real-time point cloud with a SLAM-reconstructed mesh, so that the operator sees live sensed geometry for the current view and reconstructed geometry for areas outside the immediate camera frustum (Chen et al., 2022).
Taken together, these formulations show that point cloud-based mediation is not a single interface pattern. It includes at least three recurrent roles: a geometric proxy for depth-rich scene understanding, a local interaction model for haptic rendering, and a geometric context layer that bridges latency, field-of-view mismatch, and partial observability. A plausible implication is that the common denominator is not the rendering primitive itself, but the substitution of a structured geometric model for direct dependence on delayed or viewpoint-constrained sensor streams.
2. System architectures and operational settings
Point cloud-based model-mediated teleoperation has been instantiated in several distinct control architectures. In XR manipulation, one implementation uses a leader-follower architecture with two Franka Emika Panda arms. The operator physically guides the leader arm in gravity-compensation mode, the follower mirrors the motion in the remote workspace, and three static Intel RealSense D415 RGB-D cameras mounted on the left, right, and above the table observe the workspace, while a wrist-mounted Intel RealSense D435 provides local close-up imagery. A desktop PC synchronizes the streams, produces the fused scene representation, and communicates with a Meta Quest 3 over Wi-Fi by sending compact per-point packets containing only attributes (Dincer et al., 4 Apr 2026). In industrial AR teleoperation for Programming by Demonstration, the system is divided into a Robot Real-Time Control Module, a Unity AR System Module, a Spatial Point Cloud Rendering Module, and a Robot Monitoring Module. It uses ABB Externally Guided Motion over UDP with a $250$ Hz control loop, safety-related motion shaping such as MaxSpeedDeviation mm/s and LpFilter Hz, and a robot-mounted Intel RealSense D405 rendered on a Meta Quest 3 passthrough AR headset (Gong et al., 15 Sep 2025). A mobile-phone-based architecture uses a smartphone as the leader device, a Franka Emika robot as the follower, a Cartesian impedance controller, and WebRTC over Wi-Fi or $5$G, with four Intel RealSense D435 cameras providing RGB-D data for a point cloud rendered on the phone in a VR-like interface (Werner et al., 2024).
Other systems shift the mediated model toward telepresence or haptics. A humanoid setup uses an HRP-4CR robot with a ZED Mini stereo camera, an HTC Vive Pro Eye HMD, Unity3D, and ROS-Sharp, with the operator viewing a real-time point cloud fused with a SLAM-generated mesh (Chen et al., 2022). In tele-ultrasound, the follower is a human wearing an MR headset such as Magic Leap 2 and aligning a real probe with a virtual one; a time-of-flight depth camera on the headset scans the patient torso, while a force-sensing shell, IR markers, and ArUco markers provide the measurement and registration signals required for the local patient model (Yeung et al., 18 Sep 2025). In dynamic dexterous catching, the architecture departs from direct leader-follower replication: a Manus glove supplies human teleoperation signals, a diffusion policy supplies a learned action prior, and point cloud observations are integrated into the policy through unsupervised geometric representation learning (Zhao et al., 30 Mar 2026).
These architectures differ in robot embodiment, latency constraints, and sensory topology, but they share the same mediated logic: the operator acts through a virtual or reconstructed scene whose geometry is synchronized with the remote system and exploited for visualization, alignment, compliance, or policy conditioning.
3. Point cloud acquisition, filtering, and rendering pipelines
The practical viability of point cloud-based mediation depends on acquisition and rendering pipelines that suppress clutter, preserve task-relevant geometry, and fit within network and device constraints. In the multi-view XR system, the point cloud is built in a frame-by-frame synchronized pipeline. RGB and depth images from three static cameras are captured simultaneously; each RGB image is processed by a YOLOv11 segmentation model trained on a custom dataset to detect and mask out the robot, gripper, wrist-camera mount, and table; $3$D points are generated only for unmasked pixels by back-projecting depth through the camera intrinsics and assigning the corresponding RGB values; the per-camera clouds are transformed into the Panda base frame, merged, cropped by an axis-aligned $3$D bounding box, and filtered with voxel-grid downsampling and statistical outlier removal. Rendered at $2$0 Hz on an NVIDIA RTX 3060 desktop, the output contains about $2$1 colored points. The Quest reconstructs and draws these points on-device inside Unity using GPU shaders and instanced rendering; without on-device GPU-based point rendering, the headset would be practically limited to roughly $2$2k points (Dincer et al., 4 Apr 2026).
The industrial AR system uses a single robot-mounted RealSense D405, but compensates with explicit post-processing. The selected pipeline applies a threshold filter that keeps data in the $2$3–$2$4 m range, a disparity transform, a spatial filter with magnitude $2$5, alpha $2$6, delta $2$7, a temporal filter with alpha $2$8, delta $2$9, persistence $3$0, and an inverse disparity transform. The reported effect is a more coherent and visually stable point cloud with reduced noise, holes, and temporal jitter (Gong et al., 15 Sep 2025). The mobile-phone VR system uses four D435 cameras at $3$1 and $3$2 fps, compresses color with H.264 using NVENC and depth with Jiffy reimplemented in C++ with TurboPFor for ARM compatibility, then deprojects depth into $3$3D on the phone using camera intrinsics and merges the resulting clouds using known camera extrinsics relative to the robot base (Werner et al., 2024). In the humanoid case, the live point cloud is generated from the ZED camera, while the ZED SDK Spatial Mapping module reconstructs a dense $3$4D mesh online; odometry $3$5 is used to align the mesh and the current point cloud in Unity (Chen et al., 2022).
These pipelines expose a recurring optimization pattern: semantic filtering, spatial cropping, depth post-processing, compact transmission, and GPU-side reconstruction are combined to preserve a usable geometric scene under tight compute and bandwidth budgets. The specific operations vary by platform, but the technical objective is consistent: retain the geometry that matters for manipulation or navigation while discarding structures already known to the system or irrelevant to the task.
4. Visual and haptic mediation strategies
A defining feature of point cloud-based model-mediated teleoperation is that the mediated model is rarely used alone. In the XR manipulation system, the fused point cloud is augmented with a wrist-mounted RGB stream displayed as a floating window near the virtual end-effector. The point cloud supplies the broader geometric context of the workspace and supports perception of distances, clearances, and scene layout, while the wrist RGB restores the detail needed for grasp alignment, insertion, and contact-rich manipulation. The authors explicitly frame this as resolving the “global structure vs. local detail” problem. They also report that the wrist view helps with the fuzzy edges and noise observed in the point-cloud-only condition (Dincer et al., 4 Apr 2026). In the AR Programming by Demonstration system, the point cloud is rendered “in front of” the virtual robot to improve spatial awareness, object localization, depth judgment, and grasp or placement alignment, but users noted that the point cloud should complement, not replace, direct visual observation for precise work (Gong et al., 15 Sep 2025).
For telepresence, mediation can be aimed at viewpoint coherence rather than manipulation detail. The humanoid teleoperation system decouples the operator’s HMD motion from the robot camera motion and renders the point cloud in virtual space while a SLAM-reconstructed mesh fills regions outside the live view. This addresses three impairments identified in the paper: latency between the operator’s and robot’s head motions, camera-HMD field-of-view mismatch, and mismatch between human and robot neck range of motion. The mesh is intentionally colored slightly differently from the real-time point cloud so that reconstructed geometry is not confused with live sensor data. To reduce jitter and discomfort, the system applies a low-pass filter to the virtual tracking space; at a filter rate of $3$6, the additional latency is approximately $3$7 ms (Chen et al., 2022).
In haptic mediation, the point cloud becomes part of a physically queried local model. In tele-ultrasound, the patient surface is captured as a point cloud, converted into a structured point cloud representation in cylindrical coordinates, and then voxelized into a volume where each voxel $3$8 stores a scalar potential $3$9. The ultrasound probe is modeled as a dynamic point shell. Contact forces and torques are computed from overlap between the point shell and the voxelized patient model:
0
The model is updated by solving the convex quadratic
1
with 2, thereby combining smoothness from Laplace’s equation with measured force and torque data (Yeung et al., 18 Sep 2025). A lower-fidelity haptic variant appears in the mobile-phone system, where contact is relayed through the phone’s single haptic actuator using impulse-like feedback for first contact and cyclic amplitude-modulated feedback for sustained contact, with a 3 N activation threshold used for contact detection (Werner et al., 2024).
These examples show that point cloud mediation can be visual, haptic, or hybrid. The mediated model may remain purely perceptual, or it may be queried as an interaction model that generates force and torque locally.
5. Shared autonomy and learned geometric mediation
Point cloud-based mediation has also been extended from visualization and local haptic models to learned control priors. Tele-Catch addresses dexterous dynamic object catching, a setting in which pure teleoperation often fails because of timing errors, pose errors, force errors, and retargeting errors between human and robot hands. Its framework has four stages: training an RL policy in simulation, collecting successful trajectories 4, training a diffusion policy with unsupervised 5D representations from point clouds, and teleoperating through a dynamics-aware adaptive integration mechanism called DAIM (Zhao et al., 30 Mar 2026).
DAIM injects teleoperation into the diffusion denoising process rather than treating human input as a hard external override. If 6 is the diffusion-policy action at denoising step 7 and 8 is the retargeted glove action, the blended action is
9
with
$250$0
and
$250$1
The paper sets $250$2, $250$3, $250$4, $250$5, and $250$6. As object speed or angular velocity rises, $250$7 decreases, so the controller places more weight on the learned policy and less on the glove signal (Zhao et al., 30 Mar 2026).
The geometric side of the method is DP-U3R. A point cloud $250$8 is perturbed by Gaussian noise, encoded into point-wise features $250$9, pooled into a global descriptor 0, and fused through an attention module into a learned representation 1. The diffusion policy is then conditioned on an augmented state 2. Training combines a point cloud reconstruction loss and a diffusion noise prediction loss, with 3 and 4 (Zhao et al., 30 Mar 2026). This suggests an expansion of model-mediated teleoperation beyond explicit geometric rendering: the mediated model can also take the form of a geometry-aware policy prior that stabilizes human control in dynamic tasks.
6. Empirical findings, recurrent limitations, and interpretive issues
The strongest controlled evidence in the set comes from the XR manipulation study with 5 participants. Using a within-subject design across cup insertion, T-shape assembly, and a wire-loop task, the study compared four modalities: RGBs, PC, PC+RGB, and OT. The reported outcome was that PC+RGB achieved the best overall performance on success, completion time, workload, usability, and preference, while the point-cloud-only condition also outperformed RGB streams and OpenTeleVision-style stereo video. The paper attributes this to two mechanisms: preservation of visual-motor co-location in the fused spatial frame of the leader arm, and reduction of the cognitive burden required to integrate multiple 6D views (Dincer et al., 4 Apr 2026). In the industrial AR study, a smaller within-subject experiment with 7 participants compared teleoperation with and without point cloud rendering. The system with point cloud support yielded an overall 8 increase in completion count and a 9 reduction in critical errors. SUS increased from 0 to 1, about 2, but the difference was not statistically significant, and the study explicitly reports post-hoc power of only 3, indicating that it was underpowered (Gong et al., 15 Sep 2025). In tele-ultrasound, evaluation on 4 volunteer patients showed that adding measured forces to the potential-field model reduced the force magnitude error by an average of 5 N and the force vector angle error by an average of 6 relative to using only Laplace’s equation (Yeung et al., 18 Sep 2025). The humanoid paper reports a qualitative user comparison with 7 participants rather than formal statistical testing, but describes increased perceived freedom and reduced waiting for the delayed view to recenter when using point cloud plus mesh (Chen et al., 2022).
Several limitations recur across domains. Point clouds improve spatial awareness, but close-range quality can degrade during fine manipulation, adjacent objects may merge, and visual noise, flicker, and fuzzy edges remain practical issues (Gong et al., 15 Sep 2025, Werner et al., 2024). Single-camera rendering may fail to capture side and rear obstacles, motivating multi-sensor fusion; the AR paper notes a proof-of-concept with two depth cameras as a promising direction (Gong et al., 15 Sep 2025). In XR telepresence, mesh quality and odometry precision limit alignment between reconstructed and live geometry, while low-pass filtering trades responsiveness for comfort (Chen et al., 2022). In mobile teleoperation, the major weakness is reported to be visual feedback quality rather than pose tracking, and Jiffy depth streaming is vulnerable to packet loss (Werner et al., 2024). In shared-autonomy dynamic catching, point clouds strengthen geometry-aware control, but the method still struggles with some thin or sharp-edged objects and relies heavily on simulation during development (Zhao et al., 30 Mar 2026).
A common misconception is that point clouds uniformly replace conventional sensing or direct visual feedback. The cited systems do not support that conclusion. In one line of work, point clouds outperform RGB-only interfaces yet still benefit substantially from a wrist-mounted RGB inset (Dincer et al., 4 Apr 2026). In another, users explicitly state that point clouds should complement, not replace, direct visual observation for precise work (Gong et al., 15 Sep 2025). A related misconception is that immersive stereo video alone is sufficient if the display is engaging. The XR study argues otherwise: OpenTeleVision-style ego-centric stereo video can introduce a proprioceptive-visual mismatch because the operator does not see the leader arm they are physically moving, so immersion alone is not sufficient (Dincer et al., 4 Apr 2026). The literature therefore converges on a narrower but more technically precise conclusion: point cloud-based mediation is most effective when it supplies coherent 8D structure and is paired with other mechanisms—localized RGB detail, reconstructed spatial memory, measurement-augmented force models, or geometry-aware shared autonomy—that compensate for its own failure modes.