RoboGhost: Direct Robot Control & Perception

• RoboGhost is a set of advanced frameworks that bypass intermediate representations to directly convert high-level inputs into executable robot actions using end-to-end diffusion models.
• It integrates hierarchical sub-goal planning with AR-based GhostObjects to enhance manipulation tasks and facilitate seamless human-robot interaction.
• Its robust full-waveform LiDAR processing improves mapping and object detection by significantly reducing errors, boosting overall navigation success rates.

RoboGhost denotes a set of frameworks and system designs that bypass intermediate representations and retargeting in robot perception, instruction, and control, instead translating high-level user inputs or raw sensor data directly into executable robot actions or cleansed percepts. The term spans three principal domains: (1) augmented reality (AR) interfaces that allow users to manipulate virtual twins ("GhostObjects") of real objects for goal specification; (2) hierarchical visuomotor control architectures factorizing manipulation into sub-goal planning and low-level execution; and (3) end-to-end diffusion-based locomotion policies conditioning robot behavior on high-level motion latents extracted from language or other modalities. RoboGhost systems are also associated with robust perception in mobile robotics, leveraging full-waveform LiDAR to suppress spurious "ghost" returns, thus facilitating safer navigation and mapping.

1. Retargeting-Free Robot Control and Semantic Action Inference

RoboGhost, as introduced in "From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance" (Li et al., 16 Oct 2025), reframes the text-to-locomotion pipeline for humanoid robots. Classical pipelines:

• Parse text to human demonstration sequences.
• Retarget human kinematics to robot morphologies.
• Track reconstructed poses with a motion controller.

However, these stages are prone to cumulative error, semantic drift, and high latency (~17.85 s). RoboGhost eliminates all explicit pose retargeting. The architecture is centered on a hybrid causal transformer-diffusion stack:

1. Motion generator $G$ maps a natural language prompt $T$ to a compact motion latent $l_{\rm ref}\in\mathbb{R}^{64}$.
2. Diffusion policy $\pi_s$ receives $l_{\rm ref}$, a proprioceptive state window $p_o, o_{t-H:t}$, and denoises Gaussian noise into $\mathbb{R}^{23}$-dimensional joint targets for direct control.
3. Conditioning is enforced at every diffusion step using adaptive layer normalization (AdaLN), ensuring semantic fidelity of action to prompt content.

This latent-driven diffusion policy cuts total system latency to 5.84 s and improves motion tracking metrics (e.g., mean per-joint error falls by ≈50%) while also supporting direct extension to other input modalities such as vision or audio. This direct, retargeting-free pathway is also robust across simulators (IsaacGym, MuJoCo) and real-hardware deployments.

2. Hierarchical Sub-Goal Policies for Embodiment-General Manipulation

The GHOST framework ("Hierarchical Sub-Goal Policies for Generalizing Robot Manipulation" (Krishna et al., 8 Jun 2026)) embodies a complementary paradigm for RoboGhost: instead of monolithic visuomotor policies, control is factorized:

• High-level planner predicts a distribution over 3D sub-goal (end-effector) poses using multi-view RGB-D observations ($g_t \in \mathbb{R}^{3\times 4}$), instructed optionally by language.
• Low-level controller (Diffusion Policy) executes short action sequences conditioned on a target sub-goal represented as a heatmap projected into each camera’s image plane.
• Sub-goals form an interface bottleneck that is fundamentally embodiment-agnostic, allowing direct incorporation of human demonstration (from video or hand mesh) without action retargeting.

Hierarchical training proceeds with negative log-likelihood loss at the sub-goal (planner) level and diffusion loss at the goal-conditioned controller, plus standard data regimes ($$\mathcal{D}_{\text{robot}}, \mathcal{D}_{\text{human}}$$). Empirical evaluation demonstrates a 4–6× increase in success rates over flat diffusion policies for pick-and-place, cloth-folding, and hammer-pin tasks, and near-doubling of sample efficiency for novel-object generalization. Ablation shows that the hierarchical factorization itself is a major driver of generalization, enabling zero-shot adaptation when high-level sub-goal predictions are accurate.

3. Augmented Reality Front-Ends and Spatial Manipulation Interfaces

"GhostObjects: Instructing Robots by Manipulating Spatially Aligned Virtual Twins in Augmented Reality" (Wang et al., 14 Aug 2025) formalizes a "RoboGhost" AR front-end layer, in which users manipulate world-aligned virtual twins ("GhostObjects") of real objects. This system architecture entails:

• Hardware: AR headset (e.g., Meta Quest 2) with inside-out tracking, hand-held controllers, and a ROS-controlled robot (manipulator or mobile).
• Software:
  • Unity engine with Meta's Mixed Reality Utility Kit (MRUK) generates spatial anchors and overlays GhostObjects.
  • A coordination bridge abstracts AR events (object selection, pose update, actions) to ROS topics.
  • Robot-side planner (e.g., MoveIt!) handles actuation based on goal poses.

Key procedural steps include:

• Initialization by room scanning and GhostObject spawning at anchor sites.
• User interacts via raycasting, volumetric lasso selection, and direct grab/drag, with snapping heuristics for common goal poses.
• Finalized GhostObject poses are mapped (using three-frame SLAM and hand-eye calibration, $T_{h→w}$, $T$0) directly into 6-DOF robot goal frames for execution.

Interaction primitives are structured: single-object raycast, multi-object lasso, direct manipulation with optional physics, and default pose snapping. Execution mapping is modular: completed user manipulations emit high-level ROS messages (e.g., PickAndPlace, FillAction), which robot task adapters translate into motion plans.

4. Full-Waveform LiDAR for Robust Perception in Mobile Robotics

In mobile and autonomous robotics, "ghost points"—false LiDAR returns resulting from multi-path reflections—pose severe challenges for mapping and localization. "Ghost-FWL: A Large-Scale Full-Waveform LiDAR Dataset for Ghost Detection and Removal" (Ikeda et al., 30 Mar 2026) provides a large-scale benchmark and baseline for tackling this problem with full-waveform LiDAR (FWL):

• Dataset: 24,412 annotated frames, 7.5 billion peak-level labels (Object, Glass, Ghost, Noise), across 10 diverse indoor/outdoor scenes. This scale is 100× larger than all prior annotated FWL datasets.
• Labeling protocol: Combines SLAM-based map carving and waveform analysis; peaks are tagged as ghosts if their spatial deviation exceeds a threshold relative to true object regions, or if classified by their incidence on reflective/glass surfaces.
• Network architecture: Patchified ViT-style encoder, per-patch focal loss optimized for high recall on rare ghost events ($T$1), and a masked autoencoder (FWL-MAE) for data-efficient self-supervised pretraining.
• Results: Ghost removal rate 0.918 (recall 0.751), lowering SLAM trajectory errors by ~66–84% and reducing ghost-induced false positives in 3D object detection by a factor of 50×.

This robust FWL processing pipeline, when integrated into robot perception stacks, directly improves mapping reliability and safety in challenging reflective environments—an essential capability for any mobile RoboGhost instantiation.

5. System Architectures and Cross-Domain Design Patterns

Although these RoboGhost instantiations serve different subfields, common architectural principles emerge:

Domain	High-Level Semantics	Intermediate Representation	Execution Policy	Sensor/Interface Layer
Manipulation	Sub-goals (3D pose, language)	3D keypoints, heatmaps	Diffusion Policy/Controller	RGB-D, AR UI
Locomotion	Motion latent (from text)	Vector latent ($T$2)	Diffusion-based Action Denoiser	Language, proprioception
Perception	Scene geometry	Full-waveform histograms	Transformer Detector/MAE	FWL LiDAR

Across these configurations:

• Intermediate representations, when present, are chosen for embodiment-agnosticism or information parsimony (e.g., keypoints, latents, histograms).
• Policy architectures (diffusion models, transformers) directly condition on these compact representations via strong injective mechanisms (AdaLN, cross-attention).
• Sensor interface layers are either user-facing (AR) or raw physics-based (LiDAR, proprioceptive).

This design enables efficient generalization, cross-modal alignment, and end-to-end training without brittle retargeting stages.

6. Limitations, Evaluation, and Future Directions

Several open challenges are explicit:

• GhostObjects AR: No formal user or quantitative robot-task study is reported, and the method presumes static room calibration and asset datasets, with no handling of long-term AR drift (Wang et al., 14 Aug 2025).
• GHOST Manipulation: Annotation of sub-goal boundaries and alignment between human/robot imagery are non-trivial, and current architectures do not yet support rich multi-affordance action spaces (e.g., tool use, multi-arm) (Krishna et al., 8 Jun 2026).
• Full-waveform Perception: Mobile LiDAR ghost detection is limited in real-time speed (≈2.35 FPS), and extension to adverse environments (rain, fog, reflective metals) remains future work (Ikeda et al., 30 Mar 2026).
• Latent-Guided Locomotion: Policy transfer to hardware, while improved, assumes a privileged teacher pipeline, and full statistical significance of tracking improvements is not reported (Li et al., 16 Oct 2025).

Proposed research avenues include automated sub-goal discovery (e.g., change-point detection), domain-adversarial pretraining for human-robot visual alignment, bimanual and multi-affordance action extensions, temporally smoothed sub-goal planning, and interactive human-in-the-loop refinement (Krishna et al., 8 Jun 2026). For AR front-ends, automated asset/model generation (using scene reconstruction and vision-LLMs) and robust relocalization architectures are prioritized.

7. Comparative Performance and Impact

Quantitative evaluation across these RoboGhost paradigms demonstrates substantial empirical gains:

• RoboGhost Diffusion Policy: +5–12 pp success margin, ≈48% reduction in tracking errors, 3× lower command latency compared to explicit pipelines (Li et al., 16 Oct 2025).
• GHOST Hierarchical Manipulation: 4–6× improvement in pick-and-place and cloth-folding success rates, 50%+ gains on unseen tasks when incorporating human demonstration (Krishna et al., 8 Jun 2026).
• Ghost-FWL Perception: 66–84% SLAM error reduction, 50× fewer false positives in object detection due to ghost artifacts (Ikeda et al., 30 Mar 2026).

Such results establish the RoboGhost design pattern as a keystone for bridging open-ended instruction, reliable perception, and precise end-effector execution in next-generation autonomous and human-interactive robotics. Each RoboGhost instantiation demonstrates the advantages of bypassing intermediate, error-prone retargeting and leveraging compact, semantically meaningful representations directly in policy and perception architectures.