Object-Reconstruction-Aware Whole-body Control of Mobile Manipulators (2509.04094v1)

Abstract: Object reconstruction and inspection tasks play a crucial role in various robotics applications. Identifying paths that reveal the most unknown areas of the object becomes paramount in this context, as it directly affects efficiency, and this problem is known as the view path planning problem. Current methods often use sampling-based path planning techniques, evaluating potential views along the path to enhance reconstruction performance. However, these methods are computationally expensive as they require evaluating several candidate views on the path. To this end, we propose a computationally efficient solution that relies on calculating a focus point in the most informative (unknown) region and having the robot maintain this point in the camera field of view along the path. We incorporated this strategy into the whole-body control of a mobile manipulator employing a visibility constraint without the need for an additional path planner. We conducted comprehensive and realistic simulations using a large dataset of 114 diverse objects of varying sizes from 57 categories to compare our method with a sampling-based planning strategy using Bayesian data analysis. Furthermore, we performed real-world experiments with an 8-DoF mobile manipulator to demonstrate the proposed method's performance in practice. Our results suggest that there is no significant difference in object coverage and entropy. In contrast, our method is approximately nine times faster than the baseline sampling-based method in terms of the average time the robot spends between views.

• The paper presents a focus-point strategy that integrates visibility constraints into whole-body control to efficiently plan sensor poses for object reconstruction.
• It demonstrates state-of-the-art performance by achieving comparable coverage and entropy metrics to sampling-based planners while reducing computation time by a factor of nine.
• The method leverages robot redundancy for simplified control and generalizability, though it currently assumes known static obstacle positions.

Object-Reconstruction-Aware Whole-body Control of Mobile Manipulators

Introduction and Motivation

This paper addresses the computational challenges inherent in view path planning for object reconstruction tasks using mobile manipulators. The central problem is to efficiently select and traverse sensor poses (Next-Best-Views, NBVs) to maximize object coverage and minimize entropy, while avoiding obstacles and maintaining real-time performance. Traditional sampling-based methods, which evaluate candidate views along the path using ray-casting and information gain (IG) metrics, become computationally prohibitive as model resolution increases. Learning-based approaches, while faster, are constrained by fixed candidate view sets and object sizes, limiting their generalizability.

The proposed method introduces a focus-point-based strategy that directs the robot's camera toward the most informative (unknown) region within the object's bounding box during traversal between NBVs. This is achieved by integrating a visibility constraint into the whole-body kinematic control of the mobile manipulator, obviating the need for additional path planners and candidate view evaluations along the path.

Figure 1: General framework highlighting the contribution: NBV selection via RSV, followed by focus point calculation and whole-body control with visibility and obstacle avoidance constraints.

Technical Approach

Problem Formulation

The workspace is modeled as a 3D Euclidean space $\mathcal{W}$, with the object to be reconstructed $\Omega_{\text{obj}}$ contained within a bounding box $\Omega_{\text{bound}}$. The goal is to explore all regions except the residual space $\Omega_{\text{res}}$, i.e., $$\Omega_{\text{bound}} \setminus \Omega_{\text{res}} = \Omega_{\text{free}} \cup \Omega_{\text{occ}}$$.

Whole-body Kinematic Control

The robot's configuration-space velocities $\dot{\mathbf{q}}$ are mapped to task-space velocities $\dot{\mathbf{x}}$ via the Jacobian $\mathbf{J}$:

$\dot{\mathbf{x}} = \mathbf{J} \dot{\mathbf{q}}$

The control objective is to drive the end-effector to a desired position and direction, $\mathbf{x}_d = [\mathbf{p}_d^T, \mathbf{l}_d^T]^T$, using an optimization-based control law that incorporates:

• Exponential stabilization of task-space error
• Joint velocity damping
• Soft visibility constraint (via slack variables)
• Collision avoidance (via vector field inequalities and softmin distance functions)
• Circulation constraint to mitigate local minima at obstacle boundaries

  Figure 2: Illustration of position and direction objectives for the camera attached to the end-effector.

Focus Point Calculation

At each NBV step, a dynamic focus point is computed in the most informative region (high entropy) of the object's bounding box. This is achieved by:

1. Generating a set of candidate views around a cylindrical search space (40 positions × 5 orientations = 200 views).
2. Using ray-casting within a virtual, expanded FoV to evaluate IG for each ray, based on voxel entropy.
3. Selecting the ray with maximum IG and placing the focus point at a fixed distance along this direction.

   Figure 3: Search space for candidate view generation: 40 positions around a cylinder, each with five orientations, yielding 200 candidate views.

   

   Figure 4: Stages for focus point calculation: virtual FoV, ray-casting, and selection of the most informative direction.

The robot is then controlled to maintain this focus point within its real camera FoV during traversal, enforced via a visibility constraint.

Figure 5: Geometric relationship between the focus point and camera FoV; visibility and restricted zones defined by FoV planes and threshold distance.

Obstacle Avoidance and Circulation

Obstacle avoidance is handled by modeling the robot base and obstacles as circles, using softmin functions for differentiable distance calculations. The circulation constraint ensures the robot does not get stuck in local minima near obstacle boundaries by enforcing tangential motion when close to obstacles.

Figure 6: Simulation environment with cylindrical forbidden region and randomly located circular/wall-type obstacles.

Experimental Evaluation

Simulation Setup

Simulations were conducted using CoppeliaSim, ROS Noetic, and Python, with 114 objects from the ShapeNet dataset and randomized obstacle configurations. The robot is a holonomic base with a 5-DoF arm (total 8 DoF). Three strategies were compared:

• Focus: Proposed method, maintaining an informative focus point via visibility constraint.
• No Path: Direct traversal to NBV without intermediate view selection.
• Sampling: RRT*-based informative path planning with candidate view evaluation.

  Figure 7: RRT

  -based sampling strategy for informative path planning between NBVs.*

Metrics

• Object Coverage: Fraction of the original model's surface points observed.
• Entropy: Sum of voxel entropies within the bounding box.
• Time: Total time spent traversing between NBVs.

Results

Figure 8: NBV calculation and robot movement to NBV while focusing on a point and avoiding obstacles, revealing unknown areas safely.

Figure 9: Real-world experimental setup: object enclosed in a cylinder, four obstacles, and 8-DoF mobile manipulator.

• Coverage and Entropy: Focus and sampling methods yield statistically indistinguishable coverage and entropy, both outperforming the no path baseline.
• Computation Time: Focus method is approximately nine times faster than sampling, with similar time to no path.
• Statistical Analysis: Bayesian data analysis confirms no significant difference in coverage/entropy between focus and sampling, but a significant reduction in computation time for focus.

Figure 10: Average object coverage, entropy, and total time for consecutive NBV steps across 114 simulation runs.

Figure 11: Covered volume, entropy, and total time for consecutive NBV steps in real-world experiment.

Implementation Considerations

• Computational Efficiency: The focus-point strategy eliminates the need for candidate view evaluation along the path, reducing computational load and enabling high-resolution volumetric models.
• Generalizability: The method is formulated in task space, making it applicable to arbitrary robotic platforms.
• Redundancy Utilization: By controlling only position and direction (not full pose), the method leverages robot redundancy and simplifies control.
• Obstacle Knowledge: Current implementation assumes known obstacle locations; future work should address dynamic, unknown environments via onboard sensing.

Implications and Future Directions

The proposed method demonstrates that coverage-aware whole-body control, with a focus-point strategy and visibility constraints, can achieve state-of-the-art reconstruction performance with a fraction of the computational cost. This has direct implications for scalable, real-time deployment of mobile manipulators in inspection, mapping, and industrial automation scenarios.

Future research should address dynamic obstacle avoidance using local sensing, integration with SLAM for unstructured environments, and extension to multi-robot systems for collaborative reconstruction.

Conclusion

This work presents a computationally efficient, coverage-aware whole-body control strategy for mobile manipulators performing object reconstruction. By maintaining an informative focus point within the camera FoV during NBV traversal, the method achieves comparable coverage and entropy to sampling-based planners, while reducing computation time by an order of magnitude. The approach is generalizable, leverages robot redundancy, and is validated in both simulation and real-world experiments. Future work should focus on relaxing assumptions regarding obstacle knowledge and extending the framework to dynamic, unstructured environments.