ManiDreams: Uncertainty-Aware Object Manipulation

• ManiDreams is an open-source Python library that explicitly represents, propagates, and constrains perceptual, parametric, and structural uncertainties in manipulation tasks.
• It employs a distribution-first state abstraction (DRIS) and task-specific intuitive physics to integrate modular dynamics backends and declarative constraint evaluations.
• Empirical results demonstrate improved robustness over standard PPO policies under varied perturbations, achieving higher success rates without retraining.

ManiDreams is an open-source Python library for robust object manipulation that makes uncertainty an explicit, first-class element of the planning loop. Introduced in "ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware Task-specific Intuitive Physics" (Wang et al., 18 Mar 2026), it treats robust manipulation as an integration problem: uncertainties must be represented, propagated, and constrained within action selection rather than merely suppressed during training. The framework wraps any base policy with a sample–predict–constrain loop that evaluates candidate actions against distributional outcomes, adding robustness without retraining, and exposes modular abstractions for distributional state representation, backend-agnostic dynamics prediction, declarative constraint specification, optimization, and execution.

1. Conceptual basis and uncertainty model

The central premise of ManiDreams is that perceptual, parametric, and structural uncertainties compound when handled independently. Instead of committing to a single-point state estimate and a single “best” dynamics prediction, the framework evaluates actions under a distribution of plausible current states and physical contexts, then selects actions that satisfy task constraints over that distribution (Wang et al., 18 Mar 2026).

The uncertainty model is organized around three sources. Perceptual uncertainty is uncertainty over the object’s state $s$, such as pose uncertainty induced by noisy, partial, or delayed sensing, formalized as $s \sim p(s)$. Parametric uncertainty is uncertainty over physical parameters $\theta$ such as mass, friction, and geometry; ManiDreams represents this as a context $c \in C$ and samples discrete instances to approximate $p(\theta)$. Structural uncertainty is uncertainty over the dynamics model $f$ itself, including simulator-reality mismatch or model-class mismatch, and can be handled either by using multiple backends or by accommodating multiple contexts under a single backend.

This induces a Contextual MDP $(S, U, C, P)$ with approximately deterministic transition given context $c$:

$s_{t+1} = f(s_t, u_t, c).$

A plausible implication is that ManiDreams relocates robustness from offline policy training to online decision-time evaluation. In that formulation, the operative object is not an expected trajectory under a nominal model, but a task-conditioned distribution over futures.

2. Distribution-first representation and task-specific intuitive physics

ManiDreams adopts a distribution-first state abstraction called the Domain-Randomized Instance Set (DRIS). Rather than carrying a single state estimate, the planner maintains a set of state–context pairs:

$$D_t = \{(s_t^{(i)}, c^{(i)}) \mid c^{(i)} \in \hat{C}\}_{i=1}^m,$$

where $s \sim p(s)$0 is a discrete set of $s \sim p(s)$1 contexts sampled from the parametric distribution. The projection

$s \sim p(s)$2

stores per-instance states for summary statistics such as mean and variance. The representation is explicitly modality-agnostic: $s \sim p(s)$3 may be a low-dimensional state, RGB, point cloud, or features, while the context distribution remains uniform across modalities (Wang et al., 18 Mar 2026).

Dynamics propagation is handled through Task-Specific Intuitive Physics (TSIP), a backend-agnostic interface that takes a DRIS and an action and returns the next DRIS:

$s \sim p(s)$4

This converts stochasticity at the per-instance level into a deterministic mapping over the distributional representation. The paper describes two shipped backends. SimulationBasedTSIP wraps ManiSkill3 to step all candidates and their DRIS copies in parallel on GPU. LearningBasedTSIP wraps a pre-trained diffusion world model in a feature space $s \sim p(s)$5. Because both implement the same next(dris, action) interface, downstream constraints and solvers remain unchanged when swapping dynamics backends.

Constraint handling is equally distributional. ManiDreams introduces caging constraints that evaluate and validate the spread of the full DRIS rather than a single predicted state. Each constraint exposes evaluate(D), returning a continuous score, and validate(D), returning a binary constraint-satisfaction flag. Built-in variants include geometric cages, trajectory-aligned cages, plate-based cages, pixel-space cages, and composites. This shifts safety and goal progress from state-wise feasibility to set-wise feasibility.

3. Sample–predict–constrain planning and risk-sensitive objectives

The ManiDreams solver closes the loop by sampling candidate actions, predicting their distributional outcomes, evaluating those outcomes against constraints, and selecting the best valid action (Wang et al., 18 Mar 2026). The algorithm described in the paper is:

1. update the cage with the current timestep,
2. synchronize executor state to all TSIP evaluation environments,
3. obtain the current DRIS,
4. sample $s \sim p(s)$6 candidate actions from a policy or optimizer proposal,
5. predict the next DRIS for each candidate,
6. evaluate and validate each predicted DRIS,
7. choose the valid action with minimum score.

The framework supports both one-step online replanning and horizon-based dream(horizon) planning. Planning and execution are decoupled: the inner planning loop can run once per control step for reactive behavior or roll out a horizon $s \sim p(s)$7 and then replay the selected sequence. Because execution maintains its own environment, ManiDreams can plan with one model and execute in a different simulator or on real hardware.

The cost and constraint definitions are Monte Carlo estimators over distributional rollouts. The paper lists four common formulations. The expected cost is

$s \sim p(s)$8

A variance-penalized objective is

$s \sim p(s)$9

For tail-risk control, ManiDreams supports CVaR at level $\theta$0 by sorting per-particle losses and averaging the worst $\theta$1. For hard reliability requirements, it supports chance constraints

$\theta$2

with Monte Carlo estimate

$\theta$3

A notable design choice is that ManiDreams can wrap an existing policy without retraining. A trained policy’s stochastic head can act as the sampler $\theta$4, generating $\theta$5 proposals per step; the solver then filters and ranks them using DRIS-based predictions and cage constraints. Setting num_samples=1 and disabling the cage yields the original policy execution, which makes the comparison to the unmodified baseline operationally direct.

4. Software architecture, extension points, and execution modes

The software is organized in a layered, OMPL-like architecture with abstract interfaces at the top, concrete implementations in the middle, and task-level user configuration at the top of the stack (Wang et al., 18 Mar 2026). The first layer contains the interfaces DRIS, TSIPBase, Cage, SolverBase, and ExecutorBase. The second layer includes implementations such as SimulationBasedTSIP, LearningBasedTSIP, built-in cages, PolicySampler, MPPIOptimizer, an N-best selector, simulation executors, and RealWorldExecutor. The third layer consists of task scene description, pipeline configuration, and an action sampler, without requiring changes to core code.

The single entry point is ManiDreamsEnv, which exposes reset(), step(action), and dream(horizon). This provides Gym-compatible stepping while preserving the explicit planning interface. A typical configuration instantiates a task environment, configures a TSIP backend, defines a cage, wraps a policy as a sampler, and binds a solver such as MPPI or N-best selection.

Propagation semantics are explicit. At each next(), the executor’s current observation initializes all evaluation environments; each evaluation environment spawns $\theta$6 DRIS copies under randomized contexts and steps them under the candidate action. The backend returns per-candidate DRIS objects, optionally with summary statistics for downstream constraint checks. This design supports online mode for reactive tasks such as catching and plan-then-execute mode for slower, more deliberative manipulation.

The framework’s extensibility is unusually broad for a manipulation library. Backends are added by subclassing TSIPBase with next(dris, action) and reset(). Policies are wrapped in PolicySampler. Optimizers can be simple N-best selectors or iterative methods such as MPPI. Executors can be swapped from simulation to real-world execution without changing the planning abstractions. The repository is public at https://github.com/Rice-RobotPI-Lab/ManiDreams (Wang et al., 18 Mar 2026).

5. Empirical results, ablations, and real-world deployment

The paper evaluates ManiDreams on ManiSkill3 tasks PushCube, PickCube, and PushT under three perturbation families: observation noise, observation delay, and physics parameter severity. The baseline is direct execution of a PPO policy with num_samples=1 and no cage. The ManiDreams condition uses the same PPO policy as a sampler with an N-best selector, $\theta$7 candidates, DRIS size $\theta$8, and cage weight $\theta$9 (Wang et al., 18 Mar 2026).

Across all three tasks and all three perturbation types, ManiDreams consistently outperforms PPO, with the largest gains at moderate perturbation magnitudes. The paper further reports task-specific sensitivities: PickCube is resilient to physics randomization but sensitive to delay; PushCube is sensitive to friction and mass changes; PushT is vulnerable to stale observations that accumulate over longer horizons. At extreme perturbation magnitudes, both ManiDreams and PPO degrade, which the authors attribute to the inability of a fixed DRIS to fully cover uncertainty.

The PushCube ablations quantify the effects of particle count, proposal count, and uncertainty width. Increasing DRIS instances from $c \in C$0 to $c \in C$1 changes success from 58% to 86%, with substantial gains up to about $c \in C$2 and saturation thereafter. Increasing solver samples from $c \in C$3 to $c \in C$4 changes success from 52% to 88%, again with diminishing returns. Varying distribution width shows an inverted-U effect: Narrow gives 74%, Medium gives 82%, and Wide gives 76%, indicating that overconfident and overly conservative uncertainty envelopes are both suboptimal.

Runtime overhead is reported per step on PushCube. The PPO baseline runs at 20.2 ms/step. Sample–predict–constrain with different $c \in C$5 settings yields 49.6 ms for $c \in C$6, 56.2 ms for $c \in C$7, and 77.9 ms for $c \in C$8. A diffusion-based TSIP runs at 55.2 ms, comparable to the $c \in C$9 simulation TSIP configuration. The paper characterizes typical manipulation at 10–20 Hz as feasible, while noting that high-frequency dynamic tasks may prefer plan-then-execute mode.

The real-world deployment uses a Franka Panda, Finray soft grippers, and a single NVIDIA RTX 5070 Ti laptop GPU. Sensing is bottom-up RGB under a transparent tabletop, and SAM2 segmentation yields pixel-based DRIS directly in image space. Planning uses LearningBasedTSIP, execution occurs in chunks such as 8 actions per chunk, and DRIS is updated between chunks via SAM2. Reported tasks include cluttered push-pick and wall-corner scooping of flat objects such as steak, pancake, thin box, and pizza. The reported claim is not a benchmark percentage but functional composability: the same sample–predict–constrain loop kept the target within the caged distribution across steps, and no fine-tuning was required for sim-to-real when swapping to RealWorldExecutor (Wang et al., 18 Mar 2026).

6. Position in the literature, terminological ambiguity, and limitations

Within the manipulation literature, ManiDreams differs from world-model and simulator-centric systems that prioritize prediction accuracy or throughput while propagating a single state. It also differs from training-time domain randomization: ManiDreams brings randomization into the online representation through DRIS, so uncertainty is represented, propagated, and constrained during execution rather than only injected during training (Wang et al., 18 Mar 2026). Relative to safety filters such as CBFs or robust MPC, the framework is designed for black-box backends, including simulators and learned predictors, and reasons over distributional outcomes over longer horizons.

The name “ManiDreams” is not unique across recent arXiv usage. In robotics world modeling, “ManiDreams” is given as an alias for “Manipulate in Dream” (MinD), a hierarchical diffusion-based world model that couples low-frequency video imagination and high-frequency control (Chi et al., 23 Jun 2025). In model-based reinforcement learning, Mind Dreamer is described as an instantiation of manifold-based dreaming (“ManiDreams”) centered on Active Latent Intervention, Relay Value Function, and Relay Uncertainty Function (Xu et al., 15 May 2026). By contrast, the mobile manipulation framework DREAM states that there is no separate system named “ManiDreams” in the text, and treats the term only as a query-time interpretation for manipulation with DREAM (Yan et al., 30 May 2026). The same is true for DreamConnect, whose paper explicitly states that it does not reference a system called “ManiDreams” (Sun et al., 2024). A common misconception is therefore to treat all of these as a single lineage; the available texts describe them as distinct systems that share “dream” language but pursue different technical objectives.

The limitations of the ManiDreams library are explicit. Manual configuration of DRIS ranges, DRIS size $p(\theta)$0, and cage geometry or weights remains task-specific. Performance depends on a suitable TSIP backend, and highly discontinuous dynamics may require more expressive predictors or special handling. Latency grows with large $p(\theta)$1 settings, which can challenge very high-frequency control. The authors accordingly recommend starting with modest values such as $p(\theta)$2–$p(\theta)$3 and $p(\theta)$4–$p(\theta)$5, using simple cage geometries tied to the task, and selecting online versus plan-then-execute mode according to the reactive demands of the manipulation problem (Wang et al., 18 Mar 2026).

In that sense, ManiDreams is best understood not as a single learned world model, but as a backend-agnostic robustness layer for manipulation. Its distinctive contribution is to make uncertainty operational at decision time: a distributional state representation, composable intuitive-physics backends, and declarative task constraints are brought into a single sample–predict–constrain loop that can be attached to existing policies with no retraining.