---
title: 'ManiDreams: Uncertainty-Aware Object Manipulation'
url: https://www.emergentmind.com/topics/manidreams
type: topic
---

# ManiDreams: Uncertainty-Aware Object Manipulation

ManiDreams is an open-source Python library for robust object manipulation that makes uncertainty an explicit, first-class element of the planning loop. Introduced in "ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware Task-specific Intuitive Physics" [2603.18336], it treats robust manipulation as an integration problem: uncertainties must be represented, propagated, and constrained within action selection rather than merely suppressed during training. The framework wraps any base policy with a sample–predict–constrain loop that evaluates candidate actions against distributional outcomes, adding robustness without retraining, and exposes modular abstractions for distributional state representation, backend-agnostic dynamics prediction, declarative constraint specification, optimization, and execution.

## 1. Conceptual basis and uncertainty model

The central premise of ManiDreams is that perceptual, parametric, and structural uncertainties compound when handled independently. Instead of committing to a single-point state estimate and a single “best” dynamics prediction, the framework evaluates actions under a distribution of plausible current states and physical contexts, then selects actions that satisfy task constraints over that distribution [2603.18336].

The uncertainty model is organized around three sources. **Perceptual uncertainty** is uncertainty over the object’s state $s$, such as pose uncertainty induced by noisy, partial, or delayed sensing, formalized as $s \sim p(s)$. **Parametric uncertainty** is uncertainty over physical parameters $\theta$ such as mass, friction, and geometry; ManiDreams represents this as a context $c \in C$ and samples discrete instances to approximate $p(\theta)$. **Structural uncertainty** is uncertainty over the dynamics model $f$ itself, including simulator-reality mismatch or model-class mismatch, and can be handled either by using multiple backends or by accommodating multiple contexts under a single backend.

This induces a Contextual MDP $(S, U, C, P)$ with approximately deterministic transition given context $c$:
$$
s_{t+1} = f(s_t, u_t, c).
$$
A plausible implication is that ManiDreams relocates robustness from offline policy training to online decision-time evaluation. In that formulation, the operative object is not an expected trajectory under a nominal model, but a task-conditioned distribution over futures.

## 2. Distribution-first representation and task-specific intuitive physics

ManiDreams adopts a distribution-first state abstraction called the **Domain-Randomized Instance Set (DRIS)**. Rather than carrying a single state estimate, the planner maintains a set of state–context pairs:
$$
D_t = \{(s_t^{(i)}, c^{(i)}) \mid c^{(i)} \in \hat{C}\}_{i=1}^m,
$$
where $\hat{C}$ is a discrete set of $m$ contexts sampled from the parametric distribution. The projection
$$
S_t = \mathrm{proj}_S(D_t) = \{s_t^{(i)}\}
$$
stores per-instance states for summary statistics such as mean and variance. The representation is explicitly modality-agnostic: $s_t^{(i)}$ may be a low-dimensional state, RGB, point cloud, or features, while the context distribution remains uniform across modalities [2603.18336].

Dynamics propagation is handled through **Task-Specific Intuitive Physics (TSIP)**, a backend-agnostic interface that takes a DRIS and an action and returns the next DRIS:
$$
D_{t+1} = F(D_t, u_t) = \{(f(s_t^{(i)}, u_t, c^{(i)}), c^{(i)}) \mid (s_t^{(i)}, c^{(i)}) \in D_t\}.
$$
This converts stochasticity at the per-instance level into a deterministic mapping over the distributional representation. The paper describes two shipped backends. **SimulationBasedTSIP** wraps ManiSkill3 to step all candidates and their DRIS copies in parallel on GPU. **LearningBasedTSIP** wraps a pre-trained diffusion world model in a feature space $z_t = \psi(S_t) \in \mathbb{R}^{d_z}$. Because both implement the same `next(dris, action)` interface, downstream constraints and solvers remain unchanged when swapping dynamics backends.

Constraint handling is equally distributional. ManiDreams introduces **caging constraints** that evaluate and validate the spread of the full DRIS rather than a single predicted state. Each constraint exposes `evaluate(D)`, returning a continuous score, and `validate(D)`, returning a binary constraint-satisfaction flag. Built-in variants include geometric cages, trajectory-aligned cages, plate-based cages, pixel-space cages, and composites. This shifts safety and goal progress from state-wise feasibility to set-wise feasibility.

## 3. Sample–predict–constrain planning and risk-sensitive objectives

The ManiDreams solver closes the loop by sampling candidate actions, predicting their distributional outcomes, evaluating those outcomes against constraints, and selecting the best valid action [2603.18336]. The algorithm described in the paper is:

1. update the cage with the current timestep,
2. synchronize executor state to all TSIP evaluation environments,
3. obtain the current DRIS,
4. sample $N$ candidate actions from a policy or optimizer proposal,
5. predict the next DRIS for each candidate,
6. evaluate and validate each predicted DRIS,
7. choose the valid action with minimum score.

The framework supports both one-step online replanning and horizon-based `dream(horizon)` planning. Planning and execution are decoupled: the inner planning loop can run once per control step for reactive behavior or roll out a horizon $T$ and then replay the selected sequence. Because execution maintains its own environment, ManiDreams can plan with one model and execute in a different simulator or on real hardware.

The cost and constraint definitions are Monte Carlo estimators over distributional rollouts. The paper lists four common formulations. The **expected cost** is
$$
J(u) = \mathbb{E}[c(S_{t+1}, u)] \approx \frac{1}{K}\sum_{k=1}^K c(s_{t+1}^{(k)}, u).
$$
A **variance-penalized** objective is
$$
J(u) = \mathbb{E}[c] + \lambda \mathrm{Var}[c].
$$
For tail-risk control, ManiDreams supports **CVaR** at level $\alpha \in (0,1)$ by sorting per-particle losses and averaging the worst $q=\lceil \alpha K \rceil$. For hard reliability requirements, it supports **chance constraints**
$$
P[g(S_{t+1}, u) \le 0] \ge 1-\epsilon
$$
with Monte Carlo estimate
$$
\hat{H}(u) = \frac{1}{K}\sum_{k=1}^K \mathbf{1}\{g(s_{t+1}^{(k)}, u)\le 0\} \ge 1-\epsilon.
$$

A notable design choice is that ManiDreams can wrap an existing policy without retraining. A trained policy’s stochastic head can act as the sampler $\pi$, generating $N$ proposals per step; the solver then filters and ranks them using DRIS-based predictions and cage constraints. Setting `num_samples=1` and disabling the cage yields the original policy execution, which makes the comparison to the unmodified baseline operationally direct.

## 4. Software architecture, extension points, and execution modes

The software is organized in a layered, OMPL-like architecture with abstract interfaces at the top, concrete implementations in the middle, and task-level user configuration at the top of the stack [2603.18336]. The first layer contains the interfaces **DRIS**, **TSIPBase**, **Cage**, **SolverBase**, and **ExecutorBase**. The second layer includes implementations such as **SimulationBasedTSIP**, **LearningBasedTSIP**, built-in cages, **PolicySampler**, **MPPIOptimizer**, an N-best selector, simulation executors, and **RealWorldExecutor**. The third layer consists of task scene description, pipeline configuration, and an action sampler, without requiring changes to core code.

The single entry point is **ManiDreamsEnv**, which exposes `reset()`, `step(action)`, and `dream(horizon)`. This provides Gym-compatible stepping while preserving the explicit planning interface. A typical configuration instantiates a task environment, configures a TSIP backend, defines a cage, wraps a policy as a sampler, and binds a solver such as MPPI or N-best selection.

Propagation semantics are explicit. At each `next()`, the executor’s current observation initializes all evaluation environments; each evaluation environment spawns $m$ DRIS copies under randomized contexts and steps them under the candidate action. The backend returns per-candidate DRIS objects, optionally with summary statistics for downstream constraint checks. This design supports online mode for reactive tasks such as catching and plan-then-execute mode for slower, more deliberative manipulation.

The framework’s extensibility is unusually broad for a manipulation library. Backends are added by subclassing `TSIPBase` with `next(dris, action)` and `reset()`. Policies are wrapped in `PolicySampler`. Optimizers can be simple N-best selectors or iterative methods such as MPPI. Executors can be swapped from simulation to real-world execution without changing the planning abstractions. The repository is public at `https://github.com/Rice-RobotPI-Lab/ManiDreams` [2603.18336].

## 5. Empirical results, ablations, and real-world deployment

The paper evaluates ManiDreams on ManiSkill3 tasks **PushCube**, **PickCube**, and **PushT** under three perturbation families: observation noise, observation delay, and physics parameter severity. The baseline is direct execution of a PPO policy with `num_samples=1` and no cage. The ManiDreams condition uses the same PPO policy as a sampler with an N-best selector, $N=8$ candidates, DRIS size $m=8$, and cage weight $\lambda = 0.1$ [2603.18336].

Across all three tasks and all three perturbation types, ManiDreams consistently outperforms PPO, with the largest gains at moderate perturbation magnitudes. The paper further reports task-specific sensitivities: **PickCube** is resilient to physics randomization but sensitive to delay; **PushCube** is sensitive to friction and mass changes; **PushT** is vulnerable to stale observations that accumulate over longer horizons. At extreme perturbation magnitudes, both ManiDreams and PPO degrade, which the authors attribute to the inability of a fixed DRIS to fully cover uncertainty.

The PushCube ablations quantify the effects of particle count, proposal count, and uncertainty width. Increasing DRIS instances from $m=1$ to $m=32$ changes success from **58%** to **86%**, with substantial gains up to about $m=16$ and saturation thereafter. Increasing solver samples from $N=1$ to $N=32$ changes success from **52%** to **88%**, again with diminishing returns. Varying distribution width shows an inverted-U effect: **Narrow** gives **74%**, **Medium** gives **82%**, and **Wide** gives **76%**, indicating that overconfident and overly conservative uncertainty envelopes are both suboptimal.

Runtime overhead is reported per step on PushCube. The PPO baseline runs at **20.2 ms/step**. Sample–predict–constrain with different $N \times m$ settings yields **49.6 ms** for $4 \times 2$, **56.2 ms** for $8 \times 4$, and **77.9 ms** for $16 \times 8$. A diffusion-based TSIP runs at **55.2 ms**, comparable to the $8 \times 4$ simulation TSIP configuration. The paper characterizes typical manipulation at **10–20 Hz** as feasible, while noting that high-frequency dynamic tasks may prefer plan-then-execute mode.

The real-world deployment uses a **Franka Panda**, **Finray soft grippers**, and a single **NVIDIA RTX 5070 Ti laptop GPU**. Sensing is bottom-up RGB under a transparent tabletop, and **SAM2** segmentation yields pixel-based DRIS directly in image space. Planning uses **LearningBasedTSIP**, execution occurs in chunks such as **8 actions per chunk**, and DRIS is updated between chunks via SAM2. Reported tasks include cluttered push-pick and wall-corner scooping of flat objects such as steak, pancake, thin box, and pizza. The reported claim is not a benchmark percentage but functional composability: the same sample–predict–constrain loop kept the target within the caged distribution across steps, and no fine-tuning was required for sim-to-real when swapping to `RealWorldExecutor` [2603.18336].

## 6. Position in the literature, terminological ambiguity, and limitations

Within the manipulation literature, ManiDreams differs from world-model and simulator-centric systems that prioritize prediction accuracy or throughput while propagating a single state. It also differs from training-time domain randomization: ManiDreams brings randomization into the online representation through DRIS, so uncertainty is represented, propagated, and constrained during execution rather than only injected during training [2603.18336]. Relative to safety filters such as CBFs or robust MPC, the framework is designed for black-box backends, including simulators and learned predictors, and reasons over distributional outcomes over longer horizons.

The name “ManiDreams” is not unique across recent arXiv usage. In robotics world modeling, **“ManiDreams” is given as an alias for “Manipulate in Dream” (MinD)**, a hierarchical diffusion-based world model that couples low-frequency video imagination and high-frequency control [2506.18897]. In model-based reinforcement learning, **Mind Dreamer** is described as an instantiation of manifold-based dreaming **(“ManiDreams”)** centered on Active Latent Intervention, Relay Value Function, and Relay Uncertainty Function [2605.16030]. By contrast, the mobile manipulation framework **DREAM** states that there is **no separate system named “ManiDreams” in the text**, and treats the term only as a query-time interpretation for manipulation with DREAM [2606.00576]. The same is true for **DreamConnect**, whose paper explicitly states that it **does not reference a system called “ManiDreams”** [2408.07317]. A common misconception is therefore to treat all of these as a single lineage; the available texts describe them as distinct systems that share “dream” language but pursue different technical objectives.

The limitations of the ManiDreams library are explicit. Manual configuration of DRIS ranges, DRIS size $m$, and cage geometry or weights remains task-specific. Performance depends on a suitable TSIP backend, and highly discontinuous dynamics may require more expressive predictors or special handling. Latency grows with large $N \times m$ settings, which can challenge very high-frequency control. The authors accordingly recommend starting with modest values such as $N=4$–$8$ and $m=4$–$8$, using simple cage geometries tied to the task, and selecting online versus plan-then-execute mode according to the reactive demands of the manipulation problem [2603.18336].

In that sense, ManiDreams is best understood not as a single learned world model, but as a backend-agnostic robustness layer for manipulation. Its distinctive contribution is to make uncertainty operational at decision time: a distributional state representation, composable intuitive-physics backends, and declarative task constraints are brought into a single sample–predict–constrain loop that can be attached to existing policies with no retraining.

Source: https://www.emergentmind.com/topics/manidreams