DrivePI: Robust Control & 4D Perception

• DrivePI is a unified framework that combines DRPI's robust stochastic control with a 4D multi-modal LLM for comprehensive autonomous driving tasks.
• The DRPI component leverages distributionally robust optimization and path integral control to achieve higher success rates and lower arrival times in dynamic simulations.
• The 4D MLLM module fuses LiDAR, multi-view images, and textual instructions to enhance scene understanding, occupancy prediction, and planning accuracy.

DrivePI refers to two distinct frameworks at the forefront of control and perception in autonomous systems: the Distributionally Robust Path Integral (DRPI) control scheme for stochastic optimal control under uncertainty (Park et al., 2023), and the spatial-aware 4D Multi-Modal LLM (MLLM) for unified understanding, perception, prediction, and planning in autonomous driving (Liu et al., 14 Dec 2025). Both frameworks address robustness and unification in their respective domains through advanced algorithmic and architectural innovations.

1. Distributionally Robust Path Integral Control (DRPI): Foundations

DRPI addresses continuous-time, continuous-space stochastic optimal control where the true diffusion process is unknown and only limited historical disturbance data are available. The system evolves as

$$dx(t) = f(x(t),t)dt + G(x(t),t)u(x(t),t)dt + \Sigma(x(t),t)d\xi(t),$$

where $x(t)\in\mathbb{R}^n$, $u(x(t),t)\in\mathbb{R}^k$, $\xi(t)$ follows the uncontrolled diffusion $d\xi(t) = \mu(t)dt + dw(t)$ with unknown drift $\mu(t)$ and Brownian motion $w(t)$. The controller only has access to a nominal law $\mathbb{Q}$, typically derived from empirical drifts or simulations.

The objective is to minimize expected trajectory cost $J_u(x(\cdot))$, consisting of a running cost $L_u(x,t) = q(x,t) + \frac{1}{2}u^\top Ru$ and a terminal cost $x(t)\in\mathbb{R}^n$0, evaluated as

$x(t)\in\mathbb{R}^n$1

The classical risk-neutral optimal control formulation $x(t)\in\mathbb{R}^n$2 is infeasible under model ambiguity, motivating a robust approach.

2. Distributionally Robust Optimization and Theoretical Guarantees

DRPI casts the robust control problem as a distributionally robust optimization (DRO) over a Kullback–Leibler (KL) ball of radius $x(t)\in\mathbb{R}^n$3 centered at $x(t)\in\mathbb{R}^n$4:

$x(t)\in\mathbb{R}^n$5

The objective becomes

$x(t)\in\mathbb{R}^n$6

with the robust controller solving $x(t)\in\mathbb{R}^n$7. This min–max formulation is equivalently reformulated via duality and the Donsker–Varadhan variational principle as

$x(t)\in\mathbb{R}^n$8

Theoretical results guarantee that, with probability at least $x(t)\in\mathbb{R}^n$9 (where $u(x(t),t)\in\mathbb{R}^k$0 is the number of empirical trajectories), the true law $u(x(t),t)\in\mathbb{R}^k$1 falls within the ambiguity set; explicit finite-sample bounds for $u(x(t),t)\in\mathbb{R}^k$2 are provided.

3. Path Integral Control and DRPI Algorithmic Implementation

Path Integral Control (PIC) exploits linearity under the exponential value function transform when $u(x(t),t)\in\mathbb{R}^k$3. In the risk-neutral case, the optimal control is a path-integral average over sampled trajectories:

$u(x(t),t)\in\mathbb{R}^k$4

For DRPI, the path weights become $u(x(t),t)\in\mathbb{R}^k$5, with $u(x(t),t)\in\mathbb{R}^k$6 an affine function of $u(x(t),t)\in\mathbb{R}^k$7; optimal policies are thus computed by weighted Monte Carlo averages over sampled disturbance trajectories. The control at each step is estimated as

$u(x(t),t)\in\mathbb{R}^k$8

where $u(x(t),t)\in\mathbb{R}^k$9, with $\xi(t)$0 the uncontrolled trajectory cost.

Parameter selection for $\xi(t)$1 is guided by the finite-sample rate for out-of-sample robustness, and online updating of drift estimates and $\xi(t)$2 is supported in deployment.

4. Experimental Evaluation of DRPI

DRPI was empirically evaluated on simulated autonomous control tasks involving double-integrator and unicycle dynamics with obstacle and boundary avoidance. Using online drift estimates and adaptive $\xi(t)$3, DRPI was benchmarked against risk-neutral PIC ($\xi(t)$4) across 100 Monte Carlo trials per environment.

Performance metrics included success rate (no collision), mean arrival time, standard deviation, and 95th percentile arrival time. For the 2D double integrator, DRPI yielded substantially higher success (96% vs. 66%), lower mean time (8.0s vs. 21.3s), and reduced variance. In the unicycle model, DRPI again achieved markedly higher success (78% vs. 19%) and lower mean/variance in time.

5. DrivePI: 4D Spatial-aware Multi-modal LLM for Autonomous Driving

DrivePI (Liu et al., 14 Dec 2025) extends the notion of unified and robust reasoning to vision-language-action loops in autonomous driving. It is a spatial-aware 4D MLLM architecture that fuses LiDAR point clouds, multi-view images, and language instructions to simultaneously address scene understanding, 3D occupancy, motion prediction (occupancy flow), and planning.

The architecture includes:

• Multi-modal vision encoder producing BEV feature maps.
• Spatial-projector patchifies BEV features, pools, applies cross-attention to maintain spatial detail, and projects to vision tokens.
• MLLM Backbone (Qwen2.5, 0.5B parameters) fusing vision and text tokens.
• Four task heads: text (scene QA), 3D occupancy, occupancy flow, and action diffusion for planning.

A unified multi-task loss is used:

$\xi(t)$5

where each task loss is defined using state-of-the-art metric conventions (e.g., FlashOcc losses for occupancy).

6. Training Data Engines, Modes, and Quantitative Performance of DrivePI

A large-scale data engine generates multi-turn QA pairs linking occupancy/flow/planning ground truth to textual instructions, supporting both language and spatially grounded learning. Three annotation strategies are employed:

• Caption annotation (via InternVL3-78B).
• 4D spatial understanding QA (≈420k occupancy/class and 140k flow QA).
• Planning reasoning QA (24k high-level actions), totaling ≈1M QA pairs.

DrivePI can run in Vision-Language-Action (VLA) or Vision-Action (VA) modes. Joint optimization leads to mutually reinforcing capabilities.

Performance metrics:

Task/Metric	Baseline	DrivePI (0.5B)	Improvement
3D Occ. RayIoU	FB-Occ: 39.0%	49.3%	+10.3% absolute
3D Occ. mAVE	FB-Occ: 0.591	0.509	–0.082
Planning L2 (no ego)	VAD: 0.72m, Coll: 0.22%	0.49m, Coll: 0.38%	–32% L2 error
nuScenes-QA Acc	OpenDriveVLA-7B: 58.2%	60.7%	+2.5 pp
Coll. rate (ego)	ORION: 0.37%	0.11%	–70%

Training occurs in two phases: spatial projector stage (encoder/LLM frozen), then end-to-end multimodal+task learning. Ablations confirm unified (multi-task) training improves both perception and QA beyond modality-specific baselines.

7. Limitations and Prospects

Limitations of DrivePI include the use of fixed loss weights ($\xi(t)$6), absence of reinforcement learning for planning, and questions around model scaling beyond 0.5B parameters or low-latency deployment. Improved multi-task weighting or RL-based fine-tuning are suggested for future enhancement.

A plausible implication is the extension of unified, spatially aware MLLM frameworks to broader embodied reasoning tasks with increasingly scarce or ambiguous data, in both perception and control domains.

---

References:

• (Park et al., 2023) "Distributionally Robust Path Integral Control"
• (Liu et al., 14 Dec 2025) "DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planning"