CoRAL: Contact-Rich Adaptive LLM-based Control for Robotic Manipulation

Abstract: While LLMs and Vision-LLMs (VLMs) demonstrate remarkable capabilities in high-level reasoning and semantic understanding, applying them directly to contact-rich manipulation remains a challenge due to their lack of explicit physical grounding and inability to perform adaptive control. To bridge this gap, we propose CoRAL (Contact-Rich Adaptive LLM-based control), a modular framework that enables zero-shot planning by decoupling high-level reasoning from low-level control. Unlike black-box policies, CoRAL uses LLMs not as direct controllers, but as cost designers that synthesize context-aware objective functions for a sampling-based motion planner (MPPI). To address the ambiguity of physical parameters in visual data, we introduce a neuro-symbolic adaptation loop: a VLM provides semantic priors for environmental dynamics, such as mass and friction estimates, which are then explicitly refined in real time via online system identification, while the LLM iteratively modulates the cost-function structure to correct strategic errors based on interaction feedback. Furthermore, a retrieval-based memory unit allows the system to reuse successful strategies across recurrent tasks. This hierarchical architecture ensures real-time control stability by decoupling high-level semantic reasoning from reactive execution, effectively bridging the gap between slow LLM inference and dynamic contact requirements. We validate CoRAL on both simulation and real-world hardware across challenging and novel tasks, such as flipping objects against walls by leveraging extrinsic contacts. Experiments demonstrate that CoRAL outperforms state-of-the-art VLA and foundation-model-based planner baselines by boosting success rates over 50% on average in unseen contact-rich scenarios, effectively handling sim-to-real gaps through its adaptive physical understanding.

• The paper introduces CoRAL, a modular neuro-symbolic framework in which an LLM designs adaptive costs and contact strategies for a fast MPPI controller, enabling zero-shot manipulation without demonstrations.
• CoRAL combines dedicated 6-DoF pose tracking, semantic physics priors, memory retrieval, and online correction, achieving up to 9/10 success on simulated contact-rich tasks and 6/10 to 10/10 across tested real-world tasks.
• The results show that online refinement and symbolic contact-region guidance are essential: refinement removal sharply reduces success, while guided contact strategies cut T6 control steps by 83.9% and end-effector path length by 63.9%.

CoRAL (Contact-Rich Adaptive LLM-based Control) is a modular, neuro-symbolic framework for zero-shot contact- and force-critical robotic manipulation (2605.02600). Rather than training an end-to-end policy or using a LLM as a direct controller, the framework positions the LLM as a cost designer: it synthesizes context-aware objective functions and symbolic contact strategies for a high-frequency Model Predictive Path Integral (MPPI) controller. The central claim is that decoupling slow semantic reasoning from fast reactive control—while closing the loop between them through online adaptation—yields robust manipulation without any demonstration data.

Architecture

CoRAL is organized around three tiers of control with distinct time constants: joint impedance control at 1 kHz for safety and compliance, MPPI trajectory optimization at 10 Hz, and asynchronous LLM reasoning at roughly 1 Hz. This hierarchy addresses the latency mismatch between foundation-model inference and the dynamic requirements of contact interaction.

The perception pipeline uses FoundationPose to track 6-DoF object poses from RGB-D input and known 3D models, while a vision-LLM (VLM) supplies semantic physics priors—estimates of mass and friction derived from appearance and task context. These priors are explicitly treated as beliefs subject to correction rather than ground truth, which is the key design decision enabling robustness to VLM hallucination.

The LLM operates in two roles. In Task Formulation, it generates an initial plan tuple $(J_0, C_0)$: an executable Python cost function whose structure and weights are freely composed from available state variables (not restricted to a fixed term set), plus a contact strategy consisting of a nominal reference point $c_{\mathrm{ref}}$ and expansion factor $r$ that define an ellipsoidal region on the object surface. Candidate contact points sampled uniformly from this manifold act as soft cost attractors in the MPPI objective, biasing sampling toward semantically plausible contact regions without hard-constraining the optimizer. A retrieval-augmented Memory Unit stores successful episodes indexed by task definition and environmental parameters; when a similar task recurs, the stored plan bypasses the initial LLM call entirely.

In Online Adaptation (the outer loop), triggered after $N_{\mathrm{retry}}$ persistent inner-loop failures, the LLM receives logged episode history and performs two functions: world model correction (inferring revised mass/friction values from observed action–outcome discrepancies, effectively acting as a system identification agent) and strategy refinement (rewriting the failing cost function with a natural-language diagnosis). The implementation uses GPT-4o for both roles, with MPPI rolling out $K=256$ trajectories over a horizon of $H=32$ steps in a MuJoCo-based planning world, and an effective-sample-size criterion used to adaptively select the temperature $\lambda$ at each planning iteration.

Experimental results

Evaluation covers six tasks on a simulated Franka Emika Panda (robosuite/MuJoCo): push-and-pick cutting board (T1), pick box (T2), pick-and-place in clutter (T3), push with constant force (T4), flip box (T5), and flip against a wall leveraging extrinsic contacts (T6). Each task is run 10 times with randomized poses, masses, friction coefficients, and dimensions.

Against end-to-end VLAs (OpenVLA-OFT, $\pi_{0.5}$) and the cost-generation baseline L2R, CoRAL's advantage concentrates precisely where physical reasoning matters. All baselines succeed on T2–T3, but OpenVLA-OFT and $\pi_{0.5}$ score 0/10 on T1, T4, and T6; L2R manages only 1/10 on T6 because its static code generation lacks contact strategy formulation and refinement. CoRAL achieves 7/10 on T6 and 9/10 on T4 and T5. Relative to these baselines, the reported average success-rate improvement exceeds 50% on unseen contact-rich scenarios.

Two human expert baselines calibrate this performance. The single-stage expert design matches or exceeds CoRAL on simpler tasks, while the finite-state-machine expert variant—with phase-specific costs tuned offline—represents an upper bound that CoRAL does not surpass (e.g., 9/10 vs. 7/10 on T6). The honest reading is that CoRAL recovers much of expert-designed cost structure automatically but has not closed the gap to carefully engineered multi-phase objectives.

The ablations are decisive:

Variant	T1	T3	T4	T5	T6
CoRAL (full)	5/10	10/10	9/10	9/10	7/10
w/o Memory	2/10	9/10	9/10	7/10	5/10
w/o Refinement	0/10	3/10	6/10	4/10	2/10
Unified VLM	0/10	0/10	1/10	0/10	0/10
w/o Pose Tracking	0/10	0/10	0/10	0/10	0/10

Three findings stand out. First, removing pose tracking and relying on the VLM for 6-DoF estimation produces total failure across all tasks—the paper calls this conclusive evidence that a dedicated geometric estimator is essential, since VLMs produce physically impossible "hallucinated" poses under dynamic interaction. Second, unifying perception and planning in a single multimodal prompt collapses performance nearly to zero, supporting the paper's hypothesis that role separation between VLM and LLM is critical. Third, disabling online refinement eliminates success on the long-horizon T1 task (5/10 → 0/10), mirroring the static L2R baseline; the failure mode is instructive—slightly wrong friction priors cause board slippage during the pick, which only the outer-loop parameter correction can diagnose and fix.

A targeted ablation on T6 isolates the contribution of the symbolic contact strategy $C_0$. With LLM-guided regions, execution required 32 control steps versus 199 unguided (83.9% fewer) and a 1.33 m versus 3.69 m end-effector path (63.9% shorter). This quantifies how semantic pruning converts a computationally intractable long-horizon contact search into a tractable one—an implication being that MPPI alone cannot discover extrinsic-contact strategies by sampling, even with a correct cost function.

Robustness experiments deliberately corrupt the planning world with severe parameter errors (mass 2.0 kg vs. 0.25 kg ground truth; friction 0.9 vs. 0.5). The outer loop converges the belief parameters close to true values over several adaptation cycles, in both simulation and hardware. The paper also demonstrates explainable failure recovery: the adaptation module emits natural-language diagnoses identifying misweighted cost terms and enumerates its corrections before re-entering the inner loop.

Real-world validation

Zero-shot deployment on a physical Franka Emika Panda (with motion capture replacing FoundationPose) yields 10/10 on T2 and T3, 9/10 on T4, 7/10 on T5, 6/10 on T6, and 4/10 on T1, with execution times between roughly 9 s and 25 s. Real-world friction variation on T1 was handled by the online adaptation loop diagnosing slippage mid-trial. Force regulation on T4 shows measured end-effector force held within the target band (~5 N) despite sensor noise, confirming that the architecture closes the loop on force dynamics rather than merely planning geometry. Notably, real-world T1 performance (4/10) falls below simulation (5/10), indicating residual sim-to-real sensitivity even with adaptation.

Limitations and open questions

The authors are explicit about two dependencies. First, the entire framework rests on the fidelity of the internal simulated planning world; a sufficiently poor initialization—from pose-tracking errors or grossly incorrect VLM material judgments—can prevent the planner from converging, and the feedback loop is not guaranteed to recover from arbitrary initialization error. Second, strategy quality depends on a generalist off-the-shelf LLM (GPT-4o) that is not specialized for control-oriented code generation; viability of generated costs is not guaranteed for tasks far outside pre-training coverage. Latency constraints from foundation-model inference bound the outer-loop rate. Open questions include whether learned residual dynamics models can capture non-rigid effects and complex friction within the planning world, and whether fine-tuned LLMs for objective synthesis would improve robustness on abstract novel tasks.

Conclusion

CoRAL demonstrates that treating LLMs as adaptive cost designers for a reactive optimal controller, supported by separated perception roles, online system identification via language-model reasoning, and experience retrieval, achieves competitive zero-shot contact-rich manipulation without demonstrations. Its strongest evidence is the ablation structure: every component—pose tracking, role separation, refinement, memory—is individually necessary, and their removal produces large or catastrophic degradation. The approach trades the data bottleneck of imitation learning for a dependency on world-model fidelity and generalist LLM competence, and its remaining gap to expert-designed FSM costs marks the boundary of what current automatic cost synthesis achieves.