Grounded Re-Planning Mechanism

• Grounded re-planning mechanism is a planning approach that links high-level symbolic plans to real-world perceptual data for dynamic adaptation.
• It employs methodologies such as symbolic-to-continuous controllers, multimodal grounding pipelines, and behavior trees to continuously revise plans based on real-time feedback.
• This approach is crucial for applications in mobile manipulation, multi-robot coordination, and robust task execution in uncertain or evolving environments.

A grounded re-planning mechanism refers to a planning approach in robotics and autonomous agents where high-level symbolic plans are incrementally and dynamically connected (“grounded”) to real-world perception, action, and environmental feedback. This mechanism enables systems to adapt to unexpected events, perceptual ambiguity, and execution failures by closing the loop between abstract decision-making and the continuously sensed, uncertain, or evolving environment. Grounded re-planning contrasts with classic open-loop planning by systematically integrating feedback at multiple levels: state estimation, action feasibility, sensory grounding, and physical interaction.

1. Fundamental Principles of Grounded Re-Planning

Grounded re-planning combines two core requirements: (1) explicit linkage of symbolic plans to environment state and sensory perception, and (2) dynamic plan adaptation in response to changes or execution failures.

• Grounding involves mapping between symbolic abstractions (e.g., logical state representations, PDDL predicates, language instructions) and observed states (e.g., sensor data, scene graphs, visual input, object detections).
• Re-planning denotes the process where the plan is incrementally revised (partially or fully) when execution predictions are contradicted by actual outcomes (e.g., a perceived object is missing, an action fails due to changed geometry, the environment diverges from the planner’s model).

This process may be instantiated at various abstraction levels:

• Symbolic: grounding PDDL states using perception modules (Herzog et al., 9 Apr 2025), updating symbolic belief state from sensory input (Lamanna et al., 2021), mapping parsed images to abstract states (Liberman et al., 2022).
• Language-based: grounding high-level plans or goals using tables or 3D scene graphs integrated with multimodal perception (Lin et al., 2022, Strader et al., 9 Jun 2025).
• Physical: planning open-loop actuator commands robust to ground uncertainty by simultaneous optimization over disturbance scenarios (Green et al., 2020), or aligning simulation outcomes with expected effects to validate/refine hierarchical plans (Kienle et al., 15 May 2025).

2. Architectures and Methodological Implementations

A diverse suite of architectures has been proposed for implementing grounded re-planning mechanisms:

1. Symbolic-to-Continuous Hybrid Controllers Systems combine discrete, symbolic planners (e.g., LTL-based automata (Vasilopoulos et al., 2020), domain-specific PDDL planners (Herzog et al., 9 Apr 2025)) with continuous, reactive controllers. The symbolic layer outputs abstract actions, which are grounded online to continuous motion via interface modules using real-time sensor feedback. If grounding fails (e.g., obstacles, unreachable space), the controller invokes a “fix mode” that may insert new actions such as object disassembly or pushing.
2. Multimodal and Vision-Language Grounding Pipelines Pipelines integrate LLMs or vision-LLMs to parse natural language goals, scene images, or scene graphs, producing grounded state representations. These representations are conditioned using domain-specific constraints and object detection (open-vocabulary or foundation models), forming structured scene graphs or object lists that are directly mapped to planner-compatible symbols (Herzog et al., 9 Apr 2025, Strader et al., 9 Jun 2025, Elhafsi et al., 20 May 2025). The system regularly updates the mapping as the environment evolves, supporting closed-loop plan refinement.
3. Behavioral Trees with Execution Feedback Task graphs produced by LLMs are converted into behavior trees (BT). Each node, linked to a specific semantic tag and low-level action/perceptual skill, continuously receives status feedback (success/failure/running) during execution. When a BT node fails (e.g., object not grasped), associated recovery branches or corrective subplans are triggered (Wang et al., 15 Aug 2024).
4. Domain-Independent Heuristics and GNN-based Guidance Graph neural network (GNN) heuristics trained on abstract or domain-independent representations (lifted STRIPS or FDR graphs) provide rapid estimates for successor states, enabling efficient plan revision in response to changing world models, without full instantiation (Chen et al., 2023).
5. Sample-Efficient LLM and Few-Shot Grounded Planners Plans are constructed by prompting an LLM with in-context examples retrieved based on language and scene similarity; re-planning occurs by re-invoking the LLM using updated object lists or environmental feedback, while faulty subgoals are revised via similarity matching against the detected visual objects (Song et al., 2022, Kim et al., 23 Dec 2024).

3. Technical Realizations: Algorithms, Models, and Mathematical Formulations

Grounded re-planning systems leverage a range of technical approaches:

• Automata and Metric-guided Symbolic Planning:

Temporal logic specifications are translated to automata (e.g., NBA), and discrete progress towards goal states is tracked by distance metrics (LaTeX: $d_F(q,V_F)$), which guide selection of next candidate actions (Vasilopoulos et al., 2020).

• Predicate-based Verification and Hierarchical Error Reasoning:

Prior to executing each action, predicate-based feasibility checks are performed using predicates $P_n$ derived by LLMs or domain authoring. For state $s_t$, execution proceeds only if $F(s_t, P_n)=1$. If any $p \in P_n$ is unsatisfied, the set $U_n$ is identified and used to re-prompt the planner for a corrective action (Rivera et al., 8 Oct 2024, Kienle et al., 15 May 2025).

• Multi-Modal Similarity Search:

Combined language ($S_l$) and environmental ($S_e$) similarity scores are computed as

$$S_m = w_l \cdot \frac{S_l}{\sum_{i=1}^N s_{l,i}} + w_e \cdot \frac{S_e}{\sum_{i=1}^N s_{e,i}}$$

and used to retrieve the most appropriate in-context examples for LLM-based plan generation (Kim et al., 23 Dec 2024).

• Action Effect Validation in Simulation:

Each (hierarchical) planned action is simulated prior to execution; effects $E_{\text{sim}}$ are compared to those specified in the domain model. Disagreements are reported to an error reasoner, which diagnoses the source (model, plan, or skill mapping) and triggers selective re-planning (Kienle et al., 15 May 2025).

• Physical Robustness via Disturbance-aware Trajectory Optimization:

Simultaneous optimization is performed across a set of disturbance scenarios (e.g., variations in ground height), with input links ensuring that all scenario cases share identical actuator trajectories. The constraint

$g(u_k, t_k, U, T) = u_k - LI(T, U, t_k) = 0$

synchronizes control inputs across phases (Green et al., 2020).

4. Empirical Benchmarks and Performance Considerations

Common evaluation frameworks and metrics for grounded re-planning mechanisms include:

• Success and Goal-Condition Rates:

Success Rate (SR), Goal Condition Success Rate (GC), and path-length weighted counterparts (PLWSR, PLWGC) measure successful plan completion under environment changes (Kim et al., 23 Dec 2024).

• State Estimation Accuracy:

Precision and recall of subject–relation–object triplet extraction from scene graphs quantify grounding fidelity (Herzog et al., 9 Apr 2025).

• Plan Efficiency and Robustness:

Computation times, task completion numbers, and robustness under disturbance or dynamic changes are assessed, e.g., apex state errors in robust bipedal running under disturbance (Green et al., 2020), reallocation efficiency for Earth observation resources under unreliability (Liu et al., 2020).

Several approaches demonstrate marked improvements against baseline (ungrounded or non-replanning) models:

• LLM-planners with dynamic grounding achieved significantly higher few-shot task completion rates versus LLM baselines (Song et al., 2022).
• Domain-conditioned scene graph pipelines yield higher planning success and triplet accuracy compared to unstructured LMM-based planners (Herzog et al., 9 Apr 2025).
• GNN-learned heuristics in dynamic re-planning surpass earlier domain-independent methods in both coverage and solution quality (Chen et al., 2023).
• Hierarchical, execution-grounded re-planning dramatically reduces compounding errors in long-horizon tasks (Kienle et al., 15 May 2025).

5. Applications, Limitations, and Open Directions

Grounded re-planning mechanisms are applicable to:

• Mobile manipulation in unknown/cluttered environments (Vasilopoulos et al., 2020, Lamanna et al., 2021)
• Multi-robot coordination in large-scale, outdoor settings leveraging fused 3D scene graphs (Strader et al., 9 Jun 2025)
• Task-and-motion planning under semantic, symbolic, or natural language goals (Lin et al., 2022, Song et al., 2022, Kim et al., 23 Dec 2024)
• Robust motion execution under physical parameter uncertainty (bipedal locomotion, manipulation) (Green et al., 2020, Elhafsi et al., 20 May 2025)
• Rapid reallocation for large-scale, heterogeneous resource networks in dynamic environments (observation satellites, UAVs) (Liu et al., 2020)

Notable limitations include:

• Scene graph quality bottlenecked by detection/classification errors, especially for objects not in training data or under heavy occlusion (Herzog et al., 9 Apr 2025).
• Simulation-to-real gap in physically grounded pipelines due to incomplete scene reconstruction or uncertain materials (Elhafsi et al., 20 May 2025).
• Computational overhead in continuous update cycles (e.g., Gaussian splatting, planning in large state spaces).

Potential open research avenues:

• Integrating generative models for 3D scene completion in occluded environments (Elhafsi et al., 20 May 2025).
• Scalable closed-loop feedback for physically grounded action in dynamic or partially known worlds (Robbins et al., 17 Apr 2025).
• Autonomous reduction of annotation costs for few-shot policy grounding (Kim et al., 23 Dec 2024).
• Further modularization of behavioral libraries for multi-modal, hierarchical task abstraction with human-interpretable interfaces (Wang et al., 15 Aug 2024).
• Extending fusion and scene graph approaches to fully online, incrementally flexible representations to support perpetual re-planning in non-static environments (Strader et al., 9 Jun 2025).

6. Representative Mathematical Structures

Mechanism	Key Formula/Algorithm	Source Paper(s)
Predicate-based verification	$$F(s_t, P_n) = \{1, \text{if all } p \in P_n \text{ hold in } s_t; 0, \text{otherwise}\}$$	(Rivera et al., 8 Oct 2024)
Multi-modal similarity	$$S_m = w_l \cdot \frac{S_l}{\Sigma s_{l,i}} + w_e \cdot \frac{S_e}{\Sigma s_{e,i}}$$	(Kim et al., 23 Dec 2024)
Input linking (robust opt.)	$g(u_k, t_k, U, T) = u_k - LI(T, U, t_k) = 0$	(Green et al., 2020)
Scene graph to PDDL state	$$S_{init} = \{ p(v_i, v_j) \mid (v_i, p, v_j) \in E \}$$	(Herzog et al., 9 Apr 2025)

These core mathematical and algorithmic elements underlie the current state-of-the-art in grounded re-planning across symbolic, language-based, and physical planning domains.

7. Historical Context and Outlook

The evolution of grounded re-planning mechanisms reflects a progression from hand-designed, static planners to systems capable of robust, adaptive behavior in real time. Early approaches were limited by discretized, precomputed primitive sets and limited environment models (King et al., 2016). The integration of reactive feedback, symbolic abstraction, LLMs, vision-language grounding, and robust optimization now enables adaptive re-planning for increasingly complex embodied tasks.

Continuing advances are expected to focus on: improving the fidelity and efficiency of environment-to-symbolic mapping; leveraging large, pre-trained language and perception models for rapid generalization; and tightly integrating simulation, execution, and error diagnosis to achieve reliable autonomy amidst real-world variability.