Semantic Planning: From Graphs to Tokens

• Semantic planning is the practice of using semantically rich abstractions, such as semantic graphs and latent embeddings, instead of raw sensor data for decision-making.
• It bridges high-level reasoning with low-level execution by decomposing tasks into semantic subgoals (e.g., room types, target tokens) across domains like robotics and language processing.
• Empirical results demonstrate that integrating uncertainty modeling and structured semantic guarantees improves both success rates and planning efficiency.

Across recent work, semantic planning denotes planning procedures in which the state abstraction, decision variable, or intermediate target is expressed in semantic terms rather than only as geometry, raw identifiers, or surface tokens. In robotics, this can mean rooms, doorways, objects, zones, regions, or semantic-probabilistic topological graphs; in conversation systems, dense semantic conversation states; in language modeling, explicit planning tokens that predict latent semantic representations of the response; in recommender systems, a semantic target for the next exposure; in video generation, target ViT embeddings predicted before rendering; and in epistemic planning, semantic constraints on knowledge that bound the planning space (Zhao et al., 25 Sep 2025, Kremer et al., 2023, Chen et al., 14 Mar 2025, Jin et al., 10 Jul 2026, Yin et al., 2024, Team et al., 21 May 2026, Burigana et al., 2023).

1. Conceptual scope

The term does not name a single formalism. In "SLAM-Free Visual Navigation with Hierarchical Vision-Language Perception and Coarse-to-Fine Semantic Topological Planning" (Zhao et al., 25 Sep 2025), semantic planning means building a semantic topological graph and planning over it semantically, probabilistically, and topologically rather than optimizing over a dense metric map. In "S-Nav: Semantic-Geometric Planning for Mobile Robots" (Kremer et al., 2023), it means using what the environment is—rooms, doorways, and their relations—to guide path planning. In "Semantic Task Planning for Service Robots in Open World" (Cui et al., 2020), it means planning driven by the meaning of a natural-language request and by the semantic structure of the task. In "Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation" (Wu et al., 30 Jun 2026), it means planning over objects, zones, and regions with a belief over where the target is likely to be.

Outside robotics, the same phrase is used for analogous abstractions. In "Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Space" (Chen et al., 14 Mar 2025), planning is performed in a learned continuous semantic space of conversations. In "Semformer: Transformer LLMs with Semantic Planning" (Yin et al., 2024), the model first builds a latent semantic representation of what it is going to say. In "From Raw IDs to Semantic Planning: How Recommender Systems Utilize Information at Scale" (Jin et al., 10 Jul 2026), the system first predicts the semantic target of the next exposure and only then instantiates that target as an item, offer, or generated creative. In "Bernini: Latent Semantic Planning for Video Diffusion" (Team et al., 21 May 2026), an MLLM predicts the target video in ViT embedding space and a DiT-based renderer synthesizes pixels from that plan. In "A Semantic Approach to Decidability in Epistemic Planning (Extended Version)" (Burigana et al., 2023), semantic planning refers to imposing semantic constraints on agents’ knowledge rather than syntactic restrictions on actions.

A unifying characterization is therefore plausible: semantic planning shifts the primary planning substrate from low-level state variables to semantically meaningful entities, relations, or latent targets, while leaving geometry, control, retrieval, or rendering to downstream modules.

2. Representational substrates

A dominant substrate is the semantic graph. The SLAM-free legged-navigation framework represents the environment as a graph $G=(V,E)$ whose nodes store a coarse 3D anchor $\mathbf{P}_{v_i}$, a semantic label $L_{v_i}$, a fused confidence $C_{f_{v_i}}$, and an exploration probability $P_{\text{explore}(v_i)}$, while edges carry traversal costs $c_{ij}$ (Zhao et al., 25 Sep 2025). S-Nav uses an undirected topological graph of rooms and doorways derived from S-Graphs, with doorway–room edge cost

$$c_{dr} = \left\lVert {}^{M}\mathbf{p}_i - {}^{M}\mathbf{d}_i \right\rVert^2 + p_d,$$

so that semantics enter directly through room and doorway entities and through the doorway penalty (Kremer et al., 2023). IntelliMove similarly organizes rooms and objects into a semantic graph whose nodes are semantic entities and whose edges encode room connectivity and object–room membership (Ngom et al., 2024). The HSG framework introduces a multi-granular graph $HSG_t=(V_t,E_t)$ with $V_t=O_t\cup Z_t\cup R_t$, where $O_t$, $\mathbf{P}_{v_i}$0, and $\mathbf{P}_{v_i}$1 are object, zone, and region nodes, respectively (Wu et al., 30 Jun 2026). LEAPS uses a latent semantic graph over room types, with binary variables $\mathbf{P}_{v_i}$2 denoting whether room types are directly reachable by short local exploration in a given house (Wu et al., 2018).

Another substrate is the latent semantic state. SCOPE maps a conversation state $\mathbf{P}_{v_i}$3 to a semantic embedding $\mathbf{P}_{v_i}$4, represents an action as $\mathbf{P}_{v_i}$5, learns transition and reward models in that space, and then performs MCTS without additional LLM queries for simulation (Chen et al., 14 Mar 2025). Semformer inserts planning tokens into a decoder-only Transformer and trains them to predict latent semantic codes $\mathbf{P}_{v_i}$6 produced by an autoencoder for the response, using

$\mathbf{P}_{v_i}$7

Those planning-token representations are intended to encode the latent semantic representation of the forthcoming response (Yin et al., 2024). Bernini uses the MLLM’s own ViT embedding space as the planning space: target visual tokens are masked, predicted iteratively, and then supplied to the renderer as a semantic plan (Team et al., 21 May 2026).

Discrete semantic codes also appear as planning vocabularies. The recommender-system formulation distinguishes raw IDs from semantic IDs and proposes a semantic target $\mathbf{P}_{v_i}$8 as the explicit object of planning, with

$\mathbf{P}_{v_i}$9

Here the semantic target can encode intent, desired attributes, or multi-stakeholder objectives before the system instantiates an item, offer, or creative (Jin et al., 10 Jul 2026).

Uncertainty-bearing semantic maps constitute a further substrate. DeepSemanticHPPC stores each point $L_{v_i}$0 in a semantic point cloud as $L_{v_i}$1, where $L_{v_i}$2 is a class-probability vector and $L_{v_i}$3 is per-class uncertainty (Han et al., 2020). Safe Planning in Unknown Environments using Conformalized Semantic Maps maintains a semantic grid map with per-cell PMFs $L_{v_i}$4 and converts them to prediction sets $L_{v_i}$5 by conformal prediction (Sundarsingh et al., 29 Sep 2025). Perception-Based Temporal Logic Planning in Uncertain Semantic Maps models each landmark $L_{v_i}$6 with Gaussian positional uncertainty and a discrete class distribution $L_{v_i}$7 (Kantaros et al., 2020).

3. Planning architectures and objectives

A recurring architecture is hierarchical decomposition. In the SLAM-free visual-navigation framework, coarse global planning selects a semantic subgoal on the topological graph, while local planning uses Viplanner for obstacle avoidance; the global choice is

$L_{v_i}$8

The hierarchy is then completed by RL locomotion controllers that map high-level velocity commands to joint-level actions (Zhao et al., 25 Sep 2025). S-Nav uses a high-level semantic search over rooms and doorways, then a Subproblem Solver to decompose the route into room-bounded geometric subproblems solved by IRRT* or related OMPL planners (Kremer et al., 2023). IntelliMove exposes the same pattern as Targeted Navigation Mode, Multi-target Exploration Mode, and Discovery Mode, all implemented over a semantic graph with Dijkstra plus an LLM-based room guess when no matching target is present in the map (Ngom et al., 2024). The HSG method plans first at the region level and then at the zone level, using macro-actions $L_{v_i}$9 and $C_{f_{v_i}}$0 and finite-horizon rollouts on an HSG-based simulator to estimate expected return (Wu et al., 30 Jun 2026). LEAPS likewise separates high-level semantic planning from low-level execution: a Bayesian semantic model proposes the next room-type sub-target, and a model-free visual sub-policy executes that sub-target (Wu et al., 2018).

A second pattern is explicit two-stage decision making. In recommender systems, semantic retrieval maps directly to an item or semantic ID, whereas semantic planning inserts an intermediate semantic target $C_{f_{v_i}}$1 that specifies what the next exposure should accomplish before instantiation (Jin et al., 10 Jul 2026). PSALM applies the same decomposition to symbolic planning: the LLM infers action semantics $C_{f_{v_i}}$2 from execution outcomes, maintains a belief over candidate semantic statements, and a classical planner then plans over the induced PDDL model (Zhu et al., 2024). In video generation, Bernini assigns semantic reasoning to an MLLM planner and pixel generation to a DiT-based renderer, with semantics as the interface between the two (Team et al., 21 May 2026).

A third pattern is planning directly in a learned semantic dynamical system. SCOPE formulates conversation as an MDP and replaces text-state rollouts with rollouts in latent semantic space using learned $C_{f_{v_i}}$3 and $C_{f_{v_i}}$4, allowing MCTS to run entirely in that space (Chen et al., 14 Mar 2025). Semformer does not perform tree search, but it introduces an internal planning phase in which planning tokens must summarize the forthcoming response before next-token prediction, thereby modifying the LLM’s computational structure (Yin et al., 2024).

This architecture-level diversity clarifies a common misconception: semantic planning is not merely “planning with labels.” In the literature it can refer to graph search over semantically typed nodes, latent-space planning over dense embeddings, target-first decomposition, or logical planning over learned action semantics.

4. Uncertainty, belief, and formal guarantees

Many semantic planners explicitly model uncertainty. In the SLAM-free navigation graph, node exploration probability is updated by a Bayesian rule,

$C_{f_{v_i}}$5

so that semantically exhausted nodes decay toward zero exploration value (Zhao et al., 25 Sep 2025). LEAPS keeps a Bayesian belief over room-type connectivity and updates $C_{f_{v_i}}$6 from local exploration observations, then plans on the posterior graph (Wu et al., 2018). DeepSemanticHPPC goes further by carrying per-point semantic probabilities and uncertainties, classifying points as safe, unsafe, or unclear, clustering unclear regions, and maintaining multiple path hypotheses through different uncertain terrain regions (Han et al., 2020). The HSG planner fuses semantic priors with frontier-based exploration evidence into a belief over regions and zones, then runs POUCT on that belief (Wu et al., 30 Jun 2026).

Some frameworks attach explicit correctness guarantees to semantic reasoning. Safe Planning in Unknown Environments using Conformalized Semantic Maps defines per-cell prediction sets

$C_{f_{v_i}}$7

and proves a path-agnostic coverage guarantee

$C_{f_{v_i}}$8

which in turn yields user-specified mission completion rates for semantic reach-avoid tasks (Sundarsingh et al., 29 Sep 2025). Perception-Based Temporal Logic Planning in Uncertain Semantic Maps embeds probabilistic semantic conditions directly in atomic predicates, such as

$C_{f_{v_i}}$9

and then lifts them into co-safe LTL specifications over a hybrid state space consisting of robot state, semantic-map belief, and automaton state (Kantaros et al., 2020). In epistemic planning, decidability is obtained by augmenting S5$P_{\text{explore}(v_i)}$0 with the commutativity axiom

$P_{\text{explore}(v_i)}$1

which bounds higher-order knowledge, yields a finitary characterization of common knowledge, and makes the resulting planning problem decidable (Burigana et al., 2023).

These results also show that semantic planning often trades raw expressivity for structured conservatism. Conformalized maps choose the most conservative class in a prediction set, and epistemic commutativity rules out unbounded knowledge nesting. A plausible implication is that guarantees in semantic planning usually depend on carefully chosen abstractions, calibration assumptions, or admissible semantic dynamics.

5. Representative realizations and empirical profile

Robotic navigation provides the densest empirical record. The SLAM-free legged-navigation framework reports Semantic Accuracy up to $P_{\text{explore}(v_i)}$2, Obstacle Avoidance Success Rate $P_{\text{explore}(v_i)}$3, simulation Success Rate $P_{\text{explore}(v_i)}$4 with SPL up to $P_{\text{explore}(v_i)}$5, and real-world Success Rate $P_{\text{explore}(v_i)}$6 with SPL up to $P_{\text{explore}(v_i)}$7; Table 2 further shows that GPT-4-based node scoring improves Global Node Selection Accuracy by $P_{\text{explore}(v_i)}$8 over probability-only ranking, and adding cost-aware optimization adds another $P_{\text{explore}(v_i)}$9 on average (Zhao et al., 25 Sep 2025). S-Nav evaluates three configurations on a synthetic indoor map of $c_{ij}$0 with 8 rooms and 10 doorways over 1000 planning queries with a 0.1 s timeout, and reports that IRRT* + S-Graphs + SPS yields the shortest paths and more samples within the same time budget (Kremer et al., 2023). IntelliMove reports average planning time 7 ms, maximum 10 ms, success rate 99% for known targets, and 55% for goal discovery scenarios in a simulated office environment (Ngom et al., 2024). DeepSemanticHPPC shows that semantics and active disambiguation matter in unstructured terrain: on Mann Library, B1 gives 0% safe and 100% unsafe, whereas DeepSemanticHPPC with 5 NBVs gives 86.0% safe and 3.8% unsafe, with CS = 28.0% and CN = 10.2%; on Cass Park, 5 NBVs gives 59.2% safe and 36.6% unsafe (Han et al., 2020).

Language-centered systems display the same pattern in nonphysical state spaces. SCOPE performs conversation planning 70 times faster than conventional simulation-based planning algorithms, reports average rollout time $c_{ij}$1 s versus $c_{ij}$2 s for vanilla MCTS, and yields conversations with about 150 more human tokens over 5 turns on average for the length-based reward (Chen et al., 14 Mar 2025). PSALM learns action semantics from one goal, boosting plan success rate from 36.4% to 100% across 7 environments while improving exploration efficiency for recovering ground-truth domain action semantics (Zhu et al., 2024). Semformer shows near-perfect performance on graph path-finding and improves downstream language metrics, including Wikitext perplexity 35.6 versus 37.5 for the teacher-forced baseline and LAMBADA perplexity 38.8 versus 42.5 (Yin et al., 2024).

Generation and communication systems instantiate semantic planning in yet other ways. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks by separating MLLM-based semantic planning from diffusion-based rendering (Team et al., 21 May 2026). Adaptive Semantic Communication for UAV/UGV Cooperative Path Planning defines semantics in terms of traversability and shortest-path stability, and reports that Path-SC reduces average data volume to 563.2 kb, compared with 988.3 kb for DeiT-SC and 1216.3 kb for JPEG, while maintaining path-planning accuracy under AWGN and Rayleigh fading (Zhao et al., 8 Oct 2025). The diversity of these empirical profiles indicates that semantic planning is not tied to a particular domain so much as to a particular choice of abstraction: semantically meaningful state variables become the object of planning, and downstream mechanisms handle realization.

6. Limitations, misconceptions, and open directions

A first limitation is that semantic planning rarely eliminates geometry or low-level execution; it reorganizes the interface between high-level reasoning and low-level control. S-Nav still relies on room contours, ESDF construction, and a geometric planner inside a reduced free-space region; IntelliMove still depends on costmaps and local planning; HSG planning still grounds macro-actions into frontier tours and Fast Marching paths; Bernini still requires a renderer to convert semantic plans into VAE latents and pixels (Kremer et al., 2023, Ngom et al., 2024, Wu et al., 30 Jun 2026, Team et al., 21 May 2026). A persistent misconception is therefore that semantic planning replaces geometric planning; in the cited systems it more often constrains, biases, or stages it.

A second limitation is grounding. Recommender-system semantic targets must remain instantiable under current inventory and infrastructure, and the paper explicitly identifies target grounding and ID stability as open problems (Jin et al., 10 Jul 2026). Bernini’s semantic plan must align with what the renderer can actually synthesize, which is why it uses a modular but lightly co-trained interface (Team et al., 21 May 2026). PSALM assumes typed predicates, action headers, deterministic transitions, and informative error messages; without error messages, accuracy degrades sharply in more complex domains (Zhu et al., 2024). Semformer improves planning behavior but still struggles on substantially out-of-distribution graphs and very long paths, and it increases training complexity through the autoencoder and latent-prediction objective (Yin et al., 2024).

A third limitation concerns assumptions behind uncertainty handling and guarantees. Conformalized semantic maps guarantee mission completion rates only when test missions are drawn from the same distribution as the calibration scenarios; in out-of-distribution environments, mapping success drops even when mission success remains relatively high (Sundarsingh et al., 29 Sep 2025). Perception-based temporal-logic planning assumes known geometry, static semantic landmarks, and a linear-Gaussian structure that permits Riccati-based covariance prediction (Kantaros et al., 2020). The HSG framework evaluates in simulation with perfect pose, and explicitly notes latency from VLMs and LLMs as a challenge for real-time operation (Wu et al., 30 Jun 2026). DeepSemanticHPPC assumes predefined safe and unsafe semantic classes, static terrain, and a computationally heavy MC-dropout pipeline (Han et al., 2020).

Current directions suggest two broad trajectories. One is richer semantic structure: objects, zones, regions, room types, semantic IDs, semantic targets, planning tokens, and ViT-space plans all function as intermediate representations that could be made more compositional. The other is stronger coupling between semantics and guarantees: belief updates, conformal prediction, temporal logic, and semantic axioms show that semantics can be embedded not only in heuristics but also in formal planning machinery. Taken together, the literature indicates that semantic planning is becoming less a domain-specific label for “planning with meaning” and more a general design pattern: plan in a semantic substrate, then hand off execution to modules specialized for geometry, language realization, retrieval, or rendering.