---
title: 'Semantic Planning: From Graphs to Tokens'
url: https://www.emergentmind.com/topics/semantic-planning
type: topic
---

# Semantic Planning: From Graphs to Tokens

Across recent work, semantic planning denotes planning procedures in which the state abstraction, decision variable, or intermediate target is expressed in semantic terms rather than only as geometry, raw identifiers, or surface tokens. In robotics, this can mean rooms, doorways, objects, zones, regions, or semantic-probabilistic topological graphs; in conversation systems, dense semantic conversation states; in language modeling, explicit planning tokens that predict latent semantic representations of the response; in recommender systems, a semantic target for the next exposure; in video generation, target ViT embeddings predicted before rendering; and in epistemic planning, semantic constraints on knowledge that bound the planning space [2509.20739] [2307.01613] [2503.11586] [2607.09540] [2409.11143] [2605.22344] [2307.15485].

## 1. Conceptual scope

The term does not name a single formalism. In "SLAM-Free Visual Navigation with Hierarchical Vision-Language Perception and Coarse-to-Fine Semantic Topological Planning" [2509.20739], semantic planning means building a semantic topological graph and planning over it semantically, probabilistically, and topologically rather than optimizing over a dense metric map. In "S-Nav: Semantic-Geometric Planning for Mobile Robots" [2307.01613], it means using what the environment is—rooms, doorways, and their relations—to guide path planning. In "Semantic Task Planning for Service Robots in Open World" [2011.00621], it means planning driven by the meaning of a natural-language request and by the semantic structure of the task. In "Hierarchical 3D Scene Graph Construction and Belief-based Planning for Semantic Navigation" [2606.31071], it means planning over objects, zones, and regions with a belief over where the target is likely to be.

Outside robotics, the same phrase is used for analogous abstractions. In "Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Space" [2503.11586], planning is performed in a learned continuous semantic space of conversations. In "Semformer: Transformer Language Models with Semantic Planning" [2409.11143], the model first builds a latent semantic representation of what it is going to say. In "From Raw IDs to Semantic Planning: How Recommender Systems Utilize Information at Scale" [2607.09540], the system first predicts the semantic target of the next exposure and only then instantiates that target as an item, offer, or generated creative. In "Bernini: Latent Semantic Planning for Video Diffusion" [2605.22344], an MLLM predicts the target video in ViT embedding space and a DiT-based renderer synthesizes pixels from that plan. In "A Semantic Approach to Decidability in Epistemic Planning (Extended Version)" [2307.15485], semantic planning refers to imposing semantic constraints on agents’ knowledge rather than syntactic restrictions on actions.

A unifying characterization is therefore plausible: semantic planning shifts the primary planning substrate from low-level state variables to semantically meaningful entities, relations, or latent targets, while leaving geometry, control, retrieval, or rendering to downstream modules.

## 2. Representational substrates

A dominant substrate is the semantic graph. The SLAM-free legged-navigation framework represents the environment as a graph \(G=(V,E)\) whose nodes store a coarse 3D anchor \(\mathbf{P}_{v_i}\), a semantic label \(L_{v_i}\), a fused confidence \(C_{f_{v_i}}\), and an exploration probability \(P_{\text{explore}(v_i)}\), while edges carry traversal costs \(c_{ij}\) [2509.20739]. S-Nav uses an undirected topological graph of rooms and doorways derived from S-Graphs, with doorway–room edge cost
\[
c_{dr} = \left\lVert {}^{M}\mathbf{p}_i - {}^{M}\mathbf{d}_i \right\rVert^2 + p_d,
\]
so that semantics enter directly through room and doorway entities and through the doorway penalty [2307.01613]. IntelliMove similarly organizes rooms and objects into a semantic graph whose nodes are semantic entities and whose edges encode room connectivity and object–room membership [2410.14851]. The HSG framework introduces a multi-granular graph \(HSG_t=(V_t,E_t)\) with \(V_t=O_t\cup Z_t\cup R_t\), where \(O_t\), \(Z_t\), and \(R_t\) are object, zone, and region nodes, respectively [2606.31071]. LEAPS uses a latent semantic graph over room types, with binary variables \(z_{i,j}\) denoting whether room types are directly reachable by short local exploration in a given house [1809.10842].

Another substrate is the latent semantic state. SCOPE maps a conversation state \(s\) to a semantic embedding \(f(s)\in\mathbb{R}^n\), represents an action as \(\embeda = f(s+a)-f(s)\), learns transition and reward models in that space, and then performs MCTS without additional LLM queries for simulation [2503.11586]. Semformer inserts planning tokens into a decoder-only Transformer and trains them to predict latent semantic codes \(Z=(z_1,\dots,z_k)\) produced by an autoencoder for the response, using
\[
\mathcal{L} = \mathcal{L}_{\text{LM}} + \mathcal{L}_{\text{AE}} + \alpha \mathcal{L}_{\text{RP}}.
\]
Those planning-token representations are intended to encode the latent semantic representation of the forthcoming response [2409.11143]. Bernini uses the MLLM’s own ViT embedding space as the planning space: target visual tokens are masked, predicted iteratively, and then supplied to the renderer as a semantic plan [2605.22344].

Discrete semantic codes also appear as planning vocabularies. The recommender-system formulation distinguishes raw IDs from semantic IDs and proposes a semantic target \(z_t\) as the explicit object of planning, with
\[
f_{\text{plan}}: s_t \rightarrow z_t,\qquad g_{\text{inst}}: z_t \rightarrow e_t.
\]
Here the semantic target can encode intent, desired attributes, or multi-stakeholder objectives before the system instantiates an item, offer, or creative [2607.09540].

Uncertainty-bearing semantic maps constitute a further substrate. DeepSemanticHPPC stores each point \(\mathbf{e}^i\) in a semantic point cloud as \((\mathbf{m}^i,\mathbf{p}^i,\boldsymbol{\sigma}^i)\), where \(\mathbf{p}^i\) is a class-probability vector and \(\boldsymbol{\sigma}^i\) is per-class uncertainty [2003.03464]. Safe Planning in Unknown Environments using Conformalized Semantic Maps maintains a semantic grid map with per-cell PMFs \(p_t(m_j)\) and converts them to prediction sets \(\mathcal{C}_j(\mathbf{m}_t)\subseteq\mathcal{K}\) by conformal prediction [2509.25124]. Perception-Based Temporal Logic Planning in Uncertain Semantic Maps models each landmark \(\ell_i=\{\mathbf{x}_i,c_i\}\) with Gaussian positional uncertainty and a discrete class distribution \(d_i\) [2012.10490].

## 3. Planning architectures and objectives

A recurring architecture is hierarchical decomposition. In the SLAM-free visual-navigation framework, coarse global planning selects a semantic subgoal on the topological graph, while local planning uses Viplanner for obstacle avoidance; the global choice is
\[
v^* = \arg\max_{v_c \in \{V\}_{\text{child}} \Big( S^{\text{LLM}}_{v_c}\cdot C_{f_{v_c}}\cdot \exp\big(-\gamma \cdot d(\mathbf{P}_{\text{robot}}, \mathbf{P}_{v_c})\big)\Big).
\]
The hierarchy is then completed by RL locomotion controllers that map high-level velocity commands to joint-level actions [2509.20739]. S-Nav uses a high-level semantic search over rooms and doorways, then a Subproblem Solver to decompose the route into room-bounded geometric subproblems solved by IRRT* or related OMPL planners [2307.01613]. IntelliMove exposes the same pattern as Targeted Navigation Mode, Multi-target Exploration Mode, and Discovery Mode, all implemented over a semantic graph with Dijkstra plus an LLM-based room guess when no matching target is present in the map [2410.14851]. The HSG method plans first at the region level and then at the zone level, using macro-actions \(\text{visit}(r)\) and \(\text{visit}(z)\) and finite-horizon rollouts on an HSG-based simulator to estimate expected return [2606.31071]. LEAPS likewise separates high-level semantic planning from low-level execution: a Bayesian semantic model proposes the next room-type sub-target, and a model-free visual sub-policy executes that sub-target [1809.10842].

A second pattern is explicit two-stage decision making. In recommender systems, semantic retrieval maps directly to an item or semantic ID, whereas semantic planning inserts an intermediate semantic target \(z_t\) that specifies what the next exposure should accomplish before instantiation [2607.09540]. PSALM applies the same decomposition to symbolic planning: the LLM infers action semantics \(\Phi_a\) from execution outcomes, maintains a belief over candidate semantic statements, and a classical planner then plans over the induced PDDL model [2406.02791]. In video generation, Bernini assigns semantic reasoning to an MLLM planner and pixel generation to a DiT-based renderer, with semantics as the interface between the two [2605.22344].

A third pattern is planning directly in a learned semantic dynamical system. SCOPE formulates conversation as an MDP and replaces text-state rollouts with rollouts in latent semantic space using learned \(\widetilde{T}\) and \(\widetilde{R}\), allowing MCTS to run entirely in that space [2503.11586]. Semformer does not perform tree search, but it introduces an internal planning phase in which planning tokens must summarize the forthcoming response before next-token prediction, thereby modifying the language model’s computational structure [2409.11143].

This architecture-level diversity clarifies a common misconception: semantic planning is not merely “planning with labels.” In the literature it can refer to graph search over semantically typed nodes, latent-space planning over dense embeddings, target-first decomposition, or logical planning over learned action semantics.

## 4. Uncertainty, belief, and formal guarantees

Many semantic planners explicitly model uncertainty. In the SLAM-free navigation graph, node exploration probability is updated by a Bayesian rule,
\[
P_{\text{explore}^{\text{new}}(v_i) = \frac{P(\text{evidence}\mid v_i)\, P_{\text{explore}^{\text{old}}(v_i)}}{P(\text{evidence})},
\]
so that semantically exhausted nodes decay toward zero exploration value [2509.20739]. LEAPS keeps a Bayesian belief over room-type connectivity and updates \(P(z_{i,j}=1\mid \mathcal{Y}_{i,j})\) from local exploration observations, then plans on the posterior graph [1809.10842]. DeepSemanticHPPC goes further by carrying per-point semantic probabilities and uncertainties, classifying points as safe, unsafe, or unclear, clustering unclear regions, and maintaining multiple path hypotheses through different uncertain terrain regions [2003.03464]. The HSG planner fuses semantic priors with frontier-based exploration evidence into a belief over regions and zones, then runs POUCT on that belief [2606.31071].

Some frameworks attach explicit correctness guarantees to semantic reasoning. Safe Planning in Unknown Environments using Conformalized Semantic Maps defines per-cell prediction sets
\[
\mathcal{C}_j(\mathbf{m}_t)=\{k\in\mathcal{K}\mid p_t(m_j=k)\ge 1-\hat{s}\},
\]
and proves a path-agnostic coverage guarantee
\[
\mathbb{P}_{\xi\sim\mathcal{D}}\big(C_{0:H}\mid \mathbf{m}_{0:H}\big)\ge 1-\alpha,
\]
which in turn yields user-specified mission completion rates for semantic reach-avoid tasks [2509.25124]. Perception-Based Temporal Logic Planning in Uncertain Semantic Maps embeds probabilistic semantic conditions directly in atomic predicates, such as
\[
\mathbb{P}\big(\|\mathbf{p}_j(t)-\mathbf{x}_i\|\le r\big)\ge 1-\delta,
\]
and then lifts them into co-safe LTL specifications over a hybrid state space consisting of robot state, semantic-map belief, and automaton state [2012.10490]. In epistemic planning, decidability is obtained by augmenting S5\(_n\) with the commutativity axiom
\[
K_iK_j\varphi \rightarrow K_jK_i\varphi,
\]
which bounds higher-order knowledge, yields a finitary characterization of common knowledge, and makes the resulting planning problem decidable [2307.15485].

These results also show that semantic planning often trades raw expressivity for structured conservatism. Conformalized maps choose the most conservative class in a prediction set, and epistemic commutativity rules out unbounded knowledge nesting. A plausible implication is that guarantees in semantic planning usually depend on carefully chosen abstractions, calibration assumptions, or admissible semantic dynamics.

## 5. Representative realizations and empirical profile

Robotic navigation provides the densest empirical record. The SLAM-free legged-navigation framework reports Semantic Accuracy up to \(88.8\%\), Obstacle Avoidance Success Rate \(\approx 81.3\%\), simulation Success Rate \(50\text{–}75\%\) with SPL up to \(46.3\%\), and real-world Success Rate \(30\text{–}50\%\) with SPL up to \(32.6\%\); Table 2 further shows that GPT-4-based node scoring improves Global Node Selection Accuracy by \(+5.1\%\) over probability-only ranking, and adding cost-aware optimization adds another \(+1.9\%\) on average [2509.20739]. S-Nav evaluates three configurations on a synthetic indoor map of \(17\text{ m} \times 15\text{ m}\) with 8 rooms and 10 doorways over 1000 planning queries with a 0.1 s timeout, and reports that IRRT* + S-Graphs + SPS yields the shortest paths and more samples within the same time budget [2307.01613]. IntelliMove reports average planning time 7 ms, maximum 10 ms, success rate 99% for known targets, and 55% for goal discovery scenarios in a simulated office environment [2410.14851]. DeepSemanticHPPC shows that semantics and active disambiguation matter in unstructured terrain: on Mann Library, B1 gives 0% safe and 100% unsafe, whereas DeepSemanticHPPC with 5 NBVs gives 86.0% safe and 3.8% unsafe, with CS = 28.0% and CN = 10.2%; on Cass Park, 5 NBVs gives 59.2% safe and 36.6% unsafe [2003.03464].

Language-centered systems display the same pattern in nonphysical state spaces. SCOPE performs conversation planning 70 times faster than conventional simulation-based planning algorithms, reports average rollout time \(0.18 \pm 0.028\) s versus \(16.63 \pm 10.98\) s for vanilla MCTS, and yields conversations with about 150 more human tokens over 5 turns on average for the length-based reward [2503.11586]. PSALM learns action semantics from one goal, boosting plan success rate from 36.4% to 100% across 7 environments while improving exploration efficiency for recovering ground-truth domain action semantics [2406.02791]. Semformer shows near-perfect performance on graph path-finding and improves downstream language metrics, including Wikitext perplexity 35.6 versus 37.5 for the teacher-forced baseline and LAMBADA perplexity 38.8 versus 42.5 [2409.11143].

Generation and communication systems instantiate semantic planning in yet other ways. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks by separating MLLM-based semantic planning from diffusion-based rendering [2605.22344]. Adaptive Semantic Communication for UAV/UGV Cooperative Path Planning defines semantics in terms of traversability and shortest-path stability, and reports that Path-SC reduces average data volume to 563.2 kb, compared with 988.3 kb for DeiT-SC and 1216.3 kb for JPEG, while maintaining path-planning accuracy under AWGN and Rayleigh fading [2510.06901]. The diversity of these empirical profiles indicates that semantic planning is not tied to a particular domain so much as to a particular choice of abstraction: semantically meaningful state variables become the object of planning, and downstream mechanisms handle realization.

## 6. Limitations, misconceptions, and open directions

A first limitation is that semantic planning rarely eliminates geometry or low-level execution; it reorganizes the interface between high-level reasoning and low-level control. S-Nav still relies on room contours, ESDF construction, and a geometric planner inside a reduced free-space region; IntelliMove still depends on costmaps and local planning; HSG planning still grounds macro-actions into frontier tours and Fast Marching paths; Bernini still requires a renderer to convert semantic plans into VAE latents and pixels [2307.01613] [2410.14851] [2606.31071] [2605.22344]. A persistent misconception is therefore that semantic planning replaces geometric planning; in the cited systems it more often constrains, biases, or stages it.

A second limitation is grounding. Recommender-system semantic targets must remain instantiable under current inventory and infrastructure, and the paper explicitly identifies target grounding and ID stability as open problems [2607.09540]. Bernini’s semantic plan must align with what the renderer can actually synthesize, which is why it uses a modular but lightly co-trained interface [2605.22344]. PSALM assumes typed predicates, action headers, deterministic transitions, and informative error messages; without error messages, accuracy degrades sharply in more complex domains [2406.02791]. Semformer improves planning behavior but still struggles on substantially out-of-distribution graphs and very long paths, and it increases training complexity through the autoencoder and latent-prediction objective [2409.11143].

A third limitation concerns assumptions behind uncertainty handling and guarantees. Conformalized semantic maps guarantee mission completion rates only when test missions are drawn from the same distribution as the calibration scenarios; in out-of-distribution environments, mapping success drops even when mission success remains relatively high [2509.25124]. Perception-based temporal-logic planning assumes known geometry, static semantic landmarks, and a linear-Gaussian structure that permits Riccati-based covariance prediction [2012.10490]. The HSG framework evaluates in simulation with perfect pose, and explicitly notes latency from VLMs and LLMs as a challenge for real-time operation [2606.31071]. DeepSemanticHPPC assumes predefined safe and unsafe semantic classes, static terrain, and a computationally heavy MC-dropout pipeline [2003.03464].

Current directions suggest two broad trajectories. One is richer semantic structure: objects, zones, regions, room types, semantic IDs, semantic targets, planning tokens, and ViT-space plans all function as intermediate representations that could be made more compositional. The other is stronger coupling between semantics and guarantees: belief updates, conformal prediction, temporal logic, and semantic axioms show that semantics can be embedded not only in heuristics but also in formal planning machinery. Taken together, the literature indicates that semantic planning is becoming less a domain-specific label for “planning with meaning” and more a general design pattern: plan in a semantic substrate, then hand off execution to modules specialized for geometry, language realization, retrieval, or rendering.

Source: https://www.emergentmind.com/topics/semantic-planning