Tree-Structured Linguistic MDP

• Tree-Structured Linguistic MDP is a formalism that unifies hierarchical abstraction, linguistic specifications, and branching decision-making into an efficient tree-based planning framework.
• It leverages AP-MDPs for non-Markovian reward structures and TSLMs for serializing complete search trees, enabling systematic decomposition of complex tasks.
• The combined methodologies yield near-linear planning times and significant inference efficiency gains, advancing scalable reasoning and interpretable AI applications.

A Tree-Structured Linguistic Markov Decision Process (MDP) refers to a formalism unifying hierarchical abstraction, linguistic specification, and branching sequential decision-making. This paradigm merges Markovian (state/action-based) planning with linguistic, often non-Markovian, specifications to yield compact, tree-structured representations of reasoning, planning, or control tasks. Such structures are critical both for hierarchical planning from language (as in Abstract Product MDPs with LTL supervision) and for systematic tree-based exploration during LLM reasoning (as in Tree-Structured LLMs, TSLMs). Core to both approaches is the representation of solution trajectories as explicit trees, enabling decomposition, abstraction, or efficient multi-path inference.

1. Formal Foundations: Tree-Structured MDP and Product Automata

Two principal instantiations of tree-structured linguistic MDPs are found in hierarchical planning with non-Markovian rewards and supervised tree-generation in reasoning models.

In AP-MDPs (Oh et al., 2019), the environment is modeled as a labeled MDP $M = (S, A, T, s_0, AP, L, R, \gamma)$, which is combined via a product automaton with a Linear Temporal Logic (LTL) specification. The resulting Abstract Product MDP (AP–MDP) is defined as $(S_p, A_p, T_p, s_0^p, R_p, \gamma)$, where:

• $S_p$ is the union over abstraction levels $i$ of abstract-state $\hat S^i$ paired with automaton states $Q$, i.e., $S_p \subset \bigcup_{i=0}^L (\hat S^i \times Q)$.
• $A_p$ is the action set across all abstraction levels.
• $T_p$ enforces joint transitions in abstract state and automaton ($T_p((s,q),a,(s',q')) = \hat T^i(s,a,s')$ if $q' = \delta(q, L^i(s'))$; otherwise $0$).
• $R_p$ is a possibly non-Markovian reward determined by automaton progress (e.g., high for acceptance, low for goal violation).

In TSLMs (Kim et al., 30 Jan 2026), the MDP $(\mathcal S, \mathcal A, T)$ models a proof or solution tree:

• States $\mathcal S$ are partial solutions or “nodes” in the tree.
• Actions $\mathcal A(s)$ generate successors, with the transition $s' = T(s,a)$.
• Branching semantics require generating (all or top-$k$) successor actions as a set, not just a single next state.

Both frameworks contribute formally to a general class of tree-structured MDPs where planning or reasoning is realized through explicit tree expansion, governed either by linguistic (LTL) supervision or algorithmic branch management.

2. Hierarchical Abstraction and the AMDP Tree

The AP–MDP hierarchy is constructed by organizing $L+1$ levels of abstraction $i=0,\ldots,L$ into a rooted tree of Abstract MDPs (AMDPs):

• Each node $$\tilde M^i = (\tilde S^i, \tilde A^i, \tilde T^i, \tilde R^i,\tilde E^i, F^i)$$, with $\tilde S^0=S$ and coarsening for $i>0$.
• The root corresponds to the highest abstraction ($\tilde M^L$).
• Each node’s children are found through the aggregation map $F^i: \tilde S^{i-1} \rightarrow \tilde S^i$.
• Planning and backup computations operate only at the abstraction level required by the subgoal, compressing the state-action space at each node.

This tree structure enables recursive decomposition of planning tasks, as every level can be addressed independently, offering near-linear planning time relative to the size of the base MDP, versus the exponential costs of flat product construction (Oh et al., 2019).

3. Linguistic Specification and Product Construction

Linguistic commands are translated into formal LTL specifications over atomic propositions using a neural sequence-to-sequence model (Oh et al., 2019). Given a natural language utterance $u$, the process involves:

• Translating $u \rightarrow \varphi$ (LTL formula).
• Compiling $\varphi$ into a deterministic Büchi automaton $$\mathcal{A}_\varphi = (Q,\Sigma,\delta, q_0, \mathcal{F})$$, with $\Sigma = 2^{AP}$.
• Constructing the AP–MDP as the product of the base MDP and the automaton.

Non-Markovian rewards are realized by dependencies on both world-state and automaton transitions, i.e., $R_p((s,q), a, (s',q'))$. The resulting product structure yields a tree of subproblems, each tagged at the appropriate abstraction level.

A depth-first search is employed over accepting automaton paths, generating for each a sequence of level-appropriate subgoals, and each subgoal is solved via a sub-AMDP (Oh et al., 2019). The approach is sound (guaranteeing LTL satisfaction if a plan is returned) and empirically scales with increasing abstraction.

4. Token Serialization and Tree Likelihood in LLMs

TSLMs (Kim et al., 30 Jan 2026) innovate by training a transformer to encode and decode complete search trees, not just linear traces:

• Each tree node $s_i$ is mapped to a token sequence $y_{s_i} = [\text{action-description}, m_i]$ with $$m_i \in \{\texttt{[SEP]}, \texttt{[FAIL]}, \texttt{[GOAL]}\}$$.
• Structural markers communicate branching: $[SEP]$ for viable branches, $[FAIL]$ for dead ends, $[GOAL]$ for goal states, and $[BOS]/[EOS]$ for tree boundaries.
• Training optimizes the joint tree likelihood:

$$p_\theta\bigl(\{y_{s_i}\}_{s_i\in N(T)}\bigr) = \prod_{s_i\in N(T)} p_\theta\bigl(y_{s_i}\mid \text{ctx}(s_i)\bigr)$$

where $\mathrm{ctx}(s_i)$ is the ancestral path and prior siblings.

This objective instills systematic, internalized exploration, uniting successful and failed exploration under a coherent likelihood. The resulting model can serialize the full tree for efficient inference, eliminating the computational overhead of repeated prefix computation prevalent in independent sampling or external search methods.

5. Inference Procedures and Prefix Reuse

TSLM inference executes tree-structured generation in two phases:

• One-pass generation: From the root, the model generates sequences up to $[\texttt{EOS}]$, enumerating candidate children and appropriately tagging them.
• Stitching and expansion: Viable children are queued (for BFS or DFS), and expansion proceeds in batches, sharing cached transformer states for common prefixes.

A key efficiency gain lies in cache-based prefix reuse: all siblings share context, enabling a single forward pass per batch instead of $O(k)$ independent passes. This optimization enables TSLMs to achieve inference efficiency orders of magnitude greater than external search approaches, particularly at large branching factors, often matching the performance of tree-of-thought with $k=100$ samples at $k=5$ branches with $10\times$ less inference time (Kim et al., 30 Jan 2026).

6. Comparative Properties and Empirical Results

The distinctive properties of tree-structured linguistic MDPs in both AP–MDP and TSLM settings include:

Property	AP–MDP (Oh et al., 2019)	TSLM (Kim et al., 30 Jan 2026)
Abstraction/Hierarchy	Yes: explicit AMDP tree	No: flat state, but explicit search tree
Linguistic Supervision	LTL from language; neural seq2seq translator	Learned from serialized trees
Non-Markovian reward	Supported (via automaton state)	Not explicit; success/failure in token seq
Planning/Inference	Recursive over AMDP tree, each at necessary abstraction	Batch tree expansion with prefix reuse
Efficiency gains	Near-linear in env size, exponential over flat	$10\times$ faster vs. external search
Soundness/Completeness	Provable; all runs in $\mathcal{A}_\varphi$ explored	Implicit: all candidate branches modeled

Empirical validation demonstrates that the AP-MDP approach outperforms flat product MDPs in planning time for $\geq 95\%$ of tasks in moderate-size environments and $\geq 99\%$ in large environments, reducing Bellman backups similarly (Oh et al., 2019). TSLMs show strong performance on structured and open-ended reasoning tasks, robust generalization to larger problem sizes, and emergent capabilities like detecting unsolvable instances (Kim et al., 30 Jan 2026).

7. Significance, Extensions, and Open Directions

The Tree-Structured Linguistic MDP framework enables sound, complete, and scalable reasoning and planning when faced with linguistic or non-Markovian constraints. By leveraging hierarchical (AMDP) decomposition or exhaustive tree serialization (TSLMs), they provide a foundation for efficient algorithmic search, abstracted planning, and systematic reasoning over complex tasks.

A plausible implication is that tree-structured representations will be central for both interpretable AI planning and robust, efficient LLM reasoning at scale. Extensions may address richer language interfaces, stochastic environments, adaptive subtree pruning, or integration with reinforcement learning. Connections to related work span hierarchical RL, program synthesis, and tree-of-thought prompting, though the tree-structured MDP formalisms uniquely blend abstraction hierarchy, compositional language, and tree reasoning within a single computational framework (Kim et al., 30 Jan 2026, Oh et al., 2019).