PyTOD: Executable Dialogue State Tracking

• PyTOD is a neuro‐symbolic task-oriented dialogue system that generates executable, schema-aligned Python-like statements for state tracking.
• It integrates runtime execution feedback and dialogue policy signals to incrementally update dialogue state, reducing errors from full-state re-estimation.
• Experimental results on the Schema-Guided Dialogue benchmark show enhanced cross-turn consistency and superior accuracy compared to traditional DST approaches.

Searching arXiv for the PyTOD paper and closely related task-oriented dialogue state tracking work. PyTOD, short for Programmable Task-Oriented Dialogue with Execution Feedback, is a neuro-symbolic task-oriented dialogue (TOD) agent that replaces text-form dialogue-state outputs with executable, schema-aligned code (Coca et al., 21 Aug 2025). Its central premise is that dialogue state tracking (DST) is more reliable when a LLM generates compact Python-like statements that directly manipulate a simulated API object graph, rather than re-estimating a full textual state such as JSON at every turn. In this formulation, the current dialogue state is identified with the state of executable API objects, and correction is driven by two feedback channels: policy feedback from a dialogue policy simulator and execution feedback from runtime checks and schema validation (Coca et al., 21 Aug 2025). The system was introduced as a response to brittleness in grammar-constrained decoders and error-prone sequence-to-sequence DST, and it reports state-of-the-art DST accuracy and stronger cross-turn consistency on the Schema-Guided Dialogue benchmark (Coca et al., 21 Aug 2025).

1. Problem setting and motivation

Task-oriented dialogue is a practical paradigm in which a system helps a user complete structured tasks such as booking a restaurant or querying schedules through explicit APIs that specify intents or services, parameters, and schema-level constraints including types, required or optional fields, and enumerations (Coca et al., 21 Aug 2025). Within this setting, dialogue state tracking maintains the user’s goal as slot–value pairs across turns and underpins both policy compliance and correct API invocation (Coca et al., 21 Aug 2025).

PyTOD is motivated by the observation that inaccurate DST creates compounding failures. If the maintained state is wrong or incomplete, a dialogue manager may fail to request required slots, may call the wrong function, or may violate schema constraints such as typing and required-field conditions; these errors can then cascade from turn to turn (Coca et al., 21 Aug 2025). The paper positions this as a structural problem rather than a purely modeling problem: state must remain aligned with both schema semantics and policy requirements throughout a live interaction.

The work contrasts PyTOD with two prevalent alternatives. First, grammar- or rule-based constrained decoders require extensive engineering, including grammar maintenance and tokenizer alignment, and are described as brittle under changing language-model backbones and evolving APIs (Coca et al., 21 Aug 2025). Second, pure language-model sequence-to-sequence DST systems that emit JSON or strings are reported to suffer from copy errors, schema drift, and weak error recovery; many also re-estimate the entire state at each turn, which can inflate turn-level metrics without reflecting real-world consistency (Coca et al., 21 Aug 2025). The paper further notes that even robust LLM function-calling approaches often track agent-mentioned slot values from text, which may be inconsistent with ground-truth API responses (Coca et al., 21 Aug 2025).

A plausible implication is that PyTOD should be understood not merely as a new decoder, but as a redefinition of DST around executable state transitions. This suggests a closer coupling among language understanding, tool execution, and dialogue policy than in conventional text-only trackers.

2. Core architecture and execution loop

PyTOD’s Action Parser (AP) uses a LLM to generate executable program statements that directly manipulate a simulated API object graph (Coca et al., 21 Aug 2025). These statements resemble Python expressions and function calls that mirror the API schema and policy semantics. Reported examples include search calls such as x1 = find_restaurant(area="centre", city="London", cuisine="Chinese"), iteration over results with x2 = next(x1), entity selection through select(x2, from_results=x1), transactional operations such as x16 = book_restaurant(...); confirm(x16), and dot-assignments for slot updates such as x1.destination_city = "San Diego" (Coca et al., 21 Aug 2025).

The execution loop is explicit. Generated statements are parsed to an abstract syntax tree, compiled to callables, and executed against a simulated environment called pytodlib (Coca et al., 21 Aug 2025). Execution updates the state of API objects and returns policy hints, confirmations, notifications, and retrieved entities with properties (Coca et al., 21 Aug 2025). The Dialogue Manager (DM) mediates execution, enforces basic constraints such as valid Python and known API names via edit-distance correction, and triggers supervisory modules when correction is needed (Coca et al., 21 Aug 2025).

The paper presents the pipeline in six stages (Coca et al., 21 Aug 2025):

1. Inputs consist of the user utterance, session transcript history, a linearized API schema, and dynamic context such as returned entities, properties, and completed tasks.
2. The AP generates Python-like code under constrained decoding.
3. The DM parses the AST, corrects API names, executes statements in pytodlib, and obtains hints, entities, confirmations, or notifications.
4. If parsing fails or slot names or values violate the schema, the system invokes a Schema Supervisor; if required slots are omitted or semantics are wrong, it invokes a Parser Supervisor.
5. The supervisors return schema-compliant slot names or values, or missing values and corrected bindings; the DM inserts these corrections and re-executes.
6. The updated API-object state becomes the dialogue state, and policy hints guide the next system utterance through say(...).

This architecture yields an execution-aware form of state tracking in which state updates are incremental rather than fully regenerated (Coca et al., 21 Aug 2025). The paper argues that this reduces re-estimation inconsistencies and ties dialogue progress to executable environment state. A plausible implication is that PyTOD occupies a middle ground between symbolic dialogue managers and end-to-end neural DST, using program execution as the state-transition substrate.

3. Schema adherence, constrained decoding, and supervisory correction

PyTOD represents APIs as linearized Python function signatures with service and intent names, textual descriptions, slots or parameters with types, required or optional annotations, descriptions for open-value slots, enumerations for categorical slots, and described return types (Coca et al., 21 Aug 2025). Entity properties are dynamically exposed after retrieval (Coca et al., 21 Aug 2025). This schema representation defines both the generation space and the correction space.

Schema adherence is implemented through three mechanisms (Coca et al., 21 Aug 2025). First, the DM enforces valid Python, restricts API names to those present in the AP header, and applies minimal edit-distance correction to API names when necessary. Second, the Schema Supervisor (SS) uses zero-shot constrained generation via multiple-choice question answering prompts to map unknown or memorized slot names or categorical values to schema-compliant alternatives (Coca et al., 21 Aug 2025). Third, the Parser Supervisor (PS) uses extractive QA prompts over the current-task dialogue history to fill omitted required slots or repair semantic slot misassignments (Coca et al., 21 Aug 2025).

The SS is the main alternative to grammar-based masking. Rather than hand-writing grammars, PyTOD supplies short multiple-choice prompts listing schema slot names, type or description information, or categorical enumerations as answer choices; the LLM then selects among schema-constrained options (Coca et al., 21 Aug 2025). For categorical values, the options are reduced to enumerated values only (Coca et al., 21 Aug 2025). The paper formalizes this conceptually as constrained softmax over an allowable token set $A$:

$$p(t \mid x) = \frac{\exp(z_t)}{\sum_{j \in A}\exp(z_j)} \quad \text{for } t \in A,$$

with $p(t \mid x)=0$ otherwise, or equivalently with masked logits $z'_t = z_t$ for $t \in A$ and $z'_t = -\infty$ otherwise, followed by $p(t \mid x)=\mathrm{softmax}(z')$ (Coca et al., 21 Aug 2025). In the actual system, this effect is realized through short MQA prompts rather than explicit token masking (Coca et al., 21 Aug 2025).

The PS operates after AP output has been constrained by SS and executed (Coca et al., 21 Aug 2025). When policy feedback indicates that required slots remain unfilled, PS is prompted with per-slot questions derived from schema descriptions and the current task’s dialogue history; it extracts missing values and appends assignments such as x1.departure_date = "7th of this month" (Coca et al., 21 Aug 2025). When semantic confusion is detected, for example a value already assigned to a different open-value slot, PS can swap the slot name to repair the mismatch (Coca et al., 21 Aug 2025).

The following table summarizes the principal control components.

Component	Function	Trigger
AP	Generates executable Python-like statements	Initial state update generation
DM	Parses AST, enforces validity, executes in pytodlib, inserts corrections	Every generated action
SS	Maps unknown slot names or categorical values to schema-compliant choices via MQA	Unknown slot names, memorized names, categorical violations
PS	Fills omitted required slots or repairs semantic misassignments via extractive QA	Policy-detected omissions or suspected semantic confusion

The paper’s qualitative example illustrates the correction loop. For the user utterance “Book a flight to San Diego,” AP produces x1.to_city = "San Diego"; DM detects that to_city is not in the schema and invokes SS, which answers destination_city, after which DM rewrites the code accordingly (Coca et al., 21 Aug 2025). If the system had previously asked for departure_date, DM also invokes PS, which extracts “7th of this month” and appends the corresponding assignment (Coca et al., 21 Aug 2025). This example shows that correction is grounded in both schema validity and policy state.

4. Mathematical formulation and evaluation criteria

When fine-tuned, PyTOD uses a text-to-text pretrained LLM, specifically FlanT5, trained to emit gold program code or QA answers under schema-informed prompting (Coca et al., 21 Aug 2025). The training objective for AP and PS is token-level negative log-likelihood:

$$\mathcal{L} = - \sum_{i=1}^{T} \log p(y_i \mid y_{<i}, x, schema)$$

(Coca et al., 21 Aug 2025).

Evaluation is centered on dialogue-state tracking metrics (Coca et al., 21 Aug 2025). The paper uses Joint Goal Accuracy (JGA), defined for predicted state $\hat{S}_i$ and gold state $S_i$ as

$$p(t \mid x) = \frac{\exp(z_t)}{\sum_{j \in A}\exp(z_j)} \quad \text{for } t \in A,$$0

It also notes slot-level precision, recall, and F1,

$$p(t \mid x) = \frac{\exp(z_t)}{\sum_{j \in A}\exp(z_j)} \quad \text{for } t \in A,$$1

though these are not central to the paper’s evaluation (Coca et al., 21 Aug 2025).

A distinctive contribution is turn-wise consistency (C-JGA), described as a stricter metric introduced by PyTOD (Coca et al., 21 Aug 2025). Under this criterion, a turn counts as correct only if the current state is jointly correct and all previous turns in the same task were also jointly correct (Coca et al., 21 Aug 2025). This directly penalizes re-estimation inconsistencies that would degrade real interactions. In the context of PyTOD’s design, C-JGA operationalizes the claim that correct tracking should be persistent across dialogue progression, not merely locally accurate at isolated turns.

This metric choice is significant because it aligns evaluation with the architectural emphasis on incremental executable updates. A plausible implication is that PyTOD’s reported gains are not only about slot identification but also about maintaining a coherent executable state trajectory across turns.

5. Experimental results on Schema-Guided Dialogue

Experiments are conducted on the Schema-Guided Dialogue (SGD) dataset, which contains 16,142 training dialogues covering 26 services and 4,201 test dialogues covering 21 services (Coca et al., 21 Aug 2025). Generalization is explicitly stressed: among 90 distinct test task sequences, 85.6% involve an unseen schema at test time, covering 77% of test dialogues (Coca et al., 21 Aug 2025). The system normalizes open-valued parameters before API calls and extends evaluator annotations with canonical values to align execution-tracked slots with DSTC8 evaluation (Coca et al., 21 Aug 2025).

The principal quantitative results reported on SGD with the official evaluator are as follows (Coca et al., 21 Aug 2025):

Model	JGA	C-JGA
PyTOD (Base, FlanT5-220M)	76.8	72.7
PyTOD (Large, FlanT5-780M)	82.2	78.4
D3ST (replicated, 220M)	71.2	62.2
D3ST (replicated, 780M)	76.5	67.7
SDT-Seq (replicated, 220M)	77.5	68.7
SDT-Seq (replicated, 780M)	82.7	74.2

The paper also reports seen and unseen schema performance. PyTOD (Base) obtains Seen 91.0 and Unseen 71.8, while PyTOD (Large) obtains Seen 92.1 and Unseen 78.9 (Coca et al., 21 Aug 2025). Additional baselines include T5DST (220M), reported at JGA 72.6; Seen 89.7; Unseen 66.9, and custom-evaluation results for LLaMA (7B) at JGA 75.3 and LDST (7B, augmented prompts) at JGA 84.5 (Coca et al., 21 Aug 2025).

The reported gains emphasize both absolute performance and robustness. PyTOD (Base) surpasses D3ST (220M) by +5.6% JGA and T5DST (220M) by +4.2% JGA, with stronger unseen-domain performance (Coca et al., 21 Aug 2025). PyTOD (Large) improves over D3ST (780M) by +5.7% JGA (Coca et al., 21 Aug 2025). On cross-turn consistency, D3ST’s JGA drops sharply under C-JGA, by −9.0% at 220M and −8.8% at 780M, whereas PyTOD’s drop is smaller at −4.1% at 220M and −3.8% at 780M (Coca et al., 21 Aug 2025). The paper interprets this as evidence that incremental, execution-grounded state updates better preserve dialogue continuity.

The paper further notes that PyTOD compares favorably to LLaMA while being 10–32× smaller, although LDST exceeds PyTOD (Large) by 2.3% using multi-prompt augmentation not used by PyTOD (Coca et al., 21 Aug 2025). This establishes PyTOD’s position as a compact but highly competitive DST system under schema generalization pressure.

6. Efficiency, implementation, and practical deployment

PyTOD is implemented with FlanT5-base (220M) or FlanT5-large (780M) as the AP, with the PS sharing parameters with AP and trained multitask on 34,105 extractive QA prompts constructed from SGD, including negative examples to reduce hallucinations (Coca et al., 21 Aug 2025). The default SS is zero-shot FlanT5-3B, though the paper reports that SS size can be reduced to 220M with small JGA losses of −2.7% at 220M AP and −1.7% at 780M AP (Coca et al., 21 Aug 2025).

The implementation details emphasize compact code outputs, linearized schema headers, interleaving of transcript code and utterances, and dynamic instructions that list entity properties after iteration or policy tips after confirmation (Coca et al., 21 Aug 2025). SS uses three MQA templates, and PS uses an extractive QA template filtered to the current task (Coca et al., 21 Aug 2025). Decoding is recommended with greedy or low-temperature settings to minimize generation length and maximize validity, while the DM is expected to enforce Python parsing and API-name correction and to invoke SS and PS on demand (Coca et al., 21 Aug 2025).

The simulator environment, pytodlib, models 58 SGD APIs (Coca et al., 21 Aug 2025). Execution is sandboxed: code is parsed to AST and converted to controlled callables, with no external side effects beyond simulation; tool calls and database responses are simulated and normalized (Coca et al., 21 Aug 2025). Descriptors enforce types and can be configured to return verbalizable error strings instead of exceptions (Coca et al., 21 Aug 2025). This is central to the system’s safety model, because the architectural decision to “treat TOD as code” would otherwise raise execution concerns.

Latency is presented as moderate and manageable. For approximately 53k test turns on an A100, AP=220M achieves 9.11 samples/s with no SS and 6.32 samples/s with SS=3B, corresponding to 1.46× latency; with AP=780M, no SS yields 3.33 samples/s and SS=3B yields 2.90 samples/s, corresponding to 1.15× latency (Coca et al., 21 Aug 2025). The paper states that SS size has minimal impact on runtime: with AP=220M, SS=3B versus 220M increases relative latency by only ~11%, and with AP=780M, SS=3B increases by ~2% (Coca et al., 21 Aug 2025). Most overhead comes from on-demand SS model loading, and keeping SS resident reduces load time at the cost of more memory (Coca et al., 21 Aug 2025).

One reported practical trade-off is especially notable: AP+SS at 220M can run 1.89× faster than a 780M AP while being more accurate (+1.7% JGA) (Coca et al., 21 Aug 2025). This suggests that PyTOD’s decomposition into a smaller parser plus a lightweight corrective pathway can be favorable in deployment settings where both latency and accuracy matter.

7. Interpretation, limitations, and research directions

PyTOD’s reported robustness is attributed to several design choices (Coca et al., 21 Aug 2025). Incremental code updates avoid regenerating the full state each turn. Past program variables preserve continuity. Agent-mentioned slot values are sourced from retrieved entities or API responses rather than copied from text, reducing copy errors. Policy feedback tightly couples state to downstream dialogue actions, while execution feedback surfaces concrete violations such as unknown slots, type mismatches, missing required fields, and categorical value violations (Coca et al., 21 Aug 2025). The paper states that these advantages are particularly helpful in longer dialogues with compositional constraints and strict typing or enumerations (Coca et al., 21 Aug 2025).

The ablation study supports this interpretation. The contribution of PS is reported as +1.0% JGA for the base model and +1.6% for the large model (Coca et al., 21 Aug 2025). The contribution of SS is substantially larger: +11.4% JGA for the base model and +6.5% for the large model over configurations without SS (Coca et al., 21 Aug 2025). Multitask learning with shared AP/PS parameters has negligible effect on parsing accuracy but simplifies deployment (Coca et al., 21 Aug 2025). These results indicate that schema-constrained correction via MQA is the dominant contributor among the auxiliary mechanisms.

The paper also identifies several limitations (Coca et al., 21 Aug 2025). PyTOD assumes service knowledge, so disambiguation errors among closely related services such as Buses_1, Buses_2, and Buses_3 can degrade performance if optional disambiguating slots are missing. The approach relies on accurate schema descriptions and does not explicitly evaluate brittleness under major schema-style variation. Latency remains moderately higher because of SS and PS, even if the overhead is controlled. Executable code requires sandboxing, which PyTOD addresses through AST verification and simulator confinement. Domain transfer is still sensitive to annotation errors and to some copy or coreference biases, although SS helps with schema-name drift (Coca et al., 21 Aug 2025).

The stated opportunities for further work include integrating reinforcement learning or planning with policy-level feedback, stronger typing and verification, more explicit masked decoding, dynamic schema induction and adaptive SS beyond MQA lists, multimodal extensions such as grounding in user interfaces or vision, and better API retrieval across multi-session contexts via vector stores and personalized headers (Coca et al., 21 Aug 2025). These directions follow naturally from the architecture’s emphasis on executable intermediate representations and environment-mediated correction.

In sum, PyTOD frames task-oriented dialogue state tracking as executable program synthesis under schema and policy constraints, with correction driven by runtime and policy feedback rather than solely by text generation (Coca et al., 21 Aug 2025). Its reported results suggest that execution-aware, LM-generated code can surpass strong text-based DST baselines, improve cross-turn stability, and reduce dependence on hand-crafted grammars while retaining a practical deployment path under simulator-backed tool use (Coca et al., 21 Aug 2025).