DuetSim: Dual-LLM User Simulation

• DuetSim is a dual-LLM framework that simulates user interactions in task-oriented dialogue by integrating a Generator and a Verifier to ensure complex goal constraints are met.
• It uses an iterative process where the Generator employs chain-of-thought reasoning to produce dialogue acts, and the Verifier checks for contextual consistency and adherence to requirements.
• Empirical evaluations demonstrate that DuetSim improves language diversity and user goal fulfillment compared to traditional rule-based and single-LLM simulators.

DuetSim is a framework for simulating users in task-oriented dialogue systems, leveraging two LLMs acting in tandem to enhance both the diversity and accuracy of simulated user interactions. It addresses long-standing challenges in user simulation—namely, the need for spontaneous, varied, and contextually precise user utterances while satisfying complex goal constraints that arise in realistic task-based dialogue.

1. Motivation and Comparative Limitations of Preceding Simulators

Traditional user simulators for task-oriented dialogue have typically fallen into three categories:

• Rule-based / agenda-based simulators (e.g., ABUS): These rely on expert-crafted agendas and deterministic templates, which provide strong task correctness in small domains. However, they exhibit extremely limited lexical variability and entail substantial human engineering effort, resulting in poor scalability to new domains due to the need for handcrafted rules and templates.
• Data-driven / neural simulators (e.g., Seq2Seq-based): These are trained end-to-end on human–human dialogue corpora, enabling richer language variability. However, they require extensive annotated data for each domain and encounter significant generalization limitations when domain-specific data is scarce.
• Single-LLM prompt-based simulators (e.g., PBUS): Such models utilize in-context learning or few-shot prompting with GPT-style LLMs. They require no parameter tuning and enable rapid domain transfer, but are sensitive to prompt length and ordering, frequently omitting constraints located in the middle sections of long prompts. Critically, they struggle to reliably enforce satisfaction of all user goal constraints (slots, values, sequence requirements) at each turn.

DuetSim was designed to address these deficiencies by combining LLM-driven diversity and domain flexibility with a structured, iterative constraint-verification mechanism that systematically enforces complex user goal satisfaction via a dedicated Verifier module (Luo et al., 2024).

2. Dual-LLM Architecture and Dialogue Act Generation Pipeline

DuetSim operates via two distinct LLM modules per user turn: a "Generator" and a "Verifier." Each dialogue step comprises the following:

• Prompt Construction:
  • User goal $G$: Structured set of slot–value requirements, presented as natural-language bullet points or a JSON-style list.
  • Dialogue context $C$: Past utterances concatenated as alternating “USER: …” and “SYSTEM: …” lines.
  • Requirement list $R$: Enumerated constraints (e.g., “Must request missing slots: X, Y”; “Must not mention out-of-scope information”).
• Generator LLM:
  • Input: “You are a CUSTOMER. Goal: G. History: C. Requirements: R. Generate next dialogue act via Chain-of-Thought.”
  • Output: Candidate dialogue act UG, structured as JSON with "intent," "domain," "slot," and "value."
• Verifier LLM:
  • Input: “You are a supervisor. History: C. Requirements: R. Candidate act: UG. Check for context consistency, completeness, no spurious slots.”
  • Output: “ACCEPT” or “REJECT: (reason).”
• Iterative Refinement:
  • If "ACCEPT", UG is finalized as the next user move; if "REJECT", the feedback is added to $R$ and the Generator is re-invoked, up to $K$ iterations.

Dialogue state is implicitly maintained via $G$ and $C$; there is no explicit dialogue state tracking (DST) module. The chain-of-thought decomposition in the Generator LLM (prompting sequentially for intent, then domain, slot, and value) is designed to optimize semantic control (Luo et al., 2024).

3. Formalism and Absence of Explicit Training Objectives

DuetSim utilizes off-the-shelf LLMs in a zero-shot prompting setting, with no gradient-based parameter updates or supervised learning objectives reported.

• Generator LLM: Defines a conditional distribution $p_{\mathrm{gen}}(UG \mid G,C,R_G)$, where sampling operates over next-token probabilities.
• Verifier LLM: Defines a scoring function $s_{\mathrm{verif}}(UG; G, C, R_V) \in [0, 1]$, which, in practice, is realized as binary ("accept"/"reject") plus feedback determined by prompt structure.

The iterative process can be described as:

$$UG^{(0)} \sim p_{\mathrm{gen}}(\cdot \mid G, C, R_G) \ UV^{(0)} = \text{Verifier}(UG^{(0)}) \ \text{If } UV^{(0)} = \text{REJECT}: R_G := R_G \cup \{\text{reason}\}, \text{ repeat}.$$

No explicit loss functions, supervised dataset splits, or parameter optimization routines are used. Suggested supervised enhancements (not implemented) would comprise cross-entropy loss for the Generator and classification loss for the Verifier (Luo et al., 2024).

4. Prompting Methodology and Information Encoding

Prompts are meticulously constructed and explicitly structured to support both the Generator and Verifier LLMs:

• Generator Prompt: Encodes goal, context, requirements, and a chain-of-thought instruction, requiring output of exactly one dialogue action as JSON.
• Verifier Prompt: Presents the candidate action, context, and current requirements, soliciting a "correct/incorrect" response plus violation rationale.
• Chain-of-Thought Decomposition: The Generator is prompted to output intent, then sequentially domain, slot, and value, before assembling the full result.

The requirements list $R$ is updated iteratively with any deficiencies detected by the Verifier, dynamically constraining subsequent Generator outputs. Example encoding and prompt templates are detailed in Section 4 of (Luo et al., 2024).

5. Evaluation Protocol and Comparative Data

Quantitative and qualitative evaluation leverages multiple metrics and baselines, focusing on both user goal fulfillment and language diversity, with mechanical and human evaluation:

Table 1. Goal Fulfillment Metrics on MultiWOZ (100 dialogues, averaged)

Simulator	Complete	Success	Prec.	Recall	F1	Book	Turns
ABUS (agenda-based)	0.97	0.97	.902	.983	.924	.97	10.4
PBUS (single LLM)	0.41	0.30	.580	.670	.710	.659	7.5
DuetSim (ChatGPT)	0.92	0.74	.830	.980	.881	.585	16.9
DuetSim w/o Verifier	0.85	0.67	.842	.948	.873	.544	17.8
DuetSim (FLAN-T5)	0.92	0.71	.820	.979	.872	.648	16.1
DuetSim (LLAMA2)	0.19	0.28	.748	.688	.687	.060	17.4
DuetSim (ChatGLM2)	0.19	0.25	.699	.646	.644	.332	17.2

Table 2. Utterance Diversity Metrics

Simulator	Unigrams	Bigrams	Trigrams	SE	MTLD
ABUS-templ	6.87	2.39	0.71	.75	45.97
ABUS-SC-GPT	7.04	2.37	0.76	.79	62.35
PBUS	7.40	3.00	0.70	.78	45.50
DuetSim (ChatGPT)	7.44	2.62	0.77	.78	56.98
DuetSim (FLAN-T5)	7.58	2.73	0.74	.79	50.63
DuetSim (LLAMA2)	7.11	2.14	0.77	.75	63.75

Table 3. Human Evaluation (Mean Ratings, 0–2 scale)

Simulator	Naturalness	Informativeness	Coherence
ABUS-templ	1.40	1.08	1.38
ABUS-SC-GPT	1.44	1.32	1.22
DuetSim (ChatGPT)	1.60	1.42	1.36
DuetSim (FLAN-T5)	0.66	0.30	0.52

Human judgments were collected for 200 dialogues (50 per simulator, 20 annotators). Inter-annotator agreement is not reported (Luo et al., 2024).

Qualitative inspection highlights that with the Verifier module, DuetSim prevents premature, incomplete, or context-inappropriate requests by iteratively refining output until all constraints are satisfied, as illustrated in Figure 1 of (Luo et al., 2024).

6. Strengths, Limitations, and Prospective Extensions

Strengths:

• Integrates LLM-based diversity and contextual flexibility with lightweight, plug-in constraint verification, aligning simulated dialogue acts with complex user goal requirements.
• Zero-shot and few-shot adaptability, requiring no domain-specific training or fine-tuning.
• Chain-of-thought breakdown promotes precise, semantically controlled act generation.
• Empirically demonstrated gains in language diversity and human preference compared to single-LLM prompt simulators.

Limitations:

• Dependency on commercial LLM APIs (e.g., ChatGPT) introduces costs and external service reliance.
• Iterative Generator–Verifier cycles increase latency and may impact throughput.
• No convergence guarantees for the iterative process; iteration count may exceed practical limits under some conditions.
• Absence of reported inter-annotator agreement limits confidence in human eval consistency.
• Large or complex prompts could challenge LLM context window constraints.

Future Directions:

• Extension to multi-modal dialogues incorporating visual elements.
• Employing more granular Verifier feedback (e.g., checking intermediate chain-of-thought steps).
• Adapting to longer contexts via segment-level prompting or memory augmentation.
• Investigating light Verifier fine-tuning on labeled “correct” versus “incorrect” actions to minimize iteration counts (Luo et al., 2024).

7. Context and Significance in Dialogue System Research

DuetSim marks a methodological advance by synthesizing the flexibility of LLMs with an explicit, context-sensitive constraint enforcement loop. This hybrid draws on strengths of agenda-based systems (task correctness) and neural end-to-end models (language diversity) while mitigating their principal weaknesses. Its zero-shot generality eliminates the dependency on costly labeled datasets and makes rapid domain adaptation tractable. The explicit, iterative refinement loop provides a mechanism for controlling compliance with intricate dialogue requirements, a challenge for single-pass LLM approaches.

This approach provides an empirical and architectural basis for future work on user simulation architectures—particularly in settings demanding both naturalistic variability and rigorous adherence to dialogue protocols or task constraints (Luo et al., 2024).