ConvLab-2 Dialogue Toolkit

• ConvLab-2 is an open-source toolkit that provides a modular and extensible architecture for designing, evaluating, and diagnosing task-oriented dialogue systems.
• The toolkit supports multiple pipeline configurations—including traditional, word-level, end-to-end, and hybrid models—to integrate models like BERTNLU and TRADE.
• It offers comprehensive diagnostic and interactive tools for error analysis, system debugging, and rapid prototyping across diverse datasets such as MultiWOZ and CrossWOZ.

ConvLab-2 is an open-source toolkit designed for building, evaluating, and diagnosing task-oriented dialogue systems with a modular and extensible architecture. Developed as the successor to ConvLab, ConvLab-2 introduces state-of-the-art models throughout the dialogue pipeline, supports additional datasets, and incorporates comprehensive diagnostic tools for error analysis and system debugging (Zhu et al., 2020).

1. Architectural Framework

ConvLab-2 employs a unified “Agent” abstraction, where every participant in a dialogue—be it the system or a user simulator—is modeled as an Agent. Each Agent manages a sequence of submodules corresponding to four canonical dialogue components: Natural Language Understanding (NLU), Dialogue State Tracking (DST), Policy (POL), and Natural Language Generation (NLG). The top-level framework supports several agent pipeline configurations:

• Traditional pipeline: NLU → DST → POL → NLG
• Word-level pipeline: NLU → word-level DST/POL → NLG
• End-to-end models: a single neural model maps system input to output text directly (e.g., Sequicity, DAMD)
• Hybrid systems: e.g., agents combining two policies or supporting multi-party conversations

A standard pipeline Agent processes one dialogue turn by invoking its submodules in order, formalized as:

$\mathbf{b}_t$3

Where utt_in/utt_sys denotes text utterances, da_user/da_sys represent dialogue acts, and st is the belief state. The belief-state update mechanism is generic:

$$\mathbf{b}_t = \mathcal{F}\bigl(\mathbf{b}_{t-1},\, \mathrm{DA}^{(u)}_t\bigr)$$

where $\mathbf{b}_t$ is the current belief state, $\mathrm{DA}^{(u)}_t$ is the parsed user act, and $\mathcal{F}$ may be a deterministic rule or, for neural DST, an encoder–generator as in TRADE:

$$\hat{v}_{d,s} = \mathrm{GEN}\bigl(\mathrm{ENC}(H_t,\, d,s)\bigr)$$

for each domain $d$ and slot $s$.

2. Supported Component Models and Datasets

ConvLab-2 provides a suite of established models for each pipeline component:

Component	Models
Natural Language Understanding (NLU)	STC, MILU, BERTNLU (BERT+MLP, state-of-the-art on MultiWOZ)
Dialogue State Tracking (DST)	RuleDST (rule-based), MDBT, SUMBT, TRADE (copy-based neural DST)
Dialogue Policy (POL)	RulePolicy, imitation learning, RL (REINFORCE, PPO, GDPL)
Natural Language Generation (NLG)	Template-based, SC-LSTM
Word-level Policy	MDRG, HDSA, LaRL
User Policy (Simulator)	Agenda-based, HUS and variational extensions
End-to-end Models	Sequicity (multi-domain), DAMD, RL-based ROLLOUTS for DealOrNoDeal

Dataset support spans major benchmarks with unified loading and standardized field access:

• CamRest676 (676 restaurant dialogues)
• MultiWOZ (10k multi-domain Wizard-of-Oz dialogues), including MultiWOZ 2.1
• DealOrNoDeal (5.8k negotiation dialogues)
• CrossWOZ (6k Chinese multi-domain dialogues)

3. Evaluation Process and Diagnostic Tools

ConvLab-2 includes a complete end-to-end evaluation pipeline. The BiSession class coordinates simulated dialogues between system and user Agents, alternating system outputs and user responses for multiple dialogues until explicit termination is signaled. Key dialogue system evaluation metrics are:

• Success Rate:

$$\mathrm{SuccRate} = \frac{1}{N} \sum_{i=1}^N \mathbb{1}[\text{TaskSuccess}_i]$$

• Inform $F_1$ (correct slot-value mentions):

$$\mathrm{Inform}\,F_1 = \frac{2PR}{P+R}, \quad P = \frac{\#\text{correct\_informs}}{\#\text{system\_informs}}, \quad R = \frac{\#\text{correct\_informs}}{\#\text{gold\_informs}}$$

• Joint Goal Accuracy:

$\mathbf{b}_t$0

• Average Turn Number

Two dedicated diagnostic tools are provided:

• Analysis Tool: Post-simulation, “Analyzer” generates detailed HTML reports per domain, presenting success metrics, confusion matrices (e.g., for NLU errors), and common causes for dialogue loops. For example, in the Hotel domain, “Request-Hotel-Postcode” is mis-parsed as “Request-Hospital-Postcode” 34% of the time, with 53% of dialogue loops attributed to failed “Request-Hotel-Phone” handling.
• Interactive Tool: A web-based interface supports live model selection for each dialogue component, stepwise inspection (with JSON outputs for NLU, DST, POL, NLG), and manual editing of any intermediate output. The “Recall” button allows developers to re-execute downstream processing from a corrected intermediate state.

4. Demonstrative Results and Analysis

On MultiWOZ, a reference pipeline system employing BERTNLU + RuleDST + RulePolicy + TemplateNLG (for both system and user simulator) achieves, over 1,000 dialogue simulations:

• Success Rate: 64.2%
• Inform $\mathbf{b}_t$1: 67.0%

Further diagnostic findings include:

• The Hotel domain exhibits the lowest performance (approximate Success Rate: 60.8%; Inform $\mathbf{b}_t$2: 44.5%), whereas the Hospital domain approaches perfect results.
• 53% of dialogue loops in Hotel are caused by requests for phone information.
• Misclassification of NLU domains is a predominant error, especially for postcode, address, and phone fields, leading to mis-tagging between 30–40% of the time.
• The “Parking” slot is a significant source of errors at both the policy decision and NLG stages.

Using the interactive tool, the root cause of failed dialogues can often be localized to NLU domain misclassification; correcting the NLU output and recalling subsequent components reveals the downstream impact, emphasizing the importance of improving BERTNLU's domain-intent prediction head.

5. Extensibility and Practical Usage

ConvLab-2 is structured for extensibility and rapid integration of new models and datasets. Installation can be accomplished via direct cloning (pip install -r requirements.txt), with future support for pip install convlab2. Core modules include:

• convlab2.core.agent: Agent abstractions for pipelines and end-to-end models
• convlab2.nlu, convlab2.dst, convlab2.policy, convlab2.nlg, convlab2.userSimulator: Model implementations
• convlab2.evaluator: Evaluation (BiSession, dataset-specific evaluators)
• convlab2.analysis: Analyzer for diagnostic reports
• convlab2.interact: Interactive web server

To incorporate new models, developers implement the appropriate interface in convlab2.core and register it in the configuration YAML under models/. New datasets require a loader returning the standardized fields (utterance, dialogue_acts, state_labels) and registration in the dataset loader registry.

Primary applications include research on novel NLU, DST, or NLG algorithms, rapid prototyping for RL-based policy learning, multi-domain transfer studies, and deployment for real-world tasks such as assistants in restaurant, hotel, or hospital domains, including Chinese cross-domain dialogue with CrossWOZ. ConvLab-2 serves as the platform for the MultiWOZ track in DSTC9.

ConvLab-2 thus combines a modular architecture with broad algorithmic coverage and sophisticated diagnostic facilities, supporting development and benchmarking of robust, task-oriented dialogue systems (Zhu et al., 2020).