Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking (1910.03544v4)

Abstract: Dialog state tracking (DST) is a core component in task-oriented dialog systems. Existing approaches for DST mainly fall into one of two categories, namely, ontology-based and ontology-free methods. An ontology-based method selects a value from a candidate-value list for each target slot, while an ontology-free method extracts spans from dialog contexts. Recent work introduced a BERT-based model to strike a balance between the two methods by pre-defining categorical and non-categorical slots. However, it is not clear enough which slots are better handled by either of the two slot types, and the way to use the pre-trained model has not been well investigated. In this paper, we propose a simple yet effective dual-strategy model for DST, by adapting a single BERT-style reading comprehension model to jointly handle both the categorical and non-categorical slots. Our experiments on the MultiWOZ datasets show that our method significantly outperforms the BERT-based counterpart, finding that the key is a deep interaction between the domain-slot and context information. When evaluated on noisy (MultiWOZ 2.0) and cleaner (MultiWOZ 2.1) settings, our method performs competitively and robustly across the two different settings. Our method sets the new state of the art in the noisy setting, while performing more robustly than the best model in the cleaner setting. We also conduct a comprehensive error analysis on the dataset, including the effects of the dual strategy for each slot, to facilitate future research.

Dual Strategy for Slot-Value Predictions in Multi-Domain Dialog State Tracking

The paper addresses the core challenge within task-oriented dialog systems known as Dialog State Tracking (DST). DST involves determining a user's goals and intentions based on the dialogue history, resulting in a set of triplets comprising domain, slots, and values. Existing methods in the DST domain primarily fall into two categories: ontology-based and ontology-free approaches. The former requires accessing a complete list of possible slot values (ontology) for each domain, whereas the latter extracts these values directly from dialogue context when the ontology is partially or entirely unavailable.

This research proposes a dual-strategy model, known as DS-DST, which integrates both ontology-based and ontology-free approaches via a BERT-style reading comprehension model. The primary innovation of DS-DST is its ability to adaptively handle both categorical slots—with predefined possible values—and non-categorical slots—requiring a flexible approach—simultaneously. This is achieved through the model's architecture that allows for substantial interaction between domain-slot pairs and contextual dialogue information.

Empirical results show that DS-DST significantly outperforms existing BERT-based models in a multi-domain environment, as evidenced by experiments conducted on MultiWOZ 2.0 and MultiWOZ 2.1 datasets. The model achieves a higher joint accuracy, indicating superior performance, especially within noisy and cleaner dialogue settings. Notably, DS-DST sets new state-of-the-art performance standards in noisy environments and demonstrates robust accuracy in cleaner ones.

Key Numerical Results and Comparisons

The DS-DST obtains a joint accuracy of 52.24% on MultiWOZ 2.0 and 51.21% on MultiWOZ 2.1, surpassing other competitive models such as TRADE and COMER, especially in terms of categorical slot handling. Detailed results show that while the DS-DST model excels in combining the strategies for slot value extraction, the separate training of its components yields lower performance, emphasizing the importance of joint learning in the model architecture.

Implications for Future Research and Development

The approach taken by DS-DST has broad implications for developing DST frameworks capable of more flexible domain adaptation with limited ontology access. Future work could explore enhancements in the slot-gate classification module, as error analysis suggests notable error rates in predicting some slot types. Moreover, improving the DST evaluation metrics, moving beyond string matching, might yield a more robust assessment of DST systems' capabilities.

Conclusion

The dual strategy for DST presented in this research highlights a potential pathway to advancing dialog systems through improved slot-value prediction mechanics. The model successfully manages the hybrid nature of slot-value extraction across multiple domains, leveraging the strengths of pre-trained BERT models combined with innovative joint training strategies to provide a resilient solution to dialog state tracking challenges. Future research will benefit from further exploring the interplay between categorical and non-categorical slot handling to refine these systems' accuracy and applicability.