Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition (2403.18973v1)

Published 27 Mar 2024 in cs.CL

Abstract: We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level. By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection. In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection. CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Shailesh Acharya and Glenn Fung. 2020. Using optimal embeddings to learn new intents with few examples: An application in the insurance domain. In KDD 2020 Workshop on Conversational Systems Towards Mainstream Adoption(KDD Converse 2020). CEUR-WS.org.
  2. Intent disambiguation for task-oriented dialogue systems. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 5079–5080.
  3. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning, 16(4):494–591.
  4. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations.
  5. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38–45.
  6. Improving out-of-scope detection in intent classification by using embeddings of the word graph space of the classes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3952–3961.
  7. Reinforcement learning for personalization: A systematic literature review. Data Science, 3(2):107–147.
  8. Reinforcement learning for personalized dialogue management. In IEEE/WIC/ACM International Conference on Web Intelligence, pages 59–67.
  9. David DeVault and Matthew Stone. 2007. Managing ambiguities across utterances in dialogue. In Proceedings of the 11th Workshop on the Semantics and Pragmatics of Dialogue (Decalog 2007), pages 49–56.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  11. Kaustubh D Dhole. 2020. Resolving intent ambiguities by retrieving discriminative clarifying questions. arXiv preprint arXiv:2008.07559.
  12. Conformal prediction sets with limited false positives. In International Conference on Machine Learning, pages 6514–6532. PMLR.
  13. Edwin Fong and Chris C Holmes. 2021. Conformal bayesian computation. Advances in Neural Information Processing Systems, 34:18268–18279.
  14. Patrizio Giovannotti and Alex Gammerman. 2021. Transformer-based conformal predictors for paraphrase detection. In Conformal and Probabilistic Prediction and Applications, pages 243–265. PMLR.
  15. The atis spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
  16. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2744–2751.
  17. Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. ACM Computing Surveys, 55(6):1–40.
  18. CLAM: Selective clarification for ambiguous questions with large language models. In ICML Workshop Challenges of Deploying Generative AI.
  19. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311–1316, Hong Kong, China. Association for Computational Linguistics.
  20. Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5491–5496.
  21. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Systems, pages 165–183. Springer.
  22. Bert-based conformal predictor for sentiment analysis. In Conformal and Probabilistic Prediction and Applications, pages 269–284. PMLR.
  23. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81.
  24. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13, pages 345–356. Springer.
  25. Cognitive load theory. Cambridge University Press, New York, NY, US.
  26. John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61–74.
  27. On the means for clarification in dialogue. Current and new directions in discourse and dialogue, pages 235–255.
  28. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581–3591.
  29. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223–234.
  30. Cross-lingual transfer learning for multilingual task oriented dialog. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3795–3805.
  31. Glenn Shafer and Vladimir Vovk. 2008. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3).
  32. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2911–2916.
  33. Understanding user satisfaction with task-oriented dialogue systems. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2018–2023.
  34. Collecting high-quality dialogue user satisfaction ratings with third-party annotators. In Proceedings of the 2020 Conference on Human Information Interaction and Retrieval, pages 363–367.
  35. Machine-learning applications of algorithmic randomness. In Proceedings of the Sixteenth International Conference on Machine Learning, pages 444–453.
  36. Unknown intent detection using gaussian mixture model with an application to zero-shot intent classification. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 1050–1060.
  37. Eyup Halit Yilmaz and Cagri Toraman. 2020. Kloos: Kl divergence-based out-of-scope intent detection in human-to-machine conversations. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 2105–2108.
  38. Cecilia Ying and Stephen Thomas. 2022. Label errors in banking77. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 139–143.
  39. Generating clarifying questions for information retrieval. In Proceedings of the web conference 2020, pages 418–428.
  40. Out-of-scope intent detection with self-supervision and discriminative training. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3521–3532.
  41. Zhiling Zhang and Kenny Zhu. 2021. Diverse and specific clarification question generation with keywords. In Proceedings of the Web Conference 2021, pages 3501–3511.
  42. Contrastive out-of-distribution detection for pretrained transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP).
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Floris den Hengst (7 papers)
  2. Ralf Wolter (1 paper)
  3. Patrick Altmeyer (5 papers)
  4. Arda Kaygan (1 paper)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now