Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Retrieval-Augmented Neural Response Generation Using Logical Reasoning and Relevance Scoring (2310.13566v1)

Published 20 Oct 2023 in cs.CL and cs.AI

Abstract: Constructing responses in task-oriented dialogue systems typically relies on information sources such the current dialogue state or external databases. This paper presents a novel approach to knowledge-grounded response generation that combines retrieval-augmented LLMs with logical reasoning. The approach revolves around a knowledge graph representing the current dialogue state and background information, and proceeds in three steps. The knowledge graph is first enriched with logically derived facts inferred using probabilistic logical programming. A neural model is then employed at each turn to score the conversational relevance of each node and edge of this extended graph. Finally, the elements with highest relevance scores are converted to a natural language form, and are integrated into the prompt for the neural conversational model employed to generate the system response. We investigate the benefits of the proposed approach on two datasets (KVRET and GraphWOZ) along with a human evaluation. Experimental results show that the combination of (probabilistic) logical reasoning with conversational relevance scoring does increase both the factuality and fluency of the responses.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Retrieval-augmented transformer-xl for close-domain dialog generation. In The International FLAIRS Conference Proceedings, volume 34, pages 1–4. Florida Online Journals.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712.
  4. Retrieval-guided dialogue response generation via a matching-to-generation framework. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1866–1875, Hong Kong, China. Association for Computational Linguistics.
  5. Mark Chavira and Adnan Darwiche. 2008. On probabilistic inference by weighted model counting. Artificial Intelligence, 172(6-7):772–799.
  6. KETOD: Knowledge-enriched task-oriented dialogue. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2581–2593, Seattle, United States. Association for Computational Linguistics.
  7. Wizard of wikipedia: Knowledge-powered conversational agents. In International Conference on Learning Representations.
  8. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654.
  9. Key-value retrieval networks for task-oriented dialogue. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37–49, Saarbrücken, Germany. Association for Computational Linguistics.
  10. Inference and learning in probabilistic logic programs using weighted boolean formulas. Theory and Practice of Logic Programming, 15(3):358–401.
  11. A knowledge-grounded neural conversation model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
  12. Learning the parameters of probabilistic logic programs from interpretations. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part I 11, pages 581–596. Springer.
  13. Learning to select external knowledge with multi-scale negative sampling. arXiv preprint arXiv:2102.02096.
  14. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  15. Sequential latent knowledge selection for knowledge-grounded dialogue. In International Conference on Learning Representations.
  16. On the implementation of the probabilistic logic programming language problog. Theory and Practice of Logic Programming, 11(2-3):235–262.
  17. Hayoung Lee and Okran Jeong. 2023. A knowledge-grounded task-oriented dialogue system with hierarchical structure for enhancing knowledge selection. Sensors, 23(2):685.
  18. Factuality enhanced language models for open-ended text generation. Advances in Neural Information Processing Systems, 35:34586–34599.
  19. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.
  20. Zero-resource knowledge-grounded dialogue generation. Advances in Neural Information Processing Systems, 33:8475–8485.
  21. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304.03439.
  22. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  23. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1782–1792, Hong Kong, China. Association for Computational Linguistics.
  24. Mem2Seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1468–1478, Melbourne, Australia. Association for Computational Linguistics.
  25. OpenDialKG: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 845–854, Florence, Italy. Association for Computational Linguistics.
  26. Hindsight: Posterior-guided training of retrievers for improved open-ended generation. In International Conference on Learning Representations.
  27. Prasanna Parthasarathi and Joelle Pineau. 2018. Extending neural generative conversational model using external knowledge sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 690–695, Brussels, Belgium. Association for Computational Linguistics.
  28. Godel: Large-scale pre-training for goal-directed dialog. arXiv preprint arXiv:2206.11309.
  29. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813.
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  31. Thinking globally, acting locally: Distantly supervised global-to-local knowledge selection for background based conversation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8697–8704.
  32. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
  33. Large language models can be easily distracted by irrelevant context.
  34. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  35. Interaction strategies for an affective conversational agent. Presence, 20(5):395–411.
  36. Efficient retrieval augmented generation from unstructured knowledge for task-oriented dialog. arXiv preprint arXiv:2102.04643.
  37. Compiling probabilistic logic programs into sentential decision diagrams. In Proceedings Workshop on Probabilistic Logic Programming (PLP), pages 1–10.
  38. Graphwoz: Dialogue management with conversational knowledge graphs. arXiv preprint arXiv:2211.12852.
  39. Towards information-rich, logical dialogue systems with knowledge-enhanced neural models. Neurocomputing, 465:248–264.
  40. Retrieve and refine: Improved sequence generation models for dialogue. In Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI, pages 87–92, Brussels, Belgium. Association for Computational Linguistics.
  41. Augmenting end-to-end dialogue systems with commonsense knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  42. A survey of knowledge-enhanced text generation. ACM Computing Surveys, 54(11s):1–38.
  43. Retrieval augmentation for commonsense reasoning: A unified approach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4364–4377, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  44. CoLV: A collaborative latent variable model for knowledge-grounded dialogue generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2250–2261, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  45. Grounded conversation generation as guided traverses in commonsense knowledge graphs. arXiv preprint arXiv:1911.02707.
  46. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  47. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  48. Joint retrieval and generation training for grounded text generation. arXiv preprint arXiv:2105.06597.
  49. Knowledge-grounded dialogue generation with pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3377–3390, Online. Association for Computational Linguistics.
  50. Towards a unified multi-dimensional evaluator for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023–2038, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Nicholas Thomas Walker (3 papers)
  2. Stefan Ultes (32 papers)
  3. Pierre Lison (19 papers)
Citations (2)
View on Semantic Scholar