Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Graph Guided Question Answer Generation for Procedural Question-Answering (2401.13594v1)

Published 24 Jan 2024 in cs.CL and cs.AI

Abstract: In this paper, we focus on task-specific question answering (QA). To this end, we introduce a method for generating exhaustive and high-quality training data, which allows us to train compact (e.g., run on a mobile device), task-specific QA models that are competitive against GPT variants. The key technological enabler is a novel mechanism for automatic question-answer generation from procedural text which can ingest large amounts of textual instructions and produce exhaustive in-domain QA training data. While current QA data generation methods can produce well-formed and varied data, their non-exhaustive nature is sub-optimal for training a QA model. In contrast, we leverage the highly structured aspect of procedural text and represent each step and the overall flow of the procedure as graphs. We then condition on graph nodes to automatically generate QA pairs in an exhaustive and controllable manner. Comprehensive evaluations of our method show that: 1) small models trained with our data achieve excellent performance on the target QA task, even exceeding that of GPT3 and ChatGPT despite being several orders of magnitude smaller. 2) semantic coverage is the key indicator for downstream QA performance. Crucially, while LLMs excel at syntactic diversity, this does not necessarily result in improvements on the end QA model. In contrast, the higher semantic coverage provided by our method is critical for QA performance.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Synthetic QA corpora generation with roundtrip consistency. In Proc. of the Annual Meeting of the Association for Computational Linguistics.
  2. Abstract meaning representation for sembanking. In Proc. of the Linguistic Annotation Workshop and Interoperability with Discourse.
  3. BBC. BBC good food. https://www.bbcgoodfood.com/about-bbc-good-food.
  4. Language models are few-shot learners. In Conference on Neural Information Processing systems.
  5. Learning to Ask More: Semi-Autoregressive Sequential Question Generation under Dual-Graph Interaction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 225–237, Online. Association for Computational Linguistics.
  6. Ying-Hong Chan and Yao-Chung Fan. 2019. A recurrent BERT-based model for question generation. In Workshop on Machine Reading for Question Answering.
  7. Aligning Actions Across Recipe Graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6930–6942, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  8. Searchqa: A new Q&A dataset augmented with context from a search engine. ArXiv, abs/1704.05179.
  9. Flow graph to video grounding for weakly-supervised multi-step localization. In Computer Vision – ECCV 2022, pages 319–335, Cham. Springer Nature Switzerland.
  10. Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4508–4513, Online. Association for Computational Linguistics.
  11. Two-Stage Synthesis Networks for Transfer Learning in Machine Comprehension. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 835–844, Copenhagen, Denmark. Association for Computational Linguistics.
  12. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations.
  13. Structural analysis of cooking preparation steps in japanese. In Proceedings of the Fifth International Workshop on on Information Retrieval with Asian Languages, IRAL ’00, page 157–164, New York, NY, USA. Association for Computing Machinery.
  14. DuReader: a Chinese machine reading comprehension dataset from real-world applications. In Proceedings of the Workshop on Machine Reading for Question Answering.
  15. Michael Heilman and Noah A. Smith. 2010. Good Question! Statistical Ranking for Question Generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 609–617, Los Angeles, California. Association for Computational Linguistics.
  16. Brad Jacob. 2020. AMRLib: A python library that makes AMR parsing, generation and visualization simple. https://github.com/bjascob/amrlib.
  17. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  18. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119, San Diego, California. Association for Computational Linguistics.
  19. Yoshio Momouchi. 1980. Control structures for actions in procedural texts and PT-chart. In Conference on Computational Linguistics.
  20. DocAMR: Multi-sentence AMR representation and evaluation.
  21. MS MARCO: A human generated machine reading comprehension dataset. In Conference on Neural Information Processing Systems - Workshops.
  22. Suraf Patil. 2020. Question generation using transformers. https://github.com/patil-suraj/question_generation.
  23. Designing multimodal datasets for nlp challenges. arXiv:2105.05999.
  24. Designing multimodal datasets for NLP challenges. arXiv:2105.05999.
  25. Asking It All: Generating Contextualized Questions for any Semantic Role. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1429–1441, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  26. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  27. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  28. Geetanjali Rakshit and Jeffrey Flanigan. 2021. ASQ: automatically generating question-answer pairs using AMRs. arXiv:2105.10023.
  29. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  30. The first question generation shared task evaluation challenge. In International Natural Language Generation Conference.
  31. Self-Attention Architectures for Answer-Agnostic Neural Question Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6027–6032, Florence, Italy. Association for Computational Linguistics.
  32. BLEURT: Learning Robust Metrics for Text Generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.
  33. A unified query-based generative model for question generation and question answering. arXiv:1709.01058.
  34. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
  35. Neural question generation with answer pivot. In AAAI Conference on Artificial Intelligence.
  36. Want to reduce labeling cost? GPT-3 can help. In Conference on Empirical Methods in Natural Language Processing.
  37. On decoding strategies for neural text generators. TACL.
  38. RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes. In Conference on Empirical Methods in Natural Language Processing.
  39. English Recipe Flow Graph Corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5187–5194, Marseille, France. European Language Resources Association.
  40. Just ask: Learning to answer questions from millions of narrated videos. In International Conference on Computer Vision.
  41. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  42. Selecting better samples from pre-trained LLMs: A case study on question generation. arXiv:2209.11000.
  43. Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3901–3910, Brussels, Belgium. Association for Computational Linguistics.
  44. Neural question generation from text: A preliminary study. In Natural Language Processing and Chinese Computing.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (9)
  1. Hai X. Pham (6 papers)
  2. Isma Hadji (14 papers)
  3. Xinnuo Xu (12 papers)
  4. Ziedune Degutyte (1 paper)
  5. Jay Rainey (1 paper)
  6. Evangelos Kazakos (13 papers)
  7. Afsaneh Fazly (9 papers)
  8. Georgios Tzimiropoulos (86 papers)
  9. Brais Martinez (38 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets