RDR: the Recap, Deliberate, and Respond Method for Enhanced Language Understanding (2312.09932v2)
Abstract: Natural language understanding (NLU) using neural network pipelines often requires additional context that is not solely present in the input data. Through Prior research, it has been evident that NLU benchmarks are susceptible to manipulation by neural models, wherein these models exploit statistical artifacts within the encoded external knowledge to artificially inflate performance metrics for downstream tasks. Our proposed approach, known as the Recap, Deliberate, and Respond (RDR) paradigm, addresses this issue by incorporating three distinct objectives within the neural network pipeline. Firstly, the Recap objective involves paraphrasing the input text using a paraphrasing model in order to summarize and encapsulate its essence. Secondly, the Deliberation objective entails encoding external graph information related to entities mentioned in the input text, utilizing a graph embedding model. Finally, the Respond objective employs a classification head model that utilizes representations from the Recap and Deliberation modules to generate the final prediction. By cascading these three models and minimizing a combined loss, we mitigate the potential for gaming the benchmark and establish a robust method for capturing the underlying semantic patterns, thus enabling accurate predictions. To evaluate the effectiveness of the RDR method, we conduct tests on multiple GLUE benchmark tasks. Our results demonstrate improved performance compared to competitive baselines, with an enhancement of up to 2\% on standard metrics. Furthermore, we analyze the observed evidence for semantic understanding exhibited by RDR models, emphasizing their ability to avoid gaming the benchmark and instead accurately capture the true underlying semantic patterns.
- Dbpedia: A nucleus for a web of open data. In The semantic web, 722–735. Springer.
- Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, 5185–5198.
- Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26.
- Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02922.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- Automatically Constructing a Corpus of Sentential Paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
- Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
- Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
- An introduction to the syntax and content of Cyc. UMBC Computer Science and Electrical Engineering Department Collection.
- Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007.
- Poliak, A. 2020. A survey on recognizing textual entailment as an NLP evaluation. arXiv preprint arXiv:2010.03061.
- TDLR: Top (Semantic)-Down (Syntactic) Language Representation. UMBC Faculty Collection.
- Knowledge Graph Guided Semantic Evaluation of Language Models For User Trust. arXiv preprint arXiv:2305.04989.
- ProKnow: Process knowledge for safety constrained and explainable question generation for mental health diagnostic assistance. Frontiers in big Data, 5: 1056728.
- Knowledge infused policy gradients with upper confidence bound for relational bandits. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part I 21, 35–50. Springer.
- Process Knowledge-infused Learning for Clinician-friendly Explanations. arXiv preprint arXiv:2306.09824.
- KSAT: Knowledge-infused Self Attention Transformer–Integrating Multiple Domain-Specific Contexts. arXiv preprint arXiv:2210.04307.
- Natural language understanding with the quora question pairs dataset. arXiv preprint arXiv:1907.01041.
- Knowledge-intensive language understanding for explainable AI. IEEE Internet Computing, 25(5): 19–24.
- Process Knowledge-Infused AI: Toward User-Level Explainability, Interpretability, and Safety. IEEE Internet Computing, 26(5): 76–84.
- Neurosymbolic Artificial Intelligence (Why, What, and How). IEEE Intelligent Systems, 38(3): 56–62.
- Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
- Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes. arXiv preprint arXiv:2306.01805.
- GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
- Knowledge-augmented methods for natural language processing. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, 1228–1231.
- IERL: Interpretable Ensemble Representation Learning–Combining CrowdSourced Knowledge and Distributed Semantic Representations. arXiv preprint arXiv:2306.13865.