Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NIR-Prompt: A Multi-task Generalized Neural Information Retrieval Training Framework (2212.00229v3)

Published 1 Dec 2022 in cs.IR

Abstract: Information retrieval aims to find information that meets users' needs from the corpus. Different needs correspond to different IR tasks such as document retrieval, open-domain question answering, retrieval-based dialogue, etc., while they share the same schema to estimate the relationship between texts. It indicates that a good IR model can generalize to different tasks and domains. However, previous studies indicate that state-of-the-art neural information retrieval (NIR) models, e.g, pre-trained LLMs (PLMs) are hard to generalize. Mainly because the end-to-end fine-tuning paradigm makes the model overemphasize task-specific signals and domain biases but loses the ability to capture generalized essential signals. To address this problem, we propose a novel NIR training framework named NIR-Prompt for retrieval and reranking stages based on the idea of decoupling signal capturing and combination. NIR-Prompt exploits Essential Matching Module (EMM) to capture the essential matching signals and gets the description of tasks by Matching Description Module (MDM). The description is used as task-adaptation information to combine the essential matching signals to adapt to different tasks. Experiments under in-domain multi-task, out-of-domain multi-task, and new task adaptation settings show that NIR-Prompt can improve the generalization of PLMs in NIR for both retrieval and reranking stages compared with baselines.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (87)
  1. A convergence theory for deep learning via over-parameterization. In International conference on machine learning. PMLR, 242–252.
  2. ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning. CoRR abs/2111.10952 (2021).
  3. ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 6655–6672. https://aclanthology.org/2022.emnlp-main.446
  4. Petr Baudis and Jan Sedivý. 2015. Modeling of the Question Answering Task in the YodaQA System. In Proceedings of the 2015 Conference on CLEF (Lecture Notes in Computer Science, Vol. 9283). Springer, 222–228.
  5. Semantic Parsing on Freebase from Question-Answer Pairs. In Proceedings of the 2013 Conference on EMNLP. ACL, 1533–1544.
  6. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on EMNLP. The Association for Computational Linguistics, 632–642.
  7. Description Based Text Classification with Reinforcement Learning. In Proceedings of the 2020 Conference on ICML (Proceedings of Machine Learning Research, Vol. 119). PMLR, 1371–1382.
  8. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the Conference on ACL 2017. Association for Computational Linguistics, 1870–1879.
  9. Cross-Lingual Passage Re-Ranking With Alignment Augmented Multilingual BERT. IEEE Access 8 (2020), 213232–213243.
  10. Enhanced LSTM for Natural Language Inference. In Proceedings of the 2017 Conference on ACL. Association for Computational Linguistics, 1657–1668.
  11. Dipanjan Das and Noah A. Smith. 2009. Paraphrase Identification as Probabilistic Quasi-Synchronous Recognition. In Proceedings of the Conference on ACL 2009. The Association for Computer Linguistics, 468–476.
  12. Unsupervised construction of large paraphrase corpora: exploiting massively parallel news sources. COLING (2004).
  13. William B. Dolan and Chris Brockett. 2005. Automatically Constructing a Corpus of Sentential Paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Natural Language Processing.
  14. A Linguistic Study on Relevance Modeling in Information Retrieval. In Proceedings of the 2021 Conference on WWW. ACM / IW3C2, 1053–1064.
  15. Pre-training Methods in Information Retrieval. CoRR abs/2111.13853 (2021). arXiv:2111.13853 https://arxiv.org/abs/2111.13853
  16. Luyu Gao and Jamie Callan. 2021. Condenser: a Pre-training Architecture for Dense Retrieval. In Proceedings of the 2021 Conference on EMNLP, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 981–993.
  17. PPT: Pre-trained Prompt Tuning for Few-shot Learning. CoRR abs/2109.04332 (2021). https://arxiv.org/abs/2109.04332
  18. A Deep Relevance Matching Model for Ad-hoc Retrieval. In Proceedings of the Conference on CIKM 2016. ACM, 55–64.
  19. AmazonQA: A Review-Based Question Answering Task. In Proceedings of the Conference on IJCAI 2019. ijcai.org, 4996–5002.
  20. PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge. In Proceedings of the 2020 Conference on EMNLP, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 7572–7582.
  21. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In Proceedings of the 2021 Conference on SIGIR. ACM, 113–122. https://doi.org/10.1145/3404835.3462891
  22. Convolutional Neural Network Architectures for Matching Natural Language Sentences. In Proceedings of the 2014 Conference on NIPS. 2042–2050.
  23. How Can We Know What Language Models Know. Trans. Assoc. Comput. Linguistics 8 (2020), 423–438.
  24. Billion-Scale Similarity Search with GPUs. IEEE Trans. Big Data 7, 3 (2021), 535–547.
  25. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 2017 Conference on ACL. Association for Computational Linguistics, 1601–1611.
  26. Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Online, 565–576. https://doi.org/10.18653/v1/2021.acl-long.47
  27. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on EMNLP. Association for Computational Linguistics, 6769–6781.
  28. UnifiedQA: Crossing Format Boundaries With a Single QA System. In Proceedings of the 2020 Conference on EMNLP Findings (Findings of ACL, Vol. EMNLP 2020). Association for Computational Linguistics, 1896–1907.
  29. Natural Questions: a Benchmark for Question Answering Research. Trans. Assoc. Comput. Linguistics 7 (2019), 452–466.
  30. A Continuously Growing Dataset of Sentential Paraphrases. In Proceedings of the 2017 Conference on EMNLP. Association for Computational Linguistics, 1224–1234.
  31. Contextualized Embeddings based Transformer Encoder for Sentence Similarity Modeling in Answer Selection Task. In Proceedings of the Conference on LREC 2019. European Language Resources Association, 5505–5514.
  32. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In Proceedings of the Conference on ACL 2019. Association for Computational Linguistics, 6086–6096.
  33. A Unified MRC Framework for Named Entity Recognition. In Proceedings of the 2020 Conference on ACL. Association for Computational Linguistics, 5849–5859.
  34. Xiang Lisa Li and Percy Liang. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 2021 Conference on ACL. Association for Computational Linguistics, 4582–4597.
  35. DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In Proceedings of the 2017 Conference on IJCNLP. Asian Federation of Natural Language Processing, 986–995.
  36. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. CoRR abs/2107.13586 (2021).
  37. Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of the 2019 Conference on ACL. Association for Computational Linguistics, 4487–4496.
  38. GPT Understands, Too. CoRR abs/2103.10385 (2021). arXiv:2103.10385 https://arxiv.org/abs/2103.10385
  39. Zhengdong Lu and Hang Li. 2013. A Deep Architecture for Matching Short Texts. In Proceedings of the 2013 Conference on NIPS. 1367–1375.
  40. PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval. In Proceedings of the 2021 Conference on WSDM. ACM, 283–291.
  41. CEDR: Contextualized Embeddings for Document Ranking. In Proceedings of the Conference on SIGIR 2019. ACM, 1101–1104.
  42. Multi-Task Retrieval for Knowledge-Intensive Tasks. In Proceedings of the 2021 Conference on ACL. Association for Computational Linguistics, 1098–1111.
  43. SemEval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences through Semantic Relatedness and Textual Entailment. SemEval@COLING (2014).
  44. The Natural Language Decathlon: Multitask Learning as Question Answering. CoRR abs/1806.08730 (2018).
  45. Learning to Match using Local and Distributed Representations of Text for Web Search. In Proceedings of the 2017 Conference on WWW. ACM, 1291–1299.
  46. Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching. In Proceedings of the 2021 Conference on CIKM. ACM, 1396–1405.
  47. Text Matching as Image Recognition. In Proceedings of the 2016 Conference on AAAI. AAAI Press, 2793–2799.
  48. DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval. In Proceedings of the Conference on CIKM 2017. ACM, 257–266.
  49. KILT: a Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference on NAACL. Association for Computational Linguistics, 2523–2544.
  50. Language Models as Knowledge Bases?. In Proceedings of the Conference on EMNLP 2019. Association for Computational Linguistics, 2463–2473.
  51. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, 487–503. https://doi.org/10.18653/v1/2021.eacl-main.39
  52. Raul Puri and Bryan Catanzaro. 2019. Zero-shot Text Classification With Generative Language Models. CoRR abs/1912.10165 (2019). http://arxiv.org/abs/1912.10165
  53. LETOR: A benchmark collection for research on learning to rank for information retrieval. Inf. Retr. 13, 4 (2010), 346–374.
  54. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res. 21 (2020), 140:1–140:67. http://jmlr.org/papers/v21/20-074.html
  55. A Thorough Examination on Zero-shot Dense Retrieval. CoRR abs/2204.12755 (2022).
  56. Large Test Collection Experiments on an Operational, Interactive System: Okapi at TREC. Inf. Process. Manag. 31, 3 (1995), 345–360.
  57. Multitask Prompted Training Enables Zero-Shot Task Generalization. CoRR abs/2110.08207 (2021).
  58. Timo Schick and Hinrich Schütze. 2021a. Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In Proceedings of the 2021 Conference on EACL. Association for Computational Linguistics, 255–269.
  59. Timo Schick and Hinrich Schütze. 2021b. Few-Shot Text Generation with Natural Language Instructions. In Proceedings of the 2021 Conference on EMNLP. Association for Computational Linguistics, 390–402.
  60. Simple Entity-Centric Questions Challenge Dense Retrievers. In Proceedings of the 2021 Conference on ACL. Association for Computational Linguistics, 6138–6148. https://doi.org/10.18653/v1/2021.emnlp-main.496
  61. A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval. In Proceedings of the 2014 Conference on CIKM. ACM, 101–110.
  62. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on EMNLP. Association for Computational Linguistics, 4222–4235.
  63. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In Proceedings of the 2011 Conference on ICML. Omnipress, 129–136.
  64. On Transferability of Prompt Tuning for Natural Language Processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 3949–3969. https://doi.org/10.18653/v1/2022.naacl-main.290
  65. Alon Talmor and Jonathan Berant. 2019. MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension. In Proceedings of the 2019 Conference on ACL. Association for Computational Linguistics, 4911–4921.
  66. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Proceedings of the 2021 Conference on NeurIPS Datasets and Benchmarks. https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/65b9eea6e1cc6bb9f0cd2a47751a186f-Abstract-round2.html
  67. Attention is All you Need. In Proceedings of the 2017 Conference on NIPS. 5998–6008.
  68. Ellen M. Voorhees. 2004. Overview of the TREC 2004 Robust Track. In Proceedings of the Conference on TREC 2004 (NIST Special Publication, Vol. 500-261). National Institute of Standards and Technology (NIST).
  69. SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer. CoRR abs/2110.07904 (2021). https://arxiv.org/abs/2110.07904
  70. Exploring and Predicting Transferability across NLP Tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 7882–7926. https://doi.org/10.18653/v1/2020.emnlp-main.635
  71. Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN. In Proceedings of the 2016 Conference on IJCAI. IJCAI/AAAI Press, 2922–2928.
  72. Finetuned Language Models Are Zero-Shot Learners. CoRR abs/2109.01652 (2021).
  73. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference on NAACL. Association for Computational Linguistics.
  74. Using Prior Knowledge to Guide BERT’s Attention in Semantic Textual Matching Tasks. In Proceedings of the Conference on WWW 2021. ACM / IW3C2, 2466–2475.
  75. RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 538–548. https://aclanthology.org/2022.emnlp-main.35
  76. Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 4008–4020.
  77. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In Proceedings of the 2021 Conference on ICLR. OpenReview.net.
  78. ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization. CoRR abs/2201.06910 (2022). https://arxiv.org/abs/2201.06910
  79. Match-Prompt: Improving Multi-Task Generalization Ability for Neural Text Matching via Prompt Learning. In Proceedings of the 2022 Conference on CIKM (Atlanta, GA, USA) (CIKM ’22). Association for Computing Machinery, New York, NY, USA, 2290–2300. https://doi.org/10.1145/3511808.3557388
  80. Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. In Proceedings of the 2016 Conference on SIGIR. ACM, 55–64.
  81. Simple and Effective Text Matching with Richer Alignment Features. In Proceedings of the Conference on ACL 2019. Association for Computational Linguistics, 4699–4709.
  82. A Deep Top-K Relevance Matching Model for Ad-hoc Retrieval. In Proceedings of the 2018 Conference on CCIR (Lecture Notes in Computer Science, Vol. 11168). Springer, 16–27.
  83. Learning Discriminative Projections for Text Similarity Measures. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning, CoNLL 2011, Portland, Oregon, USA, June 23-24, 2011. ACL, 247–256.
  84. Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval. CoRR abs/2208.05753 (2022).
  85. Adversarial Retriever-Ranker for Dense Text Retrieval. In Proceedings of the 2022 Conference on ICLR. OpenReview.net. https://openreview.net/forum?id=MR7XubKUFB
  86. PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation. arXiv:2208.10160 [cs.CL]
  87. Adaptive Information Seeking for Open-Domain Question Answering. In Proceedings of the 2021 Conference on EMNLP. Association for Computational Linguistics, 3615–3626. https://doi.org/10.18653/v1/2021.emnlp-main.293
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Shicheng Xu (36 papers)
  2. Liang Pang (94 papers)
  3. Huawei Shen (119 papers)
  4. Xueqi Cheng (274 papers)
Citations (4)
View on Semantic Scholar