Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An In-depth Analysis of Passage-Level Label Transfer for Contextual Document Ranking (2103.16669v3)

Published 30 Mar 2021 in cs.IR

Abstract: Pre-trained contextual LLMs such as BERT, GPT, and XLnet work quite well for document retrieval tasks. Such models are fine-tuned based on the query-document/query-passage level relevance labels to capture the ranking signals. However, the documents are longer than the passages and such document ranking models suffer from the token limitation (512) of BERT. Researchers proposed ranking strategies that either truncate the documents beyond the token limit or chunk the documents into units that can fit into the BERT. In the later case, the relevance labels are either directly transferred from the original query-document pair or learned through some external model. In this paper, we conduct a detailed study of the design decisions about splitting and label transfer on retrieval effectiveness and efficiency. We find that direct transfer of relevance labels from documents to passages introduces label noise that strongly affects retrieval effectiveness for large training datasets. We also find that query processing times are adversely affected by fine-grained splitting schemes. As a remedy, we propose a careful passage level labelling scheme using weak supervision that delivers improved performance (3-14% in terms of nDCG score) over most of the recently proposed models for ad-hoc retrieval while maintaining manageable computational complexity on four diverse document retrieval datasets.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (81)
  1. Reqa: An evaluation for end-to-end answer retrieval models. arXiv preprint arXiv:1907.04780, 2019.
  2. Cross-domain modeling of sentence-level evidence for document retrieval. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3481–3487, November 2019.
  3. Data augmentation for sample efficient and robust document ranking. 2023.
  4. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020.
  5. Spade: Improving sparse representations using a dual document encoder for first-stage retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 272–282, 2022.
  6. SDR: Efficient neural re-ranking using succinct document representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6624–6637, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.457. URL https://aclanthology.org/2022.acl-long.457.
  7. An experimental comparison of click position-bias models. In Proceedings of the 2008 international conference on web search and data mining, pages 87–94, 2008.
  8. TREC-2019-Deep-Learning. https://microsoft.github.io/TREC-2019-Deep-Learning/, 2019.
  9. Deeper text understanding for ir with contextual neural language modeling. In ACM SIGIR’19, pages 985–988, 2019.
  10. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining, WSDM ’18, pages 126–134. ACM, 2018. ISBN 978-1-4503-5581-0. doi: 10.1145/3159652.3159659. URL http://doi.acm.org/10.1145/3159652.3159659.
  11. Neural ranking models with weak supervision. In SIGIR ’17, pages 65–74. ACM, 2017. ISBN 978-1-4503-5022-8. doi: 10.1145/3077136.3080832. URL http://doi.acm.org/10.1145/3077136.3080832.
  12. Fidelity-weighted learning. In ICLR ’18, 2018. URL https://openreview.net/forum?id=B1X0mzZCW.
  13. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805.
  14. Modeling diverse relevance patterns in ad-hoc retrieval. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, pages 375–384. ACM, 2018a. ISBN 978-1-4503-5657-2. doi: 10.1145/3209978.3209980. URL http://doi.acm.org/10.1145/3209978.3209980.
  15. Modeling diverse relevance patterns in ad-hoc retrieval. In ACM SIGIR’18, pages 375–384, 2018b.
  16. Coilcr: Efficient semantic matching in contextualized exact match retrieval. In Jaap Kamps, Lorraine Goeuriot, Fabio Crestani, Maria Maistro, Hideo Joho, Brian Davis, Cathal Gurrin, Udo Kruschwitz, and Annalina Caputo, editors, Advances in Information Retrieval, pages 298–312, Cham, 2023. Springer Nature Switzerland.
  17. Luke Gallagher. Pairwise t-test on TREC Run Files. https://github.com/lgrz/pairwise-ttest/, 2019.
  18. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3030–3042, Online, June 2021a. Association for Computational Linguistics.
  19. Complement lexical retrieval model with semantic residual embeddings. In Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani, editors, Advances in Information Retrieval, pages 146–160, Cham, 2021b. Springer International Publishing.
  20. A deep relevance matching model for ad-hoc retrieval. In CIKM’16, pages 55–64. ACM, 2016. ISBN 978-1-4503-4073-1. doi: 10.1145/2983323.2983769. URL http://doi.acm.org/10.1145/2983323.2983769.
  21. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  22. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  23. Let’s measure run time! extending the ir replicability infrastructure to include performance aspects. arXiv preprint arXiv:1907.04614, 2019.
  24. Local self-attention over long text for efficient document retrieval. arXiv preprint arXiv:2005.04908, 2020a.
  25. Interpretable & time-budget-constrained contextualization for re-ranking. arXiv preprint arXiv:2002.01854, 2020b.
  26. Introducing neural bag of whole-words with colberter: Contextualized late interactions using enhanced reduction. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 737–747, 2022.
  27. Convolutional neural network architectures for matching natural language sentences. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 2042–2050, 2014.
  28. Learning deep structured semantic models for web search using clickthrough data. In CIKM ’13, pages 2333–2338. ACM, 2013. ISBN 978-1-4503-2263-8. doi: 10.1145/2505515.2505665. URL http://doi.acm.org/10.1145/2505515.2505665.
  29. PACRR: A position-aware neural ir model for relevance matching. In EMNLP ’17, pages 1049–1058, Copenhagen, Denmark, September 2017. URL https://www.aclweb.org/anthology/D17-1110.
  30. Co-PACRR: A context-aware neural ir model for ad-hoc retrieval. In WSDM ’18, pages 279–287. ACM, 2018. ISBN 978-1-4503-5581-0. doi: 10.1145/3159652.3159689. URL http://doi.acm.org/10.1145/3159652.3159689.
  31. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422–446, 2002.
  32. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020.
  33. Implicit feedback for inferring user preference: a bibliography. In Acm Sigir Forum, volume 37, pages 18–28. ACM New York, NY, USA, 2003.
  34. Colbert: Efficient and effective passage search via contextualized late interaction over bert. 2020.
  35. Relevance based language models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’01, pages 120–127, New York, NY, USA, 2001. ACM. ISBN 1-58113-331-6. doi: 10.1145/383952.383972. URL http://doi.acm.org/10.1145/383952.383972.
  36. Relevance-based language models. In ACM SIGIR Forum, volume 51, pages 260–267. ACM, 2017.
  37. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300, 2019.
  38. Efficient neural ranking using forward indexes. In Proceedings of the ACM Web Conference 2022, WWW ’22, page 266–276, 2022.
  39. Efficient neural ranking using forward indexes and lightweight encoders. ACM Trans. Inf. Syst., nov 2023a. ISSN 1046-8188. URL https://doi.org/10.1145/3631939. Just Accepted.
  40. Extractive explanations for interpretable text ranking. ACM Trans. Inf. Syst., 41(4), mar 2023b. doi: 10.1145/3576924.
  41. Parade: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093, 2020.
  42. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181, 2020.
  43. Contextualized word representations for document re-ranking. arXiv preprint arXiv:1904.07094, 2019.
  44. High accuracy retrieval with multiple nested ranker. In SIGIR ’06, pages 437–444. ACM, 2006. ISBN 1-59593-369-7. doi: 10.1145/1148170.1148246. URL http://doi.acm.org/10.1145/1148170.1148246.
  45. Deep relevance ranking using enhanced document-query interactions. In EMNLP ’18, pages 1849–1860. ACL, 2018. URL http://aclweb.org/anthology/D18-1211.
  46. A dual embedding space model for document ranking. arXiv preprint, arXiv:1602.01137, 2016. URL http://arxiv.org/abs/1602.01137.
  47. Learning to match using local and distributed representations of text for web search. In WWW’17, pages 1291–1299, 2017. ISBN 978-1-4503-4913-0. doi: 10.1145/3038912.3052579. URL https://doi.org/10.1145/3038912.3052579.
  48. Siamese recurrent architectures for learning sentence similarity. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 2786–2792, 2016.
  49. Improving document ranking with dual word embeddings. In WWW ’16 Companion, pages 83–84, 2016. ISBN 978-1-4503-4144-8. doi: 10.1145/2872518.2889361. URL https://doi.org/10.1145/2872518.2889361.
  50. Empirical study of multi-level convolution models for ir based on representations and interactions. In ICTIR ’18, pages 59–66. ACM, 2018a. ISBN 978-1-4503-5656-5. doi: 10.1145/3234944.3234954. URL http://doi.acm.org/10.1145/3234944.3234954.
  51. Multi-level abstraction convolutional model with weak supervision for information retrieval. In SIGIR ’18, pages 985–988. ACM, 2018b. ISBN 978-1-4503-5657-2. doi: 10.1145/3209978.3210123. URL http://doi.acm.org/10.1145/3209978.3210123.
  52. Passage re-ranking with BERT. CoRR, abs/1901.04085, 2019. URL http://arxiv.org/abs/1901.04085.
  53. Multi-stage document ranking with bert, 2019.
  54. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 24(4):694–707, April 2016.
  55. A study of MatchPyramid models on ad-hoc retrieval. arXiv:1606.04648, 2016. URL http://arxiv.org/abs/1606.04648.
  56. DeepRank: A new deep architecture for relevance ranking in information retrieval. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, pages 257–266. ACM, 2017. ISBN 978-1-4503-4918-5. doi: 10.1145/3132847.3132914. URL http://doi.acm.org/10.1145/3132847.3132914.
  57. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, 2018.
  58. Convolutional neural tensor network architecture for community-based question answering. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI’15, page 1305–1311, 2015.
  59. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  60. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP (1), pages 3980–3990. Association for Computational Linguistics, 2019.
  61. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, April 2009.
  62. Distant supervision in bert-based adhoc document retrieval. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, CIKM ’20, page 2197–2200, 2020.
  63. A latent semantic model with convolutional-pooling structure for information retrieval. In CIKM ’14, pages 101–110. ACM, 2014a. ISBN 978-1-4503-2598-1. doi: 10.1145/2661829.2661935. URL http://doi.acm.org/10.1145/2661829.2661935.
  64. Learning semantic representations using convolutional neural networks for web search. In WWW ’14 Companion, pages 373–374. ACM, 2014b. ISBN 978-1-4503-2745-9. doi: 10.1145/2567948.2577348. URL http://doi.acm.org/10.1145/2567948.2577348.
  65. Indri: A language model-based search engine for complex queries. In Proceedings of the International Conference on Intelligent Analysis, volume 2, pages 2–6, 2005.
  66. Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080, 2014.
  67. Deeptilebars: Visualizing term distribution for neural information retrieval. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):289–296, 2019.
  68. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
  69. Learning from noisy large-scale datasets with minimal supervision. In IEEE CVPR’17, pages 839–847, 2017.
  70. A deep architecture for semantic matching with multiple positional sentence representations. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI’16, page 2835–2841, 2016.
  71. Comparing explicit and implicit feedback techniques for web retrieval: Trec-10 interactive track report. In Proceedings of the Tenth Text Retrieval Conference (TREC-10), pages 534–538, 2002.
  72. Investigating passage-level relevance and its role in document-level relevance judgment. SIGIR’19, page 605–614, 2019.
  73. Leveraging passage-level cumulative gain for document ranking. In Proceedings of The Web Conference 2020, page 2421–2431, 2020.
  74. Learning from massive noisy labeled data for image classification. In IEEE CVPR’15, pages 2691–2699, 2015.
  75. End-to-end neural ad-hoc ranking with kernel pooling. In SIGIR ’17, pages 55–64. ACM, 2017. ISBN 978-1-4503-5022-8. doi: 10.1145/3077136.3080809. URL http://doi.acm.org/10.1145/3077136.3080809.
  76. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020.
  77. Anmm: Ranking short answer texts with attention-based neural matching model. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, page 287–296, 2016.
  78. Simple applications of bert for ad hoc document retrieval. arXiv preprint arXiv:1903.10972, 2019.
  79. Selective weak supervision for neural information retrieval. In Proceedings of The Web Conference 2020, page 474–485, 2020.
  80. Fast passage re-ranking with contextualized exact term matching and efficient passage expansion. arXiv preprint arXiv:2108.08513, 2021a.
  81. Tilde: Term independent likelihood model for passage re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 1483–1492, 2021b.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Koustav Rudra (14 papers)
  2. Zeon Trevor Fernando (7 papers)
  3. Avishek Anand (80 papers)
Citations (6)
View on Semantic Scholar