Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

FecTek: Enhancing Term Weight in Lexicon-Based Retrieval with Feature Context and Term-level Knowledge (2404.12152v1)

Published 18 Apr 2024 in cs.CL

Abstract: Lexicon-based retrieval has gained siginificant popularity in text retrieval due to its efficient and robust performance. To further enhance performance of lexicon-based retrieval, researchers have been diligently incorporating state-of-the-art methodologies like Neural retrieval and text-level contrastive learning approaches. Nonetheless, despite the promising outcomes, current lexicon-based retrieval methods have received limited attention in exploring the potential benefits of feature context representations and term-level knowledge guidance. In this paper, we introduce an innovative method by introducing FEature Context and TErm-level Knowledge modules(FecTek). To effectively enrich the feature context representations of term weight, the Feature Context Module (FCM) is introduced, which leverages the power of BERT's representation to determine dynamic weights for each element in the embedding. Additionally, we develop a term-level knowledge guidance module (TKGM) for effectively utilizing term-level knowledge to intelligently guide the modeling process of term weight. Evaluation of the proposed method on MS Marco benchmark demonstrates its superiority over the previous state-of-the-art approaches.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (31)
  1. S. Zhuang and G. Zuccon, “Fast passage re-ranking with contextualized exact term matching and efficient passage expansion,” arXiv preprint arXiv:2108.08513, 2021.
  2. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  3. Z. Dai and J. Callan, “Context-aware sentence/passage term importance estimation for first stage retrieval,” arXiv preprint arXiv:1910.10687, 2019.
  4. E. Choi, S. Lee, M. Choi, H. Ko, Y.-I. Song, and J. Lee, “Spade: Improving sparse representations using a dual document encoder for first-stage retrieval,” in Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 272–282, 2022.
  5. T. Formal, C. Lassance, B. Piwowarski, and S. Clinchant, “Splade v2: Sparse lexical and expansion model for information retrieval,” arXiv preprint arXiv:2109.10086, 2021.
  6. A. Mallia, O. Khattab, T. Suel, and N. Tonellotto, “Learning passage impacts for inverted indexes,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1723–1727, 2021.
  7. S. Zhuang and G. Zuccon, “Tilde: Term independent likelihood model for passage re-ranking,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1483–1492, 2021.
  8. Y. Bai, X. Li, G. Wang, C. Zhang, L. Shang, J. Xu, Z. Wang, F. Wang, and Q. Liu, “Sparterm: Learning term-based sparse representation for fast text retrieval,” arXiv preprint arXiv:2010.00768, 2020.
  9. J. Lin and X. Ma, “A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques,” arXiv preprint arXiv:2106.14807, 2021.
  10. J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141, 2018.
  11. L. Xiong, C. Xiong, Y. Li, K.-F. Tang, J. Liu, P. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” arXiv preprint arXiv:2007.00808, 2020.
  12. J. Zhan, J. Mao, Y. Liu, J. Guo, M. Zhang, and S. Ma, “Optimizing dense retrieval model training with hard negatives,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1503–1512, 2021.
  13. L. Gao and J. Callan, “Condenser: a pre-training architecture for dense retrieval,” arXiv preprint arXiv:2104.08253, 2021.
  14. L. Gao and J. Callan, “Unsupervised corpus aware language model pre-training for dense passage retrieval,” arXiv preprint arXiv:2108.05540, 2021.
  15. O. Khattab and M. Zaharia, “Colbert: Efficient and effective passage search via contextualized late interaction over bert,” in Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 39–48, 2020.
  16. L. Gao, Z. Dai, and J. Callan, “Coil: Revisit exact lexical match in information retrieval with contextualized inverted list,” arXiv preprint arXiv:2104.07186, 2021.
  17. A. Kazemi, V. Pérez-Rosas, and R. Mihalcea, “Biased textrank: Unsupervised graph-based content extraction,” arXiv preprint arXiv:2011. 01026, 2020.
  18. T. Formal, C. Lassance, B. Piwowarski, and S. Clinchant, “From distillation to hard negative sampling: Making sparse neural ir models more effective,” in Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2353–2359, 2022.
  19. R. Ren, Y. Qu, J. Liu, W. X. Zhao, Q. She, H. Wu, H. Wang, and J.-R. Wen, “Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking,” arXiv preprint arXiv:2110.07367, 2021.
  20. H. Zhang, Y. Gong, Y. Shen, J. Lv, N. Duan, and W. Chen, “Adversarial retriever-ranker for dense text retrieval,” arXiv preprint arXiv:2110.03611, 2021.
  21. S. Hofstätter, S.-C. Lin, J.-H. Yang, J. Lin, and A. Hanbury, “Efficiently teaching an effective dense retriever with balanced topic aware sampling,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 113–122, 2021.
  22. S.-C. Lin, J.-H. Yang, and J. Lin, “In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval,” in Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pp. 163–173, 2021.
  23. K. Santhanam, O. Khattab, J. Saad-Falcon, C. Potts, and M. Zaharia, “Colbertv2: Effective and efficient retrieval via lightweight late interaction,” arXiv preprint arXiv:2112.01488, 2021.
  24. R. Ren, S. Lv, Y. Qu, J. Liu, W. X. Zhao, Q. She, H. Wu, H. Wang, and J.-R. Wen, “Pair: Leveraging passage-centric similarity relation for improving dense passage retrieval,” arXiv preprint arXiv:2108.06027, 2021.
  25. Y. Qu, Y. Ding, J. Liu, K. Liu, R. Ren, W. X. Zhao, D. Dong, H. Wu, and H. Wang, “Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2010.08191, 2020.
  26. K. Zhang, C. Tao, T. Shen, C. Xu, X. Geng, B. Jiao, and D. Jiang, “Led: Lexicon-enlightened dense retriever for large-scale retrieval,” in Proceedings of the ACM Web Conference 2023, pp. 3203–3213, 2023.
  27. Z. Fan, L. Gao, R. Jha, and J. Callan, “Coilcr: Efficient semantic matching in contextualized exact match retrieval,” in European Conference on Information Retrieval, pp. 298–312, Springer, 2023.
  28. L. Gao, Z. Dai, T. Chen, Z. Fan, B. Van Durme, and J. Callan, “Complement lexical retrieval model with semantic residual embeddings,” in Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part I 43, pp. 146–160, Springer, 2021.
  29. J. Zhan, J. Mao, Y. Liu, J. Guo, M. Zhang, and S. Ma, “Learning discrete representations via constrained clustering for effective and efficient dense retrieval,” in Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pp. 1328–1336, 2022.
  30. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “Ms marco: A human generated machine reading comprehension dataset,” choice, vol. 2640, p. 660, 2016.
  31. R. Nogueira, J. Lin, and A. Epistemic, “From doc2query to doctttttquery,” Online preprint, vol. 6, p. 2, 2019.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now