Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Science Checker Reloaded: A Bidirectional Paradigm for Transparency and Logical Reasoning (2402.13897v2)

Published 21 Feb 2024 in cs.IR, cs.AI, cs.CL, and cs.LG

Abstract: Information retrieval is a rapidly evolving field. However it still faces significant limitations in the scientific and industrial vast amounts of information, such as semantic divergence and vocabulary gaps in sparse retrieval, low precision and lack of interpretability in semantic search, or hallucination and outdated information in generative models. In this paper, we introduce a two-block approach to tackle these hurdles for long documents. The first block enhances language understanding in sparse retrieval by query expansion to retrieve relevant documents. The second block deepens the result by providing comprehensive and informative answers to the complex question using only the information spread in the long document, enabling bidirectional engagement. At various stages of the pipeline, intermediate results are presented to users to facilitate understanding of the system's reasoning. We believe this bidirectional approach brings significant advancements in terms of transparency, logical thinking, and comprehensive understanding in the field of scientific information retrieval.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. M. Luo, A. Mitra, T. Gokhale, and C. Baral, “Improving biomedical information retrieval with neural retrievers,” in AAAI Conference on Artificial Intelligence, 2022.
  2. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, mar 2023.
  3. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ArXiv, vol. abs/2311.05232, 2023.
  4. R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green ai,” Commun. ACM, vol. 63, no. 12, p. 54–63, nov 2020. [Online]. Available: https://doi.org/10.1145/3381831
  5. M. Musser, “A cost analysis of generative language models and influence operations,” arXiv preprint arXiv:2308.03740, 2023.
  6. D. Wright, D. Wadden, K. Lo, B. Kuehl, A. Cohan, I. Augenstein, and L. Wang, “Generating scientific claims for zero-shot scientific fact checking,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
  7. E. Altuncu, J. R. C. Nurse, M. Bagriacik, S. Kaleba, H. Yuan, L. Bonheme, and S. Li, “aedfact: Scientific fact-checking made easier via semi-automatic discovery of relevant expert opinions,” in Proceedings of the 17th International AAAI Conference on Web and Social Media, 2023.
  8. J. Vladika and F. Matthes, “Scientific fact-checking: A survey of resources and approaches,” in Findings of the Association for Computational Linguistics: ACL 2023, 2023.
  9. L. Rakotoson, C. Letaillieur, S. Massip, and F. A. A. Laleye, “Extractive-boolean question answering for scientific fact checking,” in Proceedings of the 1st International Workshop on Multimedia AI against Disinformation, ser. MAD ’22.   New York, NY, USA: Association for Computing Machinery, 2022, p. 27–34.
  10. G. Kell, I. Marshall, B. Wallace, and A. Jaun, “What would it take to get biomedical QA systems into practice?” in Proceedings of the 3rd Workshop on Machine Reading for Question Answering, A. Fisch, A. Talmor, D. Chen, E. Choi, M. Seo, P. Lewis, R. Jia, and S. Min, Eds.   Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 28–41. [Online]. Available: https://aclanthology.org/2021.mrqa-1.3
  11. H.-T. Chen, F. Xu, S. Arora, and E. Choi, “Understanding retrieval augmentation for long-form question answering,” arXiv preprint arXiv:2310.12150, 2023.
  12. W. Yu, D. Iter, S. Wang, Y. Xu, M. Ju, S. Sanyal, C. Zhu, M. Zeng, and M. Jiang, “Generate rather than retrieve: Large language models are strong context generators,” arXiv preprint arXiv:2209.10063, 9 2022. [Online]. Available: https://arxiv.org/pdf/2209.10063.pdf
  13. B. Galitsky, “Truth-o-meter: Collaborating with llm in fighting its hallucinations,” Preprints, 7 2023. [Online]. Available: http://dx.doi.org/10.20944/preprints202307.1723.v1
  14. S. Xu, L. Pang, H. Shen, X. Cheng, and T.-S. Chua, “Search-in-the-chain: Towards accurate, credible and traceable large language models for knowledge-intensive tasks,” arXiv preprint arXiv:2304.14732, 2023.
  15. J. Liu, J. S. Jin, Z. Wang, J. Cheng, Z. Dou, and J.-R. Wen, “Reta-llm: A retrieval-augmented large language model toolkit,” arXiv preprint arXiv:2306.05212, 6 2023. [Online]. Available: https://arxiv.org/pdf/2306.05212.pdf
  16. S. Bhatia, J. H. Lau, and T. Baldwin, “Automatic claim review for climate science via explanation generation,” arXiv preprint arXiv:2107.14740, 8 2021. [Online]. Available: https://arxiv.org/pdf/2107.14740.pdf
  17. S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, and S. Nanayakkara, “Improving the domain adaptation of retrieval augmented generation (RAG) models for open domain question answering,” Transactions of the Association for Computational Linguistics, vol. 11, pp. 1–17, jan 2023. [Online]. Available: https://aclanthology.org/2023.tacl-1.1
  18. V. Sanca and A. Ailamaki, “E-scan: Consuming contextual data with model plugins,” in Joint Workshops at 49th International Conference on Very Large Data Bases (VLDBW’23), 2023.
  19. Y. Han, C. Liu, and P. Wang, “A comprehensive survey on vector database: Storage and retrieval technique, challenge,” arXiv preprint arXiv:2310.11703, 2023.
  20. F. Bang, “GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings,” in Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), L. Tan, D. Milajevs, G. Chauhan, J. Gwinnup, and E. Rippeth, Eds.   Singapore: Association for Computational Linguistics, Dec. 2023, pp. 212–218. [Online]. Available: https://aclanthology.org/2023.nlposs-1.24
  21. N. Muennighoff, N. Tazi, L. Magne, and N. Reimers, “Mteb: Massive text embedding benchmark,” arXiv preprint arXiv:2210.07316, 2022.
  22. N. Arabzadeh, X. Yan, and C. L. A. Clarke, “Predicting efficiency/effectiveness trade-offs for dense vs. sparse retrieval strategy selection,” arXiv preprint arXiv:2109.10739, 2021.
  23. C. Sciavolino, Z. Zhong, J. Lee, and D. Chen, “Simple entity-centric questions challenge dense retrievers,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds.   Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6138–6148. [Online]. Available: https://aclanthology.org/2021.emnlp-main.496
  24. N. Mihindukulasooriya, S. Tiwari, C. F. Enguix, and K. Lata, “Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text,” arXiv preprint arXiv:2308.02357, 2023.
  25. M. F. M. Chowdhury, M. Glass, G. Rossiello, A. Gliozzo, and N. Mihindukulasooriya, “Kgi: An integrated framework for knowledge intensive language tasks,” arXiv preprint arXiv:2204.03985, 9 2022. [Online]. Available: https://arxiv.org/pdf/2204.03985.pdf
  26. J. Baek, A. F. Aji, and A. Saffari, “Knowledge-augmented language model prompting for zero-shot knowledge graph question answering,” in Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023).   Toronto, ON, Canada: Association for Computational Linguistics, Jul. 2023, pp. 70–98. [Online]. Available: http://dx.doi.org/10.18653/v1/2023.matching-1.7
  27. I. Beltagy, K. Lo, and A. Cohan, “SciBERT: A pretrained language model for scientific text,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan, Eds.   Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 3615–3620. [Online]. Available: https://aclanthology.org/D19-1371
  28. F. Brasileiro, J. a. P. A. Almeida, V. A. Carvalho, and G. Guizzardi, “Applying a multi-level modeling theory to assess taxonomic hierarchies in wikidata,” in Proceedings of the 25th International Conference Companion on World Wide Web, ser. WWW ’16 Companion.   Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee, 2016, p. 975–980.
  29. F. B. Rogers, “Medical subject headings,” Bull. Med. Libr. Assoc., vol. 51, pp. 114–116, Jan. 1963.
  30. D. Altinok, “An ontology-based dialogue management system for banking and finance dialogue systems,” arXiv preprint arXiv:1804.04838, 2018.
  31. W. Xiong, X. L. Li, S. Iyer, J. Du, P. Lewis, W. Y. Wang, Y. Mehdad, W. tau Yih, S. Riedel, D. Kiela, and B. Oğuz, “Answering complex open-domain questions with multi-hop dense retrieval,” arXiv preprint arXiv:2009.12756, 2021.
  32. N. Reimers and I. Gurevych, “Making monolingual sentence embeddings multilingual using knowledge distillation,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 11 2020. [Online]. Available: https://arxiv.org/abs/2004.09813
  33. ——, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.   Association for Computational Linguistics, 11 2019. [Online]. Available: https://arxiv.org/abs/1908.10084
  34. N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” arXiv preprint arXiv:2307.03172, 2023.
  35. W. Shen, C. Li, H. Chen, M. Yan, X. Quan, H. Chen, J. Zhang, and F. Huang, “Small llms are weak tool learners: A multi-llm agent,” arXiv preprint arXiv:2401.07324, 2024.
  36. G. Juneja, S. Dutta, S. Chakrabarti, S. Manchanda, and T. Chakraborty, “Small language models fine-tuned to coordinate larger language models improve complex reasoning,” arXiv preprint arXiv:2310.18338, 2023.
  37. S. Verma, K. Tran, Y. Ali, and G. Min, “Reducing llm hallucinations using epistemic neural networks,” arXiv preprint arXiv:2312.15576, 2023.
  38. J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation,” arXiv preprint arXiv:2402.03216, 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Loïc Rakotoson (5 papers)
  2. Sylvain Massip (4 papers)
  3. Fréjus A. A. Laleye (5 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now