Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval (2310.01880v2)

Published 3 Oct 2023 in cs.LG

Abstract: Machine-based prediction of real-world events is garnering attention due to its potential for informed decision-making. Whereas traditional forecasting predominantly hinges on structured data like time-series, recent breakthroughs in LLMs enable predictions using unstructured text. In particular, (Zou et al., 2022) unveils AutoCast, a new benchmark that employs news articles for answering forecasting queries. Nevertheless, existing methods still trail behind human performance. The cornerstone of accurate forecasting, we argue, lies in identifying a concise, yet rich subset of news snippets from a vast corpus. With this motivation, we introduce AutoCast++, a zero-shot ranking-based context retrieval system, tailored to sift through expansive news document collections for event forecasting. Our approach first re-ranks articles based on zero-shot question-passage relevance, honing in on semantically pertinent news. Following this, the chosen articles are subjected to zero-shot summarization to attain succinct context. Leveraging a pre-trained LLM, we conduct both the relevance evaluation and article summarization without needing domain-specific training. Notably, recent articles can sometimes be at odds with preceding ones due to new facts or unanticipated incidents, leading to fluctuating temporal dynamics. To tackle this, our re-ranking mechanism gives preference to more recent articles, and we further regularize the multi-passage representation learning to align with human forecaster responses made on different dates. Empirical results underscore marked improvements across multiple metrics, improving the performance for multiple-choice questions (MCQ) by 48% and true/false (TF) questions by up to 8%. Code is available at https://github.com/BorealisAI/Autocast-plus-plus.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pp.  1877–1901, 2020.
  2. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  1870–1879, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  3. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, pp.  55–64, New York, NY, USA, 2016. Association for Computing Machinery.
  4. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
  5. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874–880, Online, April 2021. Association for Computational Linguistics.
  6. ForecastQA: A question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp.  4636–4650, Online, August 2021. Association for Computational Linguistics.
  7. Unifiedqa-v2: Stronger generalization via broader cross-format training. arXiv preprint arXiv:2202.12359, 2022.
  8. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, 2020.
  9. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
  10. SimCLS: A simple framework for contrastive learning of abstractive summarization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp.  1065–1072, Online, August 2021. Association for Computational Linguistics.
  11. BRIO: Bringing order to abstractive summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  2890–2903, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  12. Revisiting the gold standard: Grounding summarization evaluation with robust human evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  4140–4170, Toronto, Canada, July 2023. Association for Computational Linguistics.
  13. Can chatgpt forecast stock price movements? return predictability and large language models. arXiv preprint arXiv:2304.07619, 2023.
  14. Neural models for information retrieval. arXiv preprint arXiv:1705.01509, 2017.
  15. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
  16. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  17. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
  18. Okapi at trec-3. In Overview of the Third Text REtrieval Conference (TREC-3), pp.  109–126. Gaithersburg, MD: NIST, January 1995.
  19. Improving passage retrieval with zero-shot question generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.  3781–3797, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
  20. End-to-end training of multi-document reader and retriever for open-domain question answering. In Advances in Neural Information Processing Systems, 2021.
  21. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sCrnllCtjoE.
  22. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp.  3784–3803, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  23. A survey for efficient open domain question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  14447–14465, Toronto, Canada, July 2023a. Association for Computational Linguistics.
  24. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848, 2023b.
  25. Dense text retrieval based on pretrained language models: A survey. arXiv preprint arXiv:2211.14876, 2022.
  26. Large language models for information retrieval: A survey, 2023.
  27. Forecasting future world events with neural networks. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Qi Yan (45 papers)
  2. Raihan Seraj (8 papers)
  3. Jiawei He (41 papers)
  4. Lili Meng (23 papers)
  5. Tristan Sylvain (20 papers)
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets