Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

CausalCite: A Causal Formulation of Paper Citations (2311.02790v3)

Published 5 Nov 2023 in cs.CL, cs.AI, cs.CY, cs.IR, and cs.LG

Abstract: Citation count of a paper is a commonly used proxy for evaluating the significance of a paper in the scientific community. Yet citation measures are widely criticized for failing to accurately reflect the true impact of a paper. Thus, we propose CausalCite, a new way to measure the significance of a paper by assessing the causal impact of the paper on its follow-up papers. CausalCite is based on a novel causal inference method, TextMatch, which adapts the traditional matching framework to high-dimensional text embeddings. TextMatch encodes each paper using text embeddings from LLMs, extracts similar samples by cosine similarity, and synthesizes a counterfactual sample as the weighted average of similar papers according to their similarity values. We demonstrate the effectiveness of CausalCite on various criteria, such as high correlation with paper impact as reported by scientific experts on a previous dataset of 1K papers, (test-of-time) awards for past papers, and its stability across various subfields of AI. We also provide a set of findings that can serve as suggested ways for future researchers to use our metric for a better understanding of the quality of a paper. Our code is available at https://github.com/causalNLP/causal-cite.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Synthetic control methods for comparative case studies: Estimating the effect of california’s tobacco control program. Journal of the American statistical Association, 105(490):493–505.
  2. Alberto Abadie and Javier Gardeazabal. 2003. The economic costs of conflict: A case study of the basque country. American economic review, 93(1):113–132.
  3. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  4. Applied usage and performance of statistical matching in bibliometrics: The comparison of milestone and regular papers with multiple measurements of disruptiveness as an empirical example. Quantitative Science Studies, 2(4):1246–1270.
  5. Language models are few-shot learners. ArXiv, abs/2005.14165.
  6. Håkan Carlsson. 2009. Allocation of research funds using bibliometric indicators–asset and challenge to swedish higher education sector.
  7. Patent citation network analysis: A perspective from descriptive statistics and ergms. Plos one, 15(12):e0241797.
  8. Dhivya Chandrasekaran and Vijay Mago. 2022. Evolution of semantic similarity - A survey. ACM Comput. Surv., 54(2):41:1–41:37.
  9. Citesee: Augmenting citations in scientific papers with persistent and personalized historical context. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–15.
  10. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1–240:113.
  11. Speeding up to keep up: exploring the use of ai in the research process. AI & society, 37(4):1439–1457.
  12. SPECTER: Document-level Representation Learning using Citation-informed Transformers. In ACL.
  13. Artificial intelligence in information systems research: A systematic literature review and research agenda. International Journal of Information Management, 60:102383.
  14. Corinna Cortes and Neil D. Lawrence. 2021. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. CoRR, abs/2109.09774.
  15. The strong-focusing synchrotron-a new high energy accelerator. Physical Review, 88:1190–1196.
  16. Current state and future trends: A citation network analysis of the learning analytics field. In Proceedings of the fourth international conference on learning analytics and knowledge, pages 231–240.
  17. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), pages 248–255.
  18. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  19. How to conduct a bibliometric analysis: An overview and guidelines. Journal of business research, 133:285–296.
  20. Susan A Elmore. 2018. The altmetric attention score: what does it mean and why should i care?
  21. Xing Fang and Justin Zhijun Zhan. 2015. Sentiment analysis using product review data. Journal of Big Data, 2:1–14.
  22. Eugene Garfield. 1964. " science citation index"—a new dimension in indexing: This unique approach underlies versatile bibliographic systems for communicating and evaluating information. Science, 144(3619):649–654.
  23. Eugene Garfield. 1972. Citation analysis as a tool in journal evaluation: Journals can be ranked by frequency and impact of citations for science policy studies. Science, 178(4060):471–479.
  24. The use of citation data in writing the history of science.
  25. Gary Rosenberg Gary Holden and Kathleen Barker. 2005. Bibliometrics. Social Work in Health Care, 41(3-4):67–92.
  26. Miguel A Hernán and James M Robins. 2010. Causal inference.
  27. Jorge E Hirsch. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National academy of Sciences, 102(46):16569–16572.
  28. Jianhua Hou. 2017. Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy. Scientometrics, 110:1437–1452.
  29. Relative citation ratio (rcr): a new metric that uses citation rates to measure influence at the article level. PLoS biology, 14(9):e1002541.
  30. A decade of in-text citation analysis based on natural language processing and machine learning techniques: An overview of empirical studies. Scientometrics, 126(8):6551–6599.
  31. The use of citation context to detect the evolution of research topics: a large-scale analysis. Scientometrics, 126(4):2971–2989.
  32. The semantic scholar open data platform. CoRR, abs/2301.10140.
  33. Teun Kloek and Herman K. van Dijk. 1976. Bayesian estimates of equation system parameters, an application of integration by monte carlo. Econometrica, 46:1–19.
  34. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  35. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  36. Semantic textual similarity methods, tools, and applications: A survey. Computación y Sistemas, 20.
  37. New trends in scientific knowledge graphs and research impact assessment.
  38. Introduction to information retrieval. In J. Assoc. Inf. Sci. Technol.
  39. Henk F Moed. 2006. Citation analysis in research evaluation, volume 9. Springer Science & Business Media.
  40. Judea Pearl. 2009. Causality. Cambridge University Press.
  41. Fredrik Niclas Piro and Gunnar Sivertsen. 2016. How can differences in international university rankings be explained? Scientometrics, 109(3):2263–2278.
  42. A community’s perspective on the status and future of peer review in software engineering. Information and Software Technology, 95:75–85.
  43. Alec Radford and Karthik Narasimhan. 2018. Improving language understanding by generative pre-training.
  44. Language models are unsupervised multitask learners.
  45. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing.
  46. Perceptions of ethical problems with scientific journal peer review: an exploratory study. Science and engineering ethics, 14(3):305–310.
  47. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3:333–389.
  48. Program chairs’ report on peer review at acl 2023. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages xl–lxxv, Toronto, Canada. Association for Computational Linguistics.
  49. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685.
  50. Paul R Rosenbaum and Donald B Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41–55.
  51. Nihar B Shah. 2022. An overview of challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87.
  52. Surya Nath Singh. 2014. Sampling techniques & determination of sample size in applied statistics research : an overview.
  53. Abir Smiti. 2020. A critical overview of outlier detection methods. Computer Science Review, 38:100306.
  54. Mpnet: Masked and permuted pre-training for language understanding. arXiv preprint arXiv:2004.09297.
  55. Robert F. Tate. 1954. Correlation between a discrete and a continuous variable. point-biserial correlation. Annals of Mathematical Statistics, 25:603–607.
  56. Scientific papers citation analysis using textual features and smote resampling techniques. Pattern Recognition Letters, 150:250–257.
  57. Identifying meaningful citations. In Scholarly Big Data: AI Perspectives, Challenges, and Ideas, Papers from the 2015 AAAI Workshop, Austin, Texas, USA, January, 2015, volume WS-15-13 of AAAI Technical Report. AAAI Press.
  58. Identifying meaningful citations. In AAAI Workshop: Scholarly Big Data.
  59. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.
  60. The metric tide: report of the independent review of the role of metrics in research assessment and management.
  61. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics.
  62. Marta Natalia Wróblewska. 2021. Research impact evaluation and academic discourse. Humanities and Social Sciences Communications, 8(1):1–12.
  63. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ArXiv, abs/2304.13712.
  64. Nurulhuda Zainuddin and Ali Selamat. 2014. Sentiment analysis using support vector machine. 2014 International Conference on Computer, Communications, and Control Technology (I4CT), pages 333–337.
  65. Measuring academic influence: Not all citations are equal. Journal of the Association for Information Science and Technology, 66.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now