Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Did AI get more negative recently? (2202.13610v3)

Published 28 Feb 2022 in cs.CL, cs.LG, and cs.SI

Abstract: In this paper, we classify scientific articles in the domain of NLP and ML, as core subfields of AI, into whether (i) they extend the current state-of-the-art by the introduction of novel techniques which beat existing models or whether (ii) they mainly criticize the existing state-of-the-art, i.e. that it is deficient with respect to some property (e.g. wrong evaluation, wrong datasets, misleading task specification). We refer to contributions under (i) as having a 'positive stance' and contributions under (ii) as having a 'negative stance' (to related work). We annotate over 1.5 k papers from NLP and ML to train a SciBERT-based model to automatically predict the stance of a paper based on its title and abstract. We then analyse large-scale trends on over 41 k papers from the last approximately 35 years in NLP and ML, finding that papers have become substantially more positive over time, but negative papers also got more negative and we observe considerably more negative papers in recent years. Negative papers are also more influential in terms of citations they receive.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (74)
  1. Purpose and polarity of citation: Towards NLP-based bibliometrics. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 596–606, Atlanta, Georgia. Association for Computational Linguistics.
  2. Philippe Aghion and Peter Howitt. 1990. A model of growth through creative destruction.
  3. Claudio Altafini. 2012. Consensus problems on networks with antagonistic interactions. IEEE transactions on automatic control, 58(4):935–946.
  4. Chittaranjan Andrade. 2011. How to write a good abstract for a scientific paper or conference presentation. Indian Journal of Psychiatry, 53:172 – 175.
  5. Awais Athar and Simone Teufel. 2012. Context-enhanced citation sentiment detection. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 597–601, Montréal, Canada. Association for Computational Linguistics.
  6. Dominik Beese. 2023. DominikBeese/DidAIGetMoreNegativeRecently: Initial release.
  7. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620, Hong Kong, China. Association for Computational Linguistics.
  8. Tackling sparse data issue in machine translation evaluation. In Proceedings of the ACL 2010 Conference Short Papers, pages 86–91, Uppsala, Sweden. Association for Computational Linguistics.
  9. Frederique Bordignon. 2020. Self-correction of science: a comparative study of negative citations and post-publication peer review. Scientometrics, 124:1225–1239.
  10. Kaja Borthen. 2004. Predicative NPs and the annotation of reference chains. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 1175–1178, Geneva, Switzerland. COLING.
  11. Samuel Bowman. 2022. The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484–7499, Dublin, Ireland. Association for Computational Linguistics.
  12. The incidence and role of negative citations in science. Proceedings of the National Academy of Sciences, 112(45):13823–13826.
  13. Reproducibility issues for bert-based evaluation metrics. In Conference on Empirical Methods in Natural Language Processing, volume abs/2204.00004.
  14. The coefficient of determination r-squared is more informative than smape, mae, mape, mse and rmse in regression analysis evaluation. PeerJ Computer Science, 7:e623.
  15. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3586–3596, Minneapolis, Minnesota. Association for Computational Linguistics.
  16. Randall Collins. 1994. Why the social sciences won’t become high-consensus, rapid-discovery science. Sociological Forum, 9(2):155–177.
  17. Are we there yet? encoder-decoder neural networks as cognitive models of English past tense inflection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3868–3877, Florence, Italy. Association for Computational Linguistics.
  18. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  19. Yuan Ding and Martha Palmer. 2005. Machine translation using probabilistic synchronous dependency insertion grammars. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 541–548, Ann Arbor, Michigan. Association for Computational Linguistics.
  20. Ted Dunning. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74.
  21. Steffen Eger. 2016. Opinion dynamics and wisdom under out-group discrimination. Mathematical Social Sciences, 80:97–107.
  22. Dan Fass. 1991. met*: A method for discriminating metonymy and metaphor by computer. Computational Linguistics, 17(1):49–90.
  23. Science of science. Science, 359(6379).
  24. Does my rebuttal matter? insights from a major NLP conference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1274–1290, Minneapolis, Minnesota. Association for Computational Linguistics.
  25. Spandana Gella and Frank Keller. 2017. An analysis of action recognition datasets for language and vision tasks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 64–71, Vancouver, Canada. Association for Computational Linguistics.
  26. Martin Gerlach and Eduardo G. Altmann. 2014. Scaling laws and fluctuations in the statistics of word frequencies. New Journal of Physics, 16.
  27. Kyle Gorman and Steven Bedrick. 2019. We need to talk about standard splits. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2786–2791, Florence, Italy. Association for Computational Linguistics.
  28. Robbie Gunning. 1952. The Technique of Clear Writing. McGraw-Hill.
  29. Application of deep learning in ecological resource research: Theories, methods, and challenges. Science China Earth Sciences, pages 1–18.
  30. Bert & family eat word salad: Experiments with text understanding. Proceedings of the AAAI Conference on Artificial Intelligence, 35(14):12946–12954.
  31. Product classification in E-commerce using distributional semantics. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 536–546, Osaka, Japan. The COLING 2016 Organizing Committee.
  32. More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing.
  33. James Hendler. 2008. Avoiding another ai winter. IEEE Intelligent Systems, 23(2):2–4.
  34. Gustav Herdan. 1960. Language as choice and chance. Springer Berlin Heidelberg.
  35. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics.
  36. Vader: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1):216–225.
  37. The use of citation context to detect the evolution of research topics: a large-scale analysis. Scientometrics, 126(4):2971–2989.
  38. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.
  39. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1647–1661, New Orleans, Louisiana. Association for Computational Linguistics.
  40. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
  41. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097–1105.
  42. William H. Kruskal and W. Allen Wallis. 1952. Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260):583–621.
  43. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC.
  44. Meta-research: Investigating disagreement in the scientific literature. Elife, 10:e72737.
  45. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.
  46. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1875–1889, Seattle, United States. Association for Computational Linguistics.
  47. Mind the gap: Assessing temporal generalization in neural language models. In Thirty-Fifth Conference on Neural Information Processing Systems.
  48. The advantage of short paper titles. Royal Society open science, 2(8):150266.
  49. Context-aware embedding for targeted aspect-based sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4678–4683, Florence, Italy. Association for Computational Linguistics.
  50. Zachary C. Lipton and Jacob Steinhardt. 2019. Troubling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research. Queue, 17(1):45–77.
  51. Fei Liu and Yang Liu. 2009. From extractive to abstractive meeting summaries: Can it be done by sentence compression? In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 261–264, Suntec, Singapore. Association for Computational Linguistics.
  52. Roberta: A robustly optimized bert pretraining approach.
  53. Gary F. Marcus. 2018. Deep learning: A critical appraisal. ArXiv, abs/1801.00631.
  54. Scientific credibility of machine translation research: A meta-evaluation of 769 papers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7297–7306, Online. Association for Computational Linguistics.
  55. Dealing with the positive publication bias: Why you should really publish your negative results. Biochemia medica, 27(3):447–452.
  56. Machine learning with oversampling and undersampling techniques: Overview study and experimental results. In 2020 11th International Conference on Information and Communication Systems (ICICS), pages 243–248.
  57. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  58. Jiaxin Pei and David Jurgens. 2021. Measuring sentence-level and aspect-level (un)certainty in science communications. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9959–10011, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  59. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  60. Predicting the rise and fall of scientific topics from trends in their rhetorical framing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1170–1180, Berlin, Germany. Association for Computational Linguistics.
  61. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  62. Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Conference on Empirical Methods in Natural Language Processing.
  63. Defense-GAN: Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations.
  64. Julian Sienkiewicz and Eduardo G Altmann. 2016. Impact of lexical and sentiment factors on the popularity of scientific papers. Royal Society open science, 3(6):160140.
  65. On the limitations of unsupervised bilingual dictionary induction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 778–788, Melbourne, Australia. Association for Computational Linguistics.
  66. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP ’06, page 103–110, USA. Association for Computational Linguistics.
  67. ReviewRobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of the 13th International Conference on Natural Language Generation, pages 384–397, Dublin, Ireland. Association for Computational Linguistics.
  68. Norms of valence, arousal, and dominance for 13,915 english lemmas. Behavior Research Methods, 45(4):1191–1207.
  69. B. L. Welch. 1947. The generalisation of student’s problems when several different population variances are involved. Biometrika, 34(1-2):28–35.
  70. Dustin Wright and Isabelle Augenstein. 2021. CiteWorth: Cite-worthiness detection for improved scientific document understanding. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1796–1807, Online. Association for Computational Linguistics.
  71. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations.
  72. Citation count prediction: learning to estimate future citations for literature. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1247–1252.
  73. A survey on sentiment analysis of scientific citations. Artificial Intelligence Review, 52(3):1805–1838.
  74. Can we automate scientific reviewing? CoRR, abs/2102.00176.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now