Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Social Intelligence Data Infrastructure: Structuring the Present and Navigating the Future (2403.14659v1)

Published 28 Feb 2024 in cs.CY, cs.AI, and cs.CL

Abstract: As NLP systems become increasingly integrated into human social life, these technologies will need to increasingly rely on social intelligence. Although there are many valuable datasets that benchmark isolated dimensions of social intelligence, there does not yet exist any body of work to join these threads into a cohesive subfield in which researchers can quickly identify research gaps and future directions. Towards this goal, we build a Social AI Data Infrastructure, which consists of a comprehensive social AI taxonomy and a data library of 480 NLP datasets. Our infrastructure allows us to analyze existing dataset efforts, and also evaluate LLMs' performance in different social intelligence aspects. Our analyses demonstrate its utility in enabling a thorough understanding of current data landscape and providing a holistic perspective on potential directions for future dataset development. We show there is a need for multifaceted datasets, increased diversity in language and culture, more long-tailed social situations, and more interactive data in future social intelligence data efforts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (115)
  1. Khalid Alnajjar and Mika Hämäläinen. 2021. ! qu\\\backslash\’e maravilla! multimodal sarcasm detection in spanish: a dataset and a baseline. arXiv preprint arXiv:2105.05542.
  2. Explainable artificial intelligence: an analytical review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 11(5):e1424.
  3. Ian Apperly. 2010. Mindreaders: the cognitive basis of" theory of mind". Psychology Press.
  4. Mohamed Jehad Baeth. 2019. Provenance use in social media software to develop methodologies for detection of information pollution. Ph.D. thesis.
  5. Albert Bandura. 2009. Social cognitive theory of mass communication. In Media effects, pages 110–140. Routledge.
  6. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–16.
  7. Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks. arXiv preprint arXiv:2109.06275.
  8. Tweeteval: Unified benchmark and comparative evaluation for tweet classification. arXiv preprint arXiv:2010.12421.
  9. Michael L Barnes and Robert J Sternberg. 1989. Social intelligence and decoding of nonverbal cues. Intelligence, 13(3):263–287.
  10. Emily M Bender and Alexander Koller. 2020. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 5185–5198.
  11. Nicolas Bertagnolli. 2020. Counsel chat: Bootstrapping high-quality therapy data.
  12. A dataset of hindi-english code-mixed social media text for hate speech detection. In Proceedings of the second workshop on computational modeling of people’s opinions, personality, and emotions in social media, pages 36–41.
  13. Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. arXiv preprint arXiv:1810.00278.
  14. A sentiment analysis dataset for code-mixed malayalam-english. arXiv preprint arXiv:2006.00210.
  15. Corpus creation for sentiment analysis in code-mixed tamil-english text. arXiv preprint arXiv:2006.00206.
  16. Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems. arXiv preprint arXiv:2103.15721.
  17. Social influence dialogue systems: A survey of datasets and models for social influence tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 750–766.
  18. Compost: Characterizing and evaluating caricature in llm simulations. arXiv preprint arXiv:2310.11501.
  19. Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark.
  20. Robert B Cialdini and Noah J Goldstein. 2004. Social influence: Compliance and conformity. Annu. Rev. Psychol., 55:591–621.
  21. Will-they-won’t-they: A very large dataset for stance detection on twitter. arXiv preprint arXiv:2005.00388.
  22. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190.
  23. Towards computational proxemics: Inferring social relations from interpersonal distances. In 2011 IEEE Third International Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third International Conference on Social Computing, pages 290–297. IEEE.
  24. Alan Cruse. 2004. Meaning in language: An introduction to semantics and pragmatics.
  25. You shall know a user by the company it keeps: Dynamic representations for social media users in nlp. arXiv preprint arXiv:1909.00412.
  26. Goemotions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547.
  27. Emoticons and social interaction on the internet: the importance of social context. Computers in human behavior, 23(1):842–849.
  28. JA DeVito. 2016. The interpersonal communication book, global edition.
  29. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388.
  30. Mica R Endsley. 1990. Situation awareness in dynamic human decision making: Theory and measurement. Ph.D. thesis, University of Southern California Los Angeles, CA.
  31. Thomas Erickson. 2009. ‘social’systems: designing digital systems that support social intelligence. Ai & Society, 23(2):147–166.
  32. Artificial social intelligence: A comparative and holistic view. CAAI Artificial Intelligence Research, 1(2):144–160.
  33. Social chemistry 101: Learning to reason about social and moral norms. arXiv preprint arXiv:2011.00620.
  34. Martin E Ford and Marie S Tisak. 1983. A further search for social intelligence. Journal of Educational Psychology, 75(2):196.
  35. Epic: Multi-perspective annotation of a corpus of irony. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13844–13857.
  36. Distribution of emotional reactions to news articles in twitter. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
  37. Cicero: A dataset for contextualized commonsense inference in dialogues. arXiv preprint arXiv:2203.13926.
  38. Ivan Habernal and Iryna Gurevych. 2016. Which argument is more convincing? analyzing and predicting convincingness of web arguments using bidirectional lstm. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1589–1599.
  39. Edward Twitchell Hall and Mildred Reed Hall. 1987. Hidden differences: Doing business with the japanese. (No Title).
  40. A material lens on coloniality in nlp. arXiv preprint arXiv:2311.08391.
  41. Dirk Hovy and Diyi Yang. 2021. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–602.
  42. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In 13th International Conference on Natural Language Generation 2020, pages 169–182. Association for Computational Linguistics.
  43. Jing Huang and Diyi Yang. 2023. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609.
  44. Challenges in building intelligent open-domain dialog systems. ACM Transactions on Information Systems (TOIS), 38(3):1–32.
  45. Thelma Hunt. 1928. The measurement of social intelligence. Journal of Applied Psychology, 12(3):317.
  46. Towards computational persuasion via natural language argumentation dialogues. In KI 2019: Advances in Artificial Intelligence: 42nd German Conference on AI, Kassel, Germany, September 23–26, 2019, Proceedings 42, pages 18–33. Springer.
  47. Zunaira Jamil. 2017. Monitoring tweets for depression to detect at-risk users. Ph.D. thesis, Université d’Ottawa/University of Ottawa.
  48. Pratik Jayarao and Aman Srivastava. 2018. Intent detection for code-mix utterances in task oriented dialogue systems. In 2018 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT), pages 583–587. IEEE.
  49. The crecil corpus: a new dataset for extraction of relations between characters in chinese multi-party dialogues. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2337–2344.
  50. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in neural information processing systems, 35:28458–28473.
  51. Machine-aided annotation for fine-grained proposition types in argumentation. In 12th International Conference on Language Resources and Evaluation, LREC 2020, pages 1008–1018. European Language Resources Association (ELRA).
  52. John F Kihlstrom and Nancy Cantor. 2000. Social intelligence.
  53. Measuring emotions in the covid-19 real world worry dataset. arXiv preprint arXiv:2004.04225.
  54. Corinne Kosmitzki and Oliver P John. 1993. The implicit use of explicit conceptions of social intelligence. Personality and individual differences, 15(1):11–23.
  55. Dilek Küçük and Fazli Can. 2020. Stance detection: A survey. ACM Computing Surveys (CSUR), 53(1):1–37.
  56. Theory of mind and preference learning at the interface of cognitive science, neuroscience, and ai: A review. Frontiers in Artificial Intelligence, 5:62.
  57. Revisiting the evaluation of theory of mind through question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5872–5877.
  58. Personachatgen: Generating personalized dialogues using gpt-3. In Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge, pages 29–48.
  59. Coannotating: Uncertainty-guided work allocation between human and large language models for data annotation. arXiv preprint arXiv:2310.15638.
  60. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
  61. P-stance: A large dataset for stance detection in political domain. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2355–2365.
  62. Q Vera Liao and S Shyam Sundar. 2022. Designing for responsible trust in ai systems: A communication perspective. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1257–1268.
  63. Filip Lievens and David Chan. 2017. Practical intelligence, emotional intelligence, and social intelligence. Handbook of employee selection, pages 342–364.
  64. The teams corpus and entrainment in multi-party spoken dialogues. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1421–1431.
  65. Studying common ground instantiation using audio, video and brain behaviours: the brainkt corpus. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 691–702.
  66. Herbert A Marlowe. 1986. Social intelligence: Evidence for multidimensionality and construct independence. Journal of educational psychology, 78(1):52.
  67. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1–17.
  68. Semeval-2016 task 6: Detecting stance in tweets. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 31–41.
  69. The uncanny valley [from the field]. IEEE Robotics & automation magazine, 19(2):98–100.
  70. Fred August Moss and Thelma Hunt. 1927. Are you socially intelligent? Scientific American, 137(2):108–110.
  71. Creating a domain-diverse corpus for theory-based argument quality assessment. arXiv preprint arXiv:2011.01589.
  72. Silviu Oprea and Walid Magdy. 2019. isarcasm: A dataset of intended sarcasm. arXiv preprint arXiv:1911.03123.
  73. Raddle: An evaluation benchmark and analysis platform for robust task-oriented dialog systems. arXiv preprint arXiv:2012.14666.
  74. Xianglan Peng. 2021. Research on emotion recognition based on deep learning for mental health. Informatica, 45(1).
  75. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515–526.
  76. A benchmark dataset for learning to intervene in online hate speech. arXiv preprint arXiv:1909.04251.
  77. Characterizing variation in toxic language by social context. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 959–963.
  78. Semi-supervised user geolocation via graph convolutional networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2009–2019, Melbourne, Australia. Association for Computational Linguistics.
  79. Tamilatis: dataset for task-oriented dialog in tamil. In Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages, pages 25–32.
  80. Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:1811.00207.
  81. The ACL OCL corpus: Advancing open science in computational linguistics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10348–10361, Singapore. Association for Computational Linguistics.
  82. Jonathan P Roiser and Barbara J Sahakian. 2013. Hot and cold cognition in depression. CNS spectrums, 18(3):139–149.
  83. Theory of mind and decision science: Towards a typology of tasks and computational models. Neuropsychologia, 146:107488.
  84. Anthropomorphism in ai. AJOB neuroscience, 11(2):88–95.
  85. Searching brazilian twitter for signs of mental health issues. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6111–6117.
  86. The risk of racial bias in hate speech detection. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1668–1678.
  87. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312.
  88. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.
  89. Poorvi Shetty. 2023. Poorvi@ dravidianlangtech: Sentiment analysis on code-mixed tulu and tamil corpus. In Proceedings of the Third Workshop on Speech and Language Technologies for Dravidian Languages, pages 124–132.
  90. The tromsø social intelligence scale, a self-report measure of social intelligence. Scandinavian journal of psychology, 42(4):313–319.
  91. Sonali Singh and Navita Srivastava. 2023. Emotion recognition for mental health prediction using ai techniques: An overview. International Journal of Advanced Research in Computer Science, 14(3).
  92. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 551–557.
  93. Detection of depression-related posts in reddit social media forum. Ieee Access, 7:44883–44893.
  94. Chenhao Tan and Lillian Lee. 2014. A corpus of sentence-level revisions in academic writing: A step towards understanding statement strength in communication. arXiv preprint arXiv:1405.1439.
  95. Depac: a corpus for depression and anxiety detection from speech. arXiv preprint arXiv:2306.12443.
  96. Edward L Thorndike. 1921. Intelligence and its measurement: A symposium–i. Journal of Educational psychology, 12(3):124.
  97. Pride: Predicting relationships in conversations. In The Conference on Empirical Methods in Natural Language Processing, pages 4636–4650. ACL.
  98. John C Turner. 1991. Social influence. Thomson Brooks/Cole Publishing Co.
  99. Homo-mex: A mexican spanish annotated corpus for lgbt+ phobia detection on twitter. In The 7th Workshop on Online Abuse and Harms (WOAH), pages 202–214.
  100. Philip E Vernon. 1933. Some characteristics of the good judge of personality. The Journal of Social Psychology, 4(1):42–57.
  101. Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:1906.06725.
  102. Daniela Wawra. 2013. Social intelligence: The key to intercultural communication. In Intercultural Negotiations, pages 29–42. Routledge.
  103. Eugene A Weinstein. 1969. The development of interpersonal competence. Handbook of socialization theory and research, pages 753–775.
  104. Supporting artificial social intelligence with theory of mind. Frontiers in artificial intelligence, 5:750763.
  105. Joint intent detection model for task-oriented human-computer dialogue system using asynchronous training. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(5):1–17.
  106. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1956–1970.
  107. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  108. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
  109. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, 63(10):2011–2027.
  110. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667.
  111. Cobra frames: Contextual reasoning about effects and harms of offensive statements. arXiv preprint arXiv:2306.01985.
  112. Crosswoz: A large-scale chinese cross-domain task-oriented dialogue dataset. Transactions of the Association for Computational Linguistics, 8:281–295.
  113. NormBank: A knowledge bank of situational social norms. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7756–7776, Toronto, Canada. Association for Computational Linguistics.
  114. Normbank: A knowledge bank of situational social norms. arXiv preprint arXiv:2305.17008.
  115. Can large language models transform computational social science? arXiv preprint arXiv:2305.03514.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Minzhi Li (8 papers)
  2. Weiyan Shi (42 papers)
  3. Caleb Ziems (22 papers)
  4. Diyi Yang (151 papers)
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

HackerNews

  1. Social Intelligence Data Infrastructure (1 point, 0 comments)