Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions (2407.02472v2)

Published 2 Jul 2024 in cs.CL

Abstract: This study introduces ValueScope, a framework leveraging LLMs to quantify social norms and values within online communities, grounded in social science perspectives on normative structures. We employ ValueScope to dissect and analyze linguistic and stylistic expressions across 13 Reddit communities categorized under gender, politics, science, and finance. Our analysis provides a quantitative foundation showing that even closely related communities exhibit remarkably diverse norms. This diversity supports existing theories and adds a new dimension--community preference--to understanding community interactions. ValueScope not only delineates differing social norms among communities but also effectively traces their evolution and the influence of significant external events like the U.S. presidential elections and the emergence of new sub-communities. The framework thus highlights the pivotal role of social norms in shaping online interactions, presenting a substantial advance in both the theory and application of social norm studies in digital spaces.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497–511, San Diego, California. Association for Computational Linguistics.
  2. Stela: a community-centred approach to norm elicitation for ai alignment. Scientific Reports, 14(1):6616.
  3. Cristina Bicchieri. 2005. The grammar of society: The nature and dynamics of social norms. Cambridge University Press.
  4. Social Norms. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy, Winter 2023 edition. Metaphysics Research Lab, Stanford University.
  5. Evaluating the evaluation metrics for style transfer: A case study in multilingual formality transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1321–1336, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  6. Andrew D Brown. 2022. Identities in and around organizations: Towards an identity work perspective. Human relations, 75(7):1205–1237.
  7. Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press.
  8. Language models are few-shot learners.
  9. Norms matter: Contrasting social support around behavior change in online weight loss communities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–14.
  10. The internet’s hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. Proc. ACM Hum.-Comput. Interact., 2(CSCW).
  11. Walter Coutu. 1951. Role-playing vs. role-taking: An appeal for clarification. American sociological review, 16(2):180–187.
  12. "they are uncultured": Unveiling covert harms and social threats in llm generated conversations.
  13. No country for old members: user lifecycle and linguistic change in online communities. Proceedings of the 22nd international conference on World Wide Web.
  14. Do people prefer leaders who enforce norms? reputational effects of reward and punishment decisions in noisy social dilemmas. Journal of Experimental Social Psychology, 84:103800.
  15. Marco Del Tredici and Raquel Fernández. 2017. Semantic variation in online communities of practice. In Proceedings of the 12th International Conference on Computational Semantics (IWCS) — Long papers.
  16. Marco Del Tredici and Raquel Fernández. 2018. The road to success: Assessing the fate of linguistic innovations in online communities. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1591–1603, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  17. Detecting text formality: A study of text classification approaches. arXiv preprint arXiv:2204.08975.
  18. Larry Dignan. 2024. Reddit’s data licensing play: Do you want your llm trained on reddit data?
  19. Penelope Eckert. 1989. Jocks and burnouts: Social categories and identity in the high school. Teachers College Press.
  20. Penelope Eckert and Sally Mcconnell-Ginet. 1999. New generalizations and explanations in language and gender research. Language in Society, 28:185 – 201.
  21. Penelope Eckert and Sally McConnell-Ginet. 2013a. Language and Gender, 2 edition. Cambridge University Press.
  22. Penelope Eckert and Sally McConnell-Ginet. 2013b. Language and gender. Cambridge University Press.
  23. Reddit rules! characterizing an ecosystem of governance. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12.
  24. Inverse constitutional ai: Compressing preferences into principles.
  25. Dialogue response rankingtraining with large-scale human feedback data. In EMNLP.
  26. Carroll J Glynn and Michael E Huge. 2007. Opinions as norms: Applying a return potential model to the study of communication behaviors. Communication Research, 34(5):548–568.
  27. Erving Goffman. 1955. On face-work: An analysis of ritual elements in social interaction. Psychiatry, 18(3):213–231.
  28. Counterfactual probing for the influence of affect and specificity on intergroup bias. arXiv preprint arXiv:2305.16409.
  29. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pages 79–90.
  30. Deberta: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
  31. Libby Hemphill and Jahna Otterbacher. 2012. Learning the lingo? gender, prestige and linguistic adaptation in review communities. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work, CSCW ’12, page 305–314, New York, NY, USA. Association for Computing Machinery.
  32. A return potential measure of setting norms for aggression. American Journal of Community Psychology, 33(3-4):131–149.
  33. John Herrman. 2021. Everything’s a joke until it’s not.
  34. Science, askscience, and badscience: On the coexistence of highly related communities. In International Conference on Web and Social Media.
  35. Jay Jackson. 1966. A conceptual and measurement model for norms and roles. The Pacific Sociological Review, 9(1):35–47.
  36. Jay Jackson. 1975. Normative power and conflict potential. Sociological Methods & Research, 4(2):237–263.
  37. Automatic sarcasm detection: A survey. ACM Comput. Surv., 50(5).
  38. Anna Kasunic and Geoff Kaufman. 2018. " at least the pizzas you make are hot": Norms, values, and abrasive humor on the subreddit r/roastme. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12.
  39. Towards modelling language innovation acceptance in online social networks. In Proceedings of the Ninth ACM International Conference on Web Search and Data Mining, WSDM ’16, page 553–562, New York, NY, USA. Association for Computing Machinery.
  40. Amy Jo Kim. 2006. Community building on the web: Secret strategies for successful online communities. Peachpit press.
  41. Sanford Labovitz and Robert Hagedorn. 1973. Measuring social norms. Pacific Sociological Review, 16(3):283–303.
  42. Robin Lakoff. 1973. The logic of politeness: Or, minding your p’s and q’s. In Proceedings from the Annual Meeting of the Chicago Linguistic Society, volume 9, pages 292–305. Chicago Linguistic Society.
  43. Crowdsourcing civility: A natural experiment examining the effects of distributed moderation in online forums. Gov. Inf. Q., 31:317–326.
  44. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.
  45. Justin Lee and Sowmya Vajjala. 2022. A neural pairwise ranking model for readability assessment. arXiv preprint arXiv:2203.07450.
  46. Norms and their relationship to behavior in worksite settings: an application of the jackson return potential model. American journal of health behavior, 29(3):258–268.
  47. Characterizing English variation across social media communities with BERT. Transactions of the Association for Computational Linguistics, 9:538–556.
  48. Charles G McClintock. 1978. Social values: Their definition, measurement and development. Journal of Research & Development in Education.
  49. Rachel I McDonald and Christian S Crandall. 2015. Social norms and social influence. Current Opinion in Behavioral Sciences, 3:147–151.
  50. Evaluating style transfer for text. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 495–504, Minneapolis, Minnesota. Association for Computational Linguistics.
  51. Shyamal Mishra and Preetha Chatterjee. 2023. Exploring chatgpt for toxicity detection in github. arXiv preprint arXiv:2312.13105.
  52. Laetitia B Mulder. 2008. The difference between punishments and rewards in fostering moral concerns in social decision making. Journal of Experimental Social Psychology, 44(6):1436–1443.
  53. Yair Neuman and Yochai Cohen. 2023. Ai for identifying social norm violation. Scientific Reports, 13(1):8103.
  54. Jessica M Nolan. 2015. Using jackson’s return potential model to explore the normativeness of recycling. Environment and Behavior, 47(8):835–855.
  55. OpenAI. 2024a. Best practices for prompt engineering with the openai api. https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the-openai-api. Accessed:2024-01-11.
  56. OpenAI. 2024b. Prompt engineering. https://platform.openai.com/docs/guides/prompt-engineering. Accessed:2024-01-11.
  57. OpenAI. 2024. Text generation models. https://platform.openai.com/docs/guides/text-generation.
  58. Gpt-4 technical report.
  59. Detecting community sensitive norm violations in online conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3386–3397, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  60. Quick, community-specific learning: How distinctive toxicity norms are maintained in political subreddits. In International Conference on Web and Social Media.
  61. What makes it ok to set a fire? iterative self-distillation of contexts and rationales for disambiguating defeasible social and moral situations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12140–12159, Singapore. Association for Computational Linguistics.
  62. Qinlan Shen and Carolyn P Rosé. 2022. A tale of two subreddits: Measuring the impacts of quarantines on political engagement on reddit. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 932–943.
  63. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  64. Measuring misogyny in natural language generation: Preliminary results from a case study on two reddit communities. arXiv preprint arXiv:2312.03330.
  65. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In AAAI Conference on Artificial Intelligence.
  66. Claudio Vaz Torres. 1999. Leadership style norms among americans and brazilians: assessing differences using jackson’s return potential model. California School of Professional Psychology-San Diego.
  67. Llama 2: Open foundation and fine-tuned chat models.
  68. UNICEF. 2021. Defining social norms and related concepts.
  69. Kenneth Wallen and Gerard Kyle. 2018. Extending the return potential model with a descriptive normative belief measure. Society & Natural Resources, 31:1–7.
  70. Etienne Wenger-Trayner and Beverly Wenger-Trayner. 2015. Introduction to communities of practice: A brief overview of the concept and its uses.
  71. Ex machina: Personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, WWW ’17, page 1391–1399, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  72. Understanding the diverging user trajectories in highly-related online communities during the covid-19 pandemic.
  73. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  74. How we learn social norms: a three-stage model for social norm learning. Frontiers in Psychology, 14:1153809.
  75. Dialogpt: Large-scale generative pre-training for conversational response generation.
  76. Judging llm-as-a-judge with mt-bench and chatbot arena.
Citations (3)
View on Semantic Scholar

Summary

  • The paper introduces ValueScope, a framework that quantifies implicit norms by modeling community interactions on Reddit.
  • It employs two modules, NSP and CPP, to measure norm expression and predict community response with high precision.
  • Evaluations across 13 Reddit communities reveal significant insights into norm dynamics, influencing moderation and content strategies.

ValueScope: Unveiling Implicit Norms and Values in Online Communities

The paper introduces ValueScope, a novel computational framework designed to quantify social norms and values within online communities, building upon social science perspectives and the Return Potential Model (RPM). ValueScope is tested and validated through its application to 13 Reddit communities, categorized under gender, politics, science, and finance.

Overview of ValueScope Framework

ValueScope operationalizes the RPM theory, which posits that social norms are dynamic elements shaped by community interactions, manifesting through implicit (non-categorical) dimensions such as formality, supportiveness, sarcasm, humor, politeness, and verbosity. The framework includes two key components: the Normness Scale Predictor (NSP) and the Community Preference Predictor (CPP).

  1. Normness Scale Predictor (NSP): This module consists of normness measurement and distillation processes. The measurement module, using a binary classification approach, labels pairs of comments with the extent to which they exhibit a specific norm. Labels are synthesized with GPT-3.5 and GPT-4. The distillation stage refines these measurements through a community language simulation module, ensuring that generated comments vary only along the targeted norm dimensions, thereby isolating extraneous confounding factors.
  2. Community Preference Predictor (CPP): This module estimates community reactions to comments, capturing preferences via up- and down-vote counts. Similar to NSP, the CPP includes measurement and distillation stages, ensuring that the effects of norm variations on community reactions are accurately captured by comparing original and varied comments under controlled conditions.

Evaluation of ValueScope

Evaluation involved rigorous experiments with extensive data filtering and curating processes. Various subreddits representing four primary topical groups were selected for analysis. The subreddits covered gender (r/askmen, r/askwomen, r/asktransgender), politics (r/libertarian, r/democrats, r/republican), science (r/askscience, r/shittyaskscience, r/asksciencediscussion), and finance (r/wallstreetbets, r/stocks, r/pennystocks, r/wallstreetbetsnew).

Significant findings include:

  • Normness Scale Predictor (NSP): Normness was quantifiable and predictive of community preferences. Models trained on GPT-4-generated labels achieved high accuracy across most domains, validated against human-annotated benchmarks.
  • Community Preference Predictor (CPP): The inclusion of contextual metadata (e.g., post title, time of posting) improved predictive performance, with the best configurations achieving an average accuracy of 73.9\%.

Results and Implications

By applying ValueScope to various subreddits, the authors identified nuanced behaviors and preferences unique to each community. For instance:

  • Formality Preferences in Politics: Politics subreddits uniformly showed a decline in community preference as comments became more casual, affirming community guidelines against trollish behavior.
  • Supportiveness in Science: Science subreddits favored supportive over toxic comments, aligning with their guidelines promoting respectful discourse. Spin-off subreddit r/shittyaskscience differed, reflecting its parodic nature.
  • Humor Spectrum: Finance subreddits displayed a general preference for humor, with r/wallstreetbets showing marked disapproval of serious comments, illustrative of the community's laid-back culture.
  • Temporal Changes and External Events: During the 2020 U.S. presidential election, preference for humor and supportiveness in politics subreddits increased, followed by a decline post-election. This demonstrated how external events could shift community norms.
  • User Behavior Shifts: User-level analysis showed significant variances when participating in different subreddits, especially between closely related communities like sibling subreddits and spin-offs.

Conclusion and Future Directions

ValueScope presents a pioneering approach to studying implicit social norms and values at scale within online communities. Its robust, flexible nature allows for extensive applications, including enhancing moderation tools, refining content generation models, and supporting the proactive evolution of community norms. By providing a deeper understanding of norm dynamics and their temporal shifts, ValueScope offers significant utility for social scientists, platform developers, and community moderators looking to foster positive and engaging online environments.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown