Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Key to Kindness: Reducing Toxicity In Online Discourse Through Proactive Content Moderation in a Mobile Keyboard (2401.10627v1)

Published 19 Jan 2024 in cs.HC

Abstract: Growing evidence shows that proactive content moderation supported by AI can help improve online discourse. However, we know little about designing these systems, how design impacts efficacy and user experience, and how people perceive proactive moderation across public and private platforms. We developed a mobile keyboard with built-in proactive content moderation which we tested (N=575) within a semi-functional simulation of a public and private communication platform. Where toxic content was detected, we used different interventions that embedded three design factors: timing, friction, and the presentation of the AI model output. We found moderation to be effective, regardless of the design. However, friction was a source of annoyance while prompts with no friction that occurred during typing were more effective. Follow-up interviews highlight the differences in how these systems are perceived across public and private platforms, and how they can offer more than moderation by acting as educational and communication support tools.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. N.d. Perspective API - How it works. https://www.perspectiveapi.com/how-it-works/
  2. N.d. Twitter API Documentation — Docs — Twitter Developer Platform. https://developer.twitter.com/en/docs/twitter-api
  3. Bugs in our pockets: The risks of client-side scanning. arXiv preprint arXiv:2110.07450 (2021).
  4. Alessandro Acquisti. 2004. Privacy in electronic commerce and the economics of immediate gratification. In Proceedings of the 5th ACM conference on Electronic commerce. 21–29.
  5. Understanding the effect of deplatforming on social networks. In Proceedings of the 13th ACM Web Science Conference 2021. 187–195.
  6. The effectiveness of prompts to promote engagement with digital interventions: a systematic review. Journal of medical Internet research 18, 1 (2016), e6.
  7. Birds of a feather don’t fact-check each other: Partisanship and the evaluation of news in Twitter’s Birdwatch crowdsourced fact-checking program. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19.
  8. Carolina Are. 2022. The Shadowban Cycle: an autoethnography of pole dancing, nudity and censorship on Instagram. Feminist Media Studies 22, 8 (2022), 2002–2019.
  9. Carolina Are. 2023. An autoethnography of automated powerlessness: lacking platform affordances in Instagram and TikTok account deletions. Media, Culture & Society 45, 4 (2023), 822–840.
  10. The impact of timing on the salience of smartphone app privacy notices. In Proceedings of the 5th annual ACM CCS workshop on security and privacy in smartphones and mobile devices. 63–74.
  11. Prashanth Bhat and Ofra Klein. 2020. Covert hate speech: White nationalists and dog whistle communication on twitter. Twitter, the public sphere, and the chaos of online deliberation (2020), 151–172.
  12. M Blekkenhorst. 2019. Moderating online extremism on fringe and mainstream platforms; An analysis of governance by Gab & Twitter. Master’s thesis.
  13. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
  14. Debunking: A meta-analysis of the psychological efficacy of messages countering misinformation. Psychological science 28, 11 (2017), 1531–1546.
  15. Thread With Caution: Proactively Helping Users Assess and Deescalate Tension in Their Online Discussions. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–37.
  16. Hate is not binary: Studying abusive behavior of# gamergate on twitter. In Proceedings of the 28th ACM conference on hypertext and social media. 65–74.
  17. Anyone can become a troll: Causes of trolling behavior in online discussions. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing. 1217–1230.
  18. Design frictions for mindful interactions: The case for microboundaries. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 1389–1397.
  19. Fighting Hate Speech, Silencing Drag Queens? Artificial Intelligence in Content Moderation and Risks to LGBTQ Voices Online. Sexuality & Culture 25, 2 (2021), 700–732.
  20. Judith Donath. 2007. Signals in social supernets. Journal of computer-mediated communication 13, 1 (2007), 231–251.
  21. Bryan Dosono and Bryan Semaan. 2019. Moderation practices as emotional labor in sustaining online communities: The case of AAPI identity work on Reddit. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.
  22. Brianna Dym and Casey Fiesler. 2020. Social norm vulnerability and its consequences for privacy and safety in an online community. Proceedings of the ACM on Human-Computer Interaction 4, CSCW2 (2020), 1–24.
  23. Explicit warnings reduce but do not eliminate the continued influence of misinformation. Memory & cognition 38 (2010), 1087–1100.
  24. Timing is everything? The effects of timing and placement of online privacy indicators. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 319–328.
  25. Attitudes towards Client-Side Scanning for CSAM, Terrorism, Drug Trafficking, Drug Use and Tax Evasion in Germany. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 217–233.
  26. Tarleton Gillespie. 2018. Custodians of the Internet: Platforms, content moderation, and the hidden decisions that shape social media. Yale University Press.
  27. Tarleton Gillespie. 2022. Do not recommend? Reduction as a form of content moderation. Social Media+ Society 8, 3 (2022), 20563051221117552.
  28. Algorithmic content moderation: Technical and political challenges in the automation of platform governance. Big Data & Society 7, 1 (2020), 2053951719897945.
  29. Disproportionate removals and differing content moderation experiences for conservative, transgender, and black social media users: Marginalization and moderation gray areas. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–35.
  30. AI-mediated communication: Definition, research agenda, and ethical considerations. Journal of Computer-Mediated Communication 25, 1 (2020), 89–100.
  31. Evaluating the effectiveness of deplatforming as a moderation strategy on Twitter. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–30.
  32. Does transparency in moderation really matter? User behavior after content removal explanations on reddit. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–27.
  33. Characterizing community guidelines on social media platforms. In Conference companion publication of the 2020 on computer supported cooperative work and social computing. 287–291.
  34. ’The Perfect One’ Understanding Communication Practices and Challenges with Animated GIFs. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1–20.
  35. Birago Birago Korayga Jones. 2012. Reflective interfaces: Assisting teens with stressful situations online. Ph. D. Dissertation. Massachusetts Institute of Technology.
  36. Misleading Tweets and Helpful Notes: Investigating Data Labor by Twitter Birdwatch Users. In Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing. 68–71.
  37. Reconsidering tweets: Intervening during tweet creation decreases offensive content. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 16. 477–487.
  38. Confronting abusive language online: A survey from the ethical and human rights perspective. Journal of Artificial Intelligence Research 71 (2021), 431–478.
  39. Phishing in organizations: Findings from a large-scale and long-term study. In 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 842–859.
  40. Karen Levy and Solon Barocas. 2017. Designing against discrimination in online markets. Berkeley Technology Law Journal 32, 3 (2017), 1183–1238.
  41. Renkai Ma and Yubo Kou. 2022. ” I’m not sure what difference is between their content and mine, other than the person itself” A Study of Fairness Perception of Content Moderation on YouTube. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–28.
  42. Brandeis Marshall. 2021. Algorithmic misogynoir in content moderation practice. Heinrich-Böll-Stiftung European Union (2021).
  43. Slowing it Down: Towards Facilitating Interpersonal Mindfulness in Online Polarizing Conversations Over Social Media. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1–27.
  44. Analyzing Genetic Testing Discourse on the Web Through the Lens of Twitter, Reddit, and 4chan. ACM Trans. Web 14, 4, Article 17 (aug 2020), 38 pages. https://doi.org/10.1145/3404994
  45. Accidental racists: Experiences and contradictions of racism in local Amsterdam soccer fan culture. Soccer & Society 8, 2-3 (2007), 335–350.
  46. Towards a Framework for Evaluating CSAM Prevention and Detection Tools in the Context of End-to-end encryption Environments: a Case Study. https://bpb-eu-w2.wpmucdn.com/blogs.bristol.ac.uk/dist/1/670/files/2023/02/Safety-Tech-Challenge-Fund-evaluation-framework-report.pdf
  47. Sarah T Roberts. 2014. Behind the screen: The hidden digital labor of commercial content moderation. University of Illinois at Urbana-Champaign.
  48. Sarah T Roberts. 2016. Commercial content moderation: Digital laborers’ dirty work. (2016).
  49. Paul Rosenzweig. 2020. The Law and Policy of Client-Side Scanning (Originally published by Lawfare). (2020).
  50. Is this WhatsApp conversation aggressive? Adolescents’ perception of cyber dating aggression. Journal of interpersonal violence 37, 19-20 (2022), NP17369–NP17393.
  51. A framework of severity for harmful content online. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–33.
  52. Proactive Moderation of Online Discussions: Existing Practices and the Potential for Algorithmic Support. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–27.
  53. Designing user interface elements to improve the quality and civility of discourse in online commenting behaviors. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–14.
  54. Moderator engagement and community development in the age of algorithms. New Media & Society 21, 7 (2019), 1417–1443.
  55. Informing users: Effects of notification properties and user characteristics on sharing attitudes. International Journal of Human–Computer Interaction (2022), 1–29.
  56. ” I read my Twitter the next morning and was astonished” a conversational perspective on Twitter regrets. In Proceedings of the SIGCHI conference on human factors in computing systems. 3277–3286.
  57. Erin L Spottswood and Jeffrey T Hancock. 2017. Should I share that? Prompting social norms that influence privacy behaviors on a social networking site. Journal of Computer-Mediated Communication 22, 2 (2017), 55–70.
  58. The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445092
  59. ‘Unmochon’: A Tool to Combat Online Sexual Harassment over Facebook Messenger. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–18.
  60. Processing political misinformation: Comprehending the Trump phenomenon. Royal Society open science 4, 3 (2017), 160802.
  61. Richard H Thaler and Cass R Sunstein. 2009. Nudge: Improving decisions about health, wealth, and happiness. Penguin.
  62. Is it merely a question of “what” to prompt or also “when” to prompt? The role of point of presentation time of prompts in self-regulated learning. Zeitschrift für Pädagogische Psychologie 23, 2 (2009), 105–115.
  63. ” At the End of the Day Facebook Does What ItWants” How Users Experience Contesting Algorithmic Content Moderation. Proceedings of the ACM on human-computer interaction 4, CSCW2 (2020), 1–22.
  64. Measurements, algorithms, and presentations of reality: Framing interactions with AI-enabled decision support. ACM Transactions on Computer-Human Interaction 30, 2 (2023), 1–33.
  65. “Thinking before posting?” Reducing cyber harassment on social networking sites through a reflective message. Computers in human behavior 66 (2017), 345–352.
  66. J Vincent. 2020. Twitter is Bringing its ‘read before you retweet’ prompt to all users. The Verge. Retrieved October 28 (2020), 2020.
  67. A field trial of privacy nudges for facebook. In Proceedings of the SIGCHI conference on human factors in computing systems. 2367–2376.
  68. From facebook regrets to facebook privacy nudges. Ohio St. LJ 74 (2013), 1307.
  69. Privacy nudges for social media: an exploratory Facebook study. In Proceedings of the 22nd international conference on world wide web. 763–770.
  70. ” I regretted the minute I pressed share” a qualitative study of regrets on Facebook. In Proceedings of the seventh symposium on usable privacy and security. 1–16.
  71. Privacy unraveling around explicit HIV status disclosure fields in the online geosocial hookup app Grindr. Proceedings of the ACM on human-computer interaction 2, CSCW (2018), 1–22.
  72. “Oops…”: Mobile Message Deletion in Conversation Error and Regret Remediation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.
  73. Sarah Wiseman and Sandy JJ Gould. 2018. Repurposing emoji for personalised communication: Why [pizza] means “I love you”. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–10.
  74. Savvas Zannettou. 2021. ” I Won the Election!”: An Empirical Analysis of Soft Moderation Interventions on Twitter. In Proceedings of the International AAAI Conference on Web and Social Media, Vol. 15. 865–876.
  75. On the origins of memes by means of fringe web communities. In Proceedings of the internet measurement conference 2018. 188–202.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets