Papers
Topics
Authors
Recent
2000 character limit reached

OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence Investigations (2401.00928v1)

Published 1 Jan 2024 in cs.HC

Abstract: Open Source Intelligence (OSINT) investigations, which rely entirely on publicly available data such as social media, play an increasingly important role in solving crimes and holding governments accountable. The growing volume of data and complex nature of tasks, however, means there is a pressing need to scale and speed up OSINT investigations. Expert-led crowdsourcing approaches show promise but tend to either focus on narrow tasks or domains or require resource-intense, long-term relationships between expert investigators and crowds. We address this gap by providing a flexible framework that enables investigators across domains to enlist crowdsourced support for the discovery and verification of OSINT. We use a design-based research (DBR) approach to develop OSINT Research Studios (ORS), a sociotechnical system in which novice crowds are trained to support professional investigators with complex OSINT investigations. Through our qualitative evaluation, we found that ORS facilitates ethical and effective OSINT investigations across multiple domains. We also discuss broader implications of expert-crowd collaboration and opportunities for future work.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (139)
  1. 2015. A Call to Arms: Open Source Intelligence and Evidence Based Policymaking. https://www.bellingcat.com/resources/articles/2015/01/20/a-call-to-arms-open-source-intelligence-and-evidence-based-policymaking/
  2. 2021. J298 OSINT Seminar — Open Source Investigations. https://journalism.berkeley.edu/course-section/j298-human-rights-center-seminar-f21/
  3. 2021. WhatsApp can be a black box of misinformation, but Maldita may have opened a window. https://www.poynter.org/fact-checking/2021/whatsapp-can-be-a-black-box-of-misinformation-but-maldita-may-have-opened-a-window/
  4. 2022. Hunchly - OSINT Software for Cybersecurity, Law Enforcement, Journalists, Private Investigators, and more. https://www.hunch.ly/
  5. 2022. Russia is losing so much equipment in Ukraine that weapons monitors can’t keep up. https://www.independent.co.uk/news/world/europe/russia-ukraine-military-equipment-losses-b2049613.html Section: News.
  6. 2022. TraceLabs Twitter Account. https://twitter.com/tracelabs/status/1558831625986777088
  7. 2023. About Bellingcat. https://www.bellingcat.com/about/
  8. 2023. archive.is. https://archive.is/
  9. 2023. Bot Sentinel - Dashboard. https://botsentinel.com/
  10. 2023. Certified in Open Source Intelligence (C|OSINT) from McAfee Institute | NICCS. https://niccs.cisa.gov/education-training/catalog/mcafee-institute/certified-open-source-intelligence-cosint
  11. 2023. Check (@checkdesk) / Twitter. https://twitter.com/checkdesk
  12. 2023. EXIF Data Viewer. https://exifdata.com/
  13. 2023. Google Images. https://images.google.com/
  14. 2023. Join us in pushing back on misinformation. https://our.news/
  15. 2023a. Newsgathering and Monitoring on the Social Web. https://firstdraftnews.org:443/long-form-article/newsgathering-and-monitoring-on-the-social-web/
  16. 2023. OSINT: Open-Source Intelligence. https://www.udemy.com/course/osint-open-source-intelligence/
  17. 2023. Practical Open-Source Intelligence (OSINT) Training — SANS SEC497. https://www.sans.org/cyber-security-courses/practical-open-source-intelligence/
  18. 2023. Search Party Rules. https://www.tracelabs.org/about/search-party-rules
  19. 2023. Syrian Archive | Syrian Archive. https://syrianarchive.org/
  20. 2023. TinEye Reverse Image Search. https://tineye.com/
  21. 2023b. Verifying Online Information. https://firstdraftnews.org:443/long-form-article/verifying-online-information/
  22. 2023. Yandex. https://yandex.com/
  23. OSINT Explorer: A Tool Recommender Framework for OSINT Sources. In Advances in Cyber Security (Communications in Computer and Information Science), Nibras Abdullah, Selvakumar Manickam, and Mohammed Anbar (Eds.). Springer, Singapore, 389–400. https://doi.org/10.1007/978-981-16-8059-5_24
  24. Crowdsourcing in the field: A case study using local crowds for event reporting. In Third AAAI Conference on Human Computation and Crowdsourcing. Association for the Advancement of Artificial Intelligence, 11. https://www.microsoft.com/en-us/research/publication/crowdsourcing-in-the-field-a-case-study-using-local-crowds-for-event-reporting/
  25. Mobile Collaboration for Human and Canine Police Explosive Detection Teams. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 925–933. https://doi.org/10.1145/2998181.2998271
  26. An Activity Theory Analysis of Search and; Rescue Collective Sensemaking and Planning Practices. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 146, 20 pages. https://doi.org/10.1145/3411764.3445272
  27. CAPER: Collaborative Information, Acquisition, Processing, Exploitation and Reporting for the Prevention of Organised Crime. Communications in Computer and Information Science (2014), 6.
  28. Scaling up fact-checking using the wisdom of crowds. Science advances 7, 36 (2021), eabf4393.
  29. Birds of a feather don’t fact-check each other: Partisanship and the evaluation of news in Twitter’s Birdwatch crowdsourced fact-checking program. In CHI Conference on Human Factors in Computing Systems. PsyArXiv, 1–19. https://doi.org/10.31234/osf.io/57e3q
  30. Data Migrations: Exploring the Use of Social Media Data as Evidence for Human Rights Advocacy. Proc. ACM Hum.-Comput. Interact. 4, CSCW3, Article 268 (jan 2021), 25 pages. https://doi.org/10.1145/3434177
  31. Adriana Alvarado Garcia and Christopher A. Le Dantec. 2018. Quotidian Report: Grassroots Data Practices to Address Public Safety. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 17 (nov 2018), 18 pages. https://doi.org/10.1145/3274286
  32. Amnesty International. 2020. Syria: ’Nowhere is Safe for Us’: Unlawful Attacks and Mass Displacement in North-West Syria. Technical Report. https://www.amnesty.org/en/documents/document/?indexNumber=MDE24%2f2089%2f2020&language=en
  33. A Closer Look at the Self-Correcting Crowd: Examining Corrections in Online Rumors. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17). ACM, Portland Oregon USA, 155–168. https://doi.org/10.1145/2998181.2998294
  34. Anne Aula and Daniel M Russell. 2008. Complex and exploratory web search. In Information Seeking Support Systems Workshop (ISSS 2008), Chapel Hill, NC, USA. Citeseer.
  35. Charlie Beckett. 2017. Wikitribune: can crowd-sourced journalism solve the crisis of trust in news? POLIS: journalism and society at the LSE (2017).
  36. Compete, Collaborate, Investigate: Exploring the Social Structures of Open Source Intelligence Investigations. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI ’22). Association for Computing Machinery, New York, NY, USA, 1–18. https://doi.org/10.1145/3491102.3517526
  37. Bellingcat Investigation Team. 2015. Diversifying OSINT: Women Experts. https://www.bellingcat.com/resources/articles/2015/12/08/women-in-osint-diversifying-the-field/
  38. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313–322.
  39. Turning the spotlight on the crowd: Examining the participatory ethics and practices of crisis mapping. ACME: An International Journal for Critical Geographies 15, 1 (2016), 207–229.
  40. Social media sourcing practices: How Dutch newspapers use tweets in political news coverage. Managing democracy in the digital age: Internet regulation, social media use, and online civic engagement (2018), 159–178.
  41. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (Jan. 2006), 77–101. https://doi.org/10.1191/1478088706qp063oa Publisher: Routledge _eprint: https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa.
  42. The use of social media in the recruitment process. (2014). 81 (2014).
  43. Chen Cao. 2023. Leveraging Large Language Model and Story-Based Gamification in Intelligent Tutoring System to Scaffold Introductory Programming Courses: A Design-Based Research Study. arXiv preprint arXiv:2302.12834 (2023).
  44. Deb8: A tool for collaborative analysis of video. In Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video. 47–58.
  45. Southern Poverty Law Center. 2021. The Road to Jan. 6: A Year of Extremist Mobilization. https://www.splcenter.org/news/2021/12/30/road-jan-6-year-extremist-mobilization
  46. SOLVENT: A Mixed Initiative System for Finding Analogies Between Research Papers. 2 (2018), 31:1–31:21. Issue CSCW. https://doi.org/10.1145/3274300
  47. Design experiments in educational research. Educational researcher 32, 1 (2003), 9–13.
  48. Josie Cochrane. 2022. Citizen OSINT Analysts: Motivations of Open-Source Intelligence Volunteers.
  49. Designing for Hybrid Intelligence: A Taxonomy and Survey of Crowd-Machine Interaction. Applied Sciences 13, 4 (2023), 2198.
  50. Joseph Cox. 2018. The Hackers Hunting Down Missing People: Nonprofit TraceLabs ran DEF CON’s first crowdsourced event for tracking missing people through public information. Vice (2018). https://www.vice.com/en_us/article/qvmm3x/hackers-hunting-missing-people-osint-defcon-tracelabs
  51. Dharma Dailey and Kate Starbird. 2014. Journalists as Crowdsourcerers: Responding to Crisis by Reporting with a Crowd. Computer Supported Cooperative Work (CSCW) 23, 4 (01 Dec 2014), 445–481. https://doi.org/10.1007/s10606-014-9208-z
  52. Dharma Dailey and Kate Starbird. 2015. ”It’s Raining Dispersants”: Collective Sensemaking of Complex Information in Crisis Contexts. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work and; Social Computing (Vancouver, BC, Canada) (CSCW’15 Companion). Association for Computing Machinery, New York, NY, USA, 155–158. https://doi.org/10.1145/2685553.2698995
  53. Towards Understanding and Supporting Journalistic Practices Using Semi-Automated News Discovery Tools. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (Oct. 2021), 406:1–406:30. https://doi.org/10.1145/3479550
  54. Toward a Learning Science for Complex Crowdsourcing Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, New York, NY, USA, 2623–2634. https://doi.org/10.1145/2858036.2858268
  55. Shepherding the Crowd Yields Better Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1013–1022. https://doi.org/10.1145/2145204.2145355
  56. Digital Witness: Using Open Source Information for Human Rights Investigation, Documentation, and Accountability. Oxford University Press, Oxford, New York.
  57. Design-based research process: Problems, phases, and applications. Boulder, CO: International Society of the Learning Sciences.
  58. Sheena L. Erete. 2015. Engaging Around Neighborhood Issues: How Online Communication Affects Offline Behavior. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 1590–1601. https://doi.org/10.1145/2675133.2675182
  59. Esteban Borges. 2019. SecurityTrails | OSINT Framework: The Perfect Cybersecurity Intel Gathering Tool. https://securitytrails.com/blog/osint-framework
  60. Giancarlo Fiorella. 2021. First Steps to Getting Started in Open Source Research. https://www.bellingcat.com/resources/2021/11/09/first-steps-to-getting-started-in-open-source-research/
  61. Distributed Sensemaking: Improving Sensemaking by Leveraging the Efforts of Previous Users. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA) (CHI ’12). ACM, New York, NY, USA, 247–256. https://doi.org/10.1145/2207676.2207711
  62. Measuring the reach of” fake news” and online disinformation in Europe. Australasian Policing 10, 2 (2018).
  63. Datavoidant: An AI System for Addressing Political Data Voids on Social Media. Proceedings of the ACM on Human-Computer Interaction 6, CSCW2 (2022), 1–29.
  64. Open Source Intelligence and AI: a Systematic Review of the GELSI Literature. AI & Society (2022).
  65. Michael Glassman and Min Ju Kang. 2012. Intelligence in the internet age: The emergence and evolution of Open Source Intelligence (OSINT). Computers in Human Behavior 28, 2 (March 2012), 673–682. https://doi.org/10.1016/j.chb.2011.11.014
  66. Moderating with the mob: Evaluating the efficacy of real-time crowdsourced fact-checking. Journal of Online Trust and Safety 1, 1 (2021).
  67. A Social Network Engaged Crowdsourcing Framework for Expert Tasks. In 2019 IEEE 23rd International Conference on Computer Supported Cooperative Work in Design (CSCWD). IEEE, 249–254. https://doi.org/10.1109/CSCWD.2019.8791923
  68. Catherine Grevet and Eric Gilbert. 2015. Piggyback prototyping: Using existing, large-scale social computing systems to prototype new ones. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 4047–4056.
  69. Pablo Gutierrez and Paul Torpey. 2015. How Digital Detectives Say They Proved Ukraine Attacks Came from Russia. The Guardian (2015).
  70. The Knowledge Accelerator: Big Picture Thinking in Small Pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (New York, NY, USA) (CHI ’16). ACM, 2258–2270. http://doi.acm.org/10.1145/2858036.2858364
  71. Melissa Hanham and Jaewoo Shin. 2020. Ethics in the Age of OSINT Innocence. Stanley Center for Peace and Security (May 2020), 6. https://stanleycenter.org/publications/ethics-osint-innocence/
  72. Joining together online: the trajectory of CSCW scholarship on group formation. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–27.
  73. ClaimBuster: the first-ever end-to-end fact-checking system. Proceedings of the VLDB Endowment 10, 12 (Aug. 2017), 1945–1948. https://doi.org/10.14778/3137765.3137815
  74. Annique Mossou Higgins, Ross. 2021a. A Beginner’s Guide to Social Media Verification. https://www.bellingcat.com/resources/2021/11/01/a-beginners-guide-to-social-media-verification/
  75. Eliot Higgins. 2021b. We Are Bellingcat : An Intelligence Agency for the People. Bloomsbury Publishing, London.
  76. The Web Archives Workbench (WAW) Tool Suite: Taking an Archival Approach to the Preservation of Web Content. Library Trends 57, 3 (2009), 442–460. https://doi.org/10.1353/lib.0.0046 Publisher: Johns Hopkins University Press.
  77. Connected Through Crisis: Emotional Proximity and the Spread of Misinformation Online. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). Association for Computing Machinery, New York, NY, USA, 969–980. https://doi.org/10.1145/2675133.2675202
  78. Arthur S. Hulnick. 2002. The Downside of Open Source Intelligence. International Journal of Intelligence and CounterIntelligence 15, 4 (Nov. 2002), 565–579. https://doi.org/10.1080/08850600290101767
  79. Early Usability Evaluation to Enhance User Interfaces-A Use Case on the Yggdrasil Cybersecurity Mockup.. In RoCHI. 103–110.
  80. Objects of violence: synthetic data for practical ML in human rights investigations. arXiv:2004.01030 [cs] (April 2020), 12. http://arxiv.org/abs/2004.01030 arXiv: 2004.01030.
  81. Toward a fine-grained evaluation of the Pwnable CTF. In Information Security Applications: 21st International Conference, WISA 2020, Jeju Island, South Korea, August 26–28, 2020, Revised Selected Papers. Springer, 179–190.
  82. Hit-or-wait: Coordinating opportunistic low-effort contributions to achieve global outcomes in on-the-go crowdsourcing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
  83. Standing on the Schemas of Giants: Socially Augmented Information Foraging. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (New York, NY, USA) (CSCW ’14). ACM, 999–1010. https://doi.org/10.1145/2531602.2531644
  84. CrowdForge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology (New York, NY, USA) (UIST ’11). ACM, 43–52. https://doi.org/10.1145/2047196.2047202
  85. Alice Kolb and David Kolb. 2017. Experiential Learning Theory as a Guide for Experiential Educators in Higher Education. Experiential Learning & Teaching in Higher Education 1, 1 (June 2017), 7–44. https://nsuworks.nova.edu/elthe/vol1/iss1/7
  86. Meryl Kornfield. 2021. The wrong ID: Retired firefighter, comedian and Chuck Norris falsely accused of being Capitol rioters. Washington Post. https://www.washingtonpost.com/technology/2021/01/16/sleuths-falsely-identify-rioters/
  87. Jean Lave and Etienne Wenger. 1991. Situated Learning: Legitimate Peripheral Participation. Cambridge University Press, Cambridge, UK. Google-Books-ID: CAVIOrW3vYAC.
  88. CrowdIA: Solving Mysteries with Crowdsourced Sensemaking. 2 (2018), 105:1–105:29. Issue CSCW. https://doi.org/10.1145/3274374
  89. Dropping the baton? Understanding errors and bottlenecks in a crowdsourced sensemaking pipeline. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–26.
  90. DejaVu: A System for Journalists to Collaboratively Address Visual Misinformation.
  91. On the Misinformation Beat: Understanding the Work of Investigative Journalists Reporting on Problematic Information Online. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 133 (Oct. 2020), 22 pages. https://doi.org/10.1145/3415204
  92. Investigating people: a qualitative analysis of the search behaviours of open-source intelligence analysts. In Proceedings of the 5th Information Interaction in Context Symposium (IIiX ’14). Association for Computing Machinery, New York, NY, USA, 175–184. https://doi.org/10.1145/2637002.2637023
  93. Stephen C. Mercado. 2004. Sailing the sea of OSINT in the information age. Technical Report. dataset, American Psychological Association. type. https://doi.org/10.1037/e741272011-005
  94. Panagiotis Metaxas and Samantha T Finn. 2017. The infamous# Pizzagate conspiracy theory: Insight from a TwitterTrails investigation. (2017).
  95. True or False: Studying the Work Practices of Professional Fact-Checkers. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1–44.
  96. Comparing person-and process-centric strategies for obtaining quality data on amazon mechanical turk. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1345–1354.
  97. Nina C. Müller and Jenny Wiik. 2021. From Gatekeeper to Gate-opener: Open-Source Spaces in Investigative Journalism. Journalism Practice 0, 0 (May 2021), 1–20. https://doi.org/10.1080/17512786.2021.1919543 Publisher: Routledge _eprint: https://doi.org/10.1080/17512786.2021.1919543.
  98. Digilantism: An analysis of crowdsourcing and the Boston marathon bombings. The British journal of criminology 57, 2 (March 2017), 341–361. https://doi.org/10.1093/bjc/azv118
  99. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 1–12.
  100. Department of Homeland Security. 2010. (U//FOUO//LES) DHS Terrorist Use of Social Networking Facebook Case Study | Public Intelligence. https://publicintelligence.net/ufouoles-dhs-terrorist-use-of-social-networking-facebook-case-study/
  101. Massachusetts Institute of Technology. 2023. Urban Cyber Defense: Cybersecurity Clinic. http://urbancyberdefense.mit.edu/cybersecurityclinic Accessed: 2023-10-02.
  102. Evaluating a worker in performing crowd sourced tasks and providing in-task training through programmatically generated test tasks. US Patent 8,554,605.
  103. Crowdsourcing from Scratch: A Pragmatic Experiment in Data Collection by Novice Requesters. In Third AAAI Conference on Human Computation and Crowdsourcing. https://www.aaai.org/ocs/index.php/HCOMP/HCOMP15/paper/view/11582
  104. Gordon Pennycook and David G. Rand. 2019. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences 116, 7 (Feb. 2019), 2521–2526. https://doi.org/10.1073/pnas.1806781116 Publisher: National Academy of Sciences Section: Social Sciences.
  105. Paul R Pintrich. 2004. A conceptual framework for assessing motivation and self-regulated learning in college students. Educational psychology review 16, 4 (2004), 385–407.
  106. Tjeerd Plomp et al. 2013. Educational design research: An introduction. Educational design research (2013), 11–50.
  107. As if being there: mediated reality for crime scene investigation. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW ’12). Association for Computing Machinery, New York, NY, USA, 1267–1276. https://doi.org/10.1145/2145204.2145394
  108. Amy Rechkemmer and Ming Yin. 2020. Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing 8 (Oct. 2020), 122–131. https://doi.org/10.1609/hcomp.v8i1.7470
  109. No Workflow Can Ever Be Enough: How Crowdsourcing Workflows Constrain Complex Work. Proc. ACM Hum.-Comput. Interact. 1, CSCW, Article 89 (Dec. 2017), 23 pages. https://doi.org/10.1145/3134724
  110. Daniel Rouach and Patrice Santi. 2001. Competitive Intelligence Adds Value:: Five Intelligence Attitudes. European Management Journal 19, 5 (Oct. 2001), 552–559. https://doi.org/10.1016/S0263-2373(01)00069-X
  111. Ryan Hunt. 2012. Thirty-Seven Percent of Companies Use Social Networks to Research Potential Job Candidates, According to New CareerBuilder Survey - Apr 18, 2012. http://press.careerbuilder.com/2012-04-18-Thirty-Seven-Percent-of-Companies-Use-Social-Networks-to-Research-Potential-Job-Candidates-According-to-New-CareerBuilder-Survey
  112. Crowdsourced Fact-Checking at Twitter: How Does the Crowd Compare With Experts?. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 1736–1746.
  113. Edward Schumacher-Matos. 2012. Election 1: Fact Checking The NPR Fact Checkers. NPR (Oct. 2012). https://www.npr.org/sections/publiceditor/2012/10/28/161839145/election-1-fact-checking-the-npr-fact-checkers
  114. Ricky Sethi and Raghuram Rangaraju. 2018. Extinguishing the backfire effect: using emotions in online social collaborative argumentation for fact checking. In 2018 IEEE International conference on web services (ICWS). IEEE, 363–366.
  115. Hoaxy: A Platform for Tracking Online Misinformation. In Proceedings of the 25th International Conference Companion on World Wide Web (WWW ’16 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 745–750. https://doi.org/10.1145/2872518.2890098
  116. Craig Silverman. 2013. Verification Handbook: A Definitive Guide to Verifying Digital Content for Emergency Coverage. http://verificationhandbook.com/
  117. The Psychological Well-Being of Content Moderators: The Emotional Labor of Commercial Moderation and Avenues for Improving Support. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 341, 14 pages. https://doi.org/10.1145/3411764.3445092
  118. Atelier: Repurposing Expert Crowdsourcing Tasks As Micro-internships. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New York, NY, USA, 2645–2656. https://doi.org/10.1145/2858036.2858121
  119. Yla Tausczik and Mark Boons. 2018. Distributed Knowledge in Crowds: Crowd Performance on Hidden Profile Tasks. In Twelfth International AAAI Conference on Web and Social Media. https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17817
  120. Qualitative and quantitative feedback in the context of competency-based education. Medical teacher 39, 12 (2017), 1245–1249.
  121. Daniel Trottier. 2017. Digital vigilantism as weaponisation of visibility. Philosophy & Technology 30, 1 (March 2017), 55–72. https://doi.org/10.1007/s13347-016-0216-4
  122. Roli Varma. 2010. Why so few women enroll in computing? Gender and ethnic differences in students’ perception. Computer Science Education 20, 4 (2010), 301–316.
  123. Sukrit Venkatagiri. 2022. Supporting and Transforming High-Stakes Investigations with Expert-Led Crowdsourcing. Ph. D. Dissertation. Virginia Tech.
  124. CrowdSolve: Managing Tensions in an Expert-Led Crowdsourced Investigation. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 118 (April 2021), 30 pages. https://doi.org/10.1145/3449192
  125. CoSINT: Designing a Collaborative Capture the Flag Competition to Investigate Misinformation. In Proceedings of the 2023 ACM Designing Interactive Systems Conference (Pittsburgh, PA, USA) (DIS ’23). Association for Computing Machinery, New York, NY, USA, 2551–2572. https://doi.org/10.1145/3563657.3595997
  126. GroundTruth: Augmenting Expert Image Geolocation with Crowdsourcing and Shared Representations. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 107 (Nov. 2019), 30 pages. https://doi.org/10.1145/3359209
  127. Co-located Collaborative Sensemaking on a Large High-Resolution Display with Multiple Input Devices. In Human-Computer Interaction – INTERACT 2011. Springer Berlin Heidelberg, Berlin, Heidelberg, 589–604.
  128. Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History. Proc. ACM Hum.-Comput. Interact. 2, CSCW (Nov. 2018), 178:1–178:24. https://doi.org/10.1145/3274447
  129. Claire Wardle. 2014. Verifying user-generated content. Verification handbook: A definitive guide to verifying digital content for emergency coverage (2014), 24–33.
  130. Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policymaking.
  131. Etienne Wenger et al. 1998. Communities of practice: Learning as a social system. Systems thinker 9, 5 (1998), 2–3.
  132. Digital mobilization in disaster response: the work & self-organization of on-line pet advocates in response to hurricane sandy. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 866–876.
  133. Heather J. Williams and Ilana Blum. 2018a. Defining Second Generation Open Source Intelligence (OSINT) for the Defense Enterprise. Technical Report. RAND Corporation. https://www.rand.org/pubs/research_reports/RR1964.html
  134. Heather J. Williams and Ilana Blum. 2018b. Defining second generation open source intelligence (OSINT) for the defense enterprise. Technical Report. RAND Corporation Santa Monica United States.
  135. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. 379–388.
  136. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 1637–1648.
  137. Agile research studios: Orchestrating communities of practice to advance research training. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 220–232.
  138. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1445–1455.
  139. H. Akın Ünver. 2018. Digital Open Source Intelligence and International Security: A Primer. Technical Report. Centre for Economics and Foreign Policy Studies. https://www.jstor.org/stable/resrep21048
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Video Overview

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up