Computer-Aided Tagging on Wikimedia Commons: Designing for Human-AI Collaboration in Open Knowledge Work

Abstract: This study investigates Wikimedia Commons contributors' lived experiences with the Computer-Aided Tagging (CAT) tool, an AI-assisted image tagging system designed to improve Commons' discoverability, searchability, accessibility, and multilingual support. Using a qualitative analysis of 595 CAT-related community comments from 11 wiki pages and 16 in-depth interviews, we identify seven key issues that contributed to CAT's mixed reception and eventual deactivation. We also offer community-informed suggestions for improving the tool. We reflect on the implications for designing human-AI collaboration on Commons and for developing AI-assisted tools that support open knowledge work. This work contributes to HCI and CSCW research by extending the understanding of human-AI collaboration beyond Anglophone, text-centric, corporate platforms.

• The paper demonstrates that community-driven validation of AI-suggested tags is crucial for quality and system adoption.
• It employs qualitative analysis of 595 comments and 16 interviews to uncover misalignments between system design and community values.
• The study highlights the need for collaborative, context-aware AI tools that integrate legacy metadata with structured data standards.

Computer-Aided Tagging on Wikimedia Commons: Empirical Insights into Human–AI Collaboration in Open Knowledge Work

Introduction

The paper "Computer-Aided Tagging on Wikimedia Commons: Designing for Human–AI Collaboration in Open Knowledge Work" (2605.30800) presents one of the most comprehensive empirical studies of an AI-assisted image tagging system—Computer-Aided Tagging (CAT)—within the Wikimedia Commons ecosystem. By integrating qualitative analysis of 595 community comments and 16 in-depth interviews, the study investigates the lifecycle, reception, and deactivation of the CAT tool, drawing generalizable insights for Collaborative and Social Computing (CSCW) and Human-Computer Interaction (HCI).

Background: Metadata, AI Tagging, and Commons

Wikimedia Commons operates as a decentralized, global-scale repository for multimedia content, historically organized via a social, hierarchical category system. While vital for distributed sensemaking, this folksonomic approach lacks a structured, machine-readable schema, impeding semantic search and multilingual accessibility. To address these deficits, the Structured Data on Commons (SDC) initiative introduced ontology-linked property–value statements—anchored in Wikidata—that function as a semantic metadata layer. The CAT tool was engineered as a human-in-the-loop image tagging system to facilitate community adoption of SDC, providing AI-generated, Wikidata-linked tagging suggestions for human review and approval.

Figure 1: The SDC interface supplements traditional, unstructured wikitext metadata (right) with structured, ontology-linked metadata (left).

This system represents a technical and epistemic shift: contributors validate AI-suggested tags (Depicts statements), which are expressed as Wikidata Q-IDs, thereby enabling machine- and language-independent semantic search.

Figure 2: The Structured Data tab exposes Depicts statements (left), in direct comparison to traditional, plaintext category markup (right).

CAT leverages the Google Cloud Vision API to generate concept candidates, mapped to Wikidata IDs through Freebase correspondences. Critically, the human validation phase acts as both quality control and editorial authorship, reifying the collaborative goal of "content creation" over mere moderation.

Figure 3: The CAT user interface shows AI-suggested tags beneath the image, awaiting human verification before being written as structured Depicts statements.

Through language-agnostic Q-IDs, CAT aims to enable true multilingual authoring and search, supporting Wikimedia’s polyglot user base.

Figure 4: The SDC interface demonstrates automatic multilingual rendering of the same Depicts statements, supporting interface localization for over 300 language editions.

Methodology

The paper’s methodological rigor lies in its integration of large-scale community textual analysis with semi-structured, in-depth interviews. Thematic analysis was applied to 595 comments across 11 wiki discussion pages, yielding a taxonomy of the Commons community’s concerns and lived experiences with CAT. This qualitative map was deepened through interviews with 16 selected power users, spanning engagement cohorts from contributors with 25,000 to over 600,000 edits and supporting a diversity of perspectives on category and SDC workflows.

Key Findings: Barrier Analysis of CAT Deployment

1. Misalignment of Initiative Goals Versus Editorial Values

A fundamental divergence emerged between the Wikimedia Foundation’s (WMF) SDC/CAT vision and the Commons editing community’s values. The Foundation prioritized semantic search, metadata interoperability, and multilingualism; core editors expressed that overgeneralized search undermines Commons' role as an educational, context-rich resource distinct from generic stock photo repositories. Participants highlighted a lack of transparent communication and insufficient consensus-building processes as critical precursors to CAT’s reception problems.

2. Ambiguity and Policy Vacuum for Depicts Statements

Contradictory practices and definitions circulated around the Depicts statement. The absence of formalized editorial guidelines for Depicts led to inconsistent granularity (e.g., "animal" vs. "Putnam's jumping spider") and recurring "edit wars" over ontological precision. Even experienced editors reported reliance on personal or group heuristics rather than shared standards, in sharp contrast to the relatively settled norms of category assignment.

3. Usability Friction and Gamified Tagging

The CAT interface’s encouragement of high-volume tagging, especially among newcomers, resulted in systematic misapplication—users would uncritically confirm suggested tags (e.g., “depicts green”) with little understanding of structured data best practices. This incentive misalignment generated significant data quality noise, requiring retrospective correction by more experienced editors. Interviewees advocated for guided onboarding, including negative feedback and example-based training to resolve these issues.

4. Isolation from Category System and Knowledge Loss

Despite SDC’s technical promise, the parallel existence of categories and structured metadata (with minimal cross-system integration) introduced redundancy and overlooked two decades of community-curated, context-rich category knowledge. Editors emphasized the necessity of leveraging, rather than bypassing, the legacy category system when designing future AI-powered structured tagging.

5. Ill-Specified ML Objectives

CAT’s underlying vision model—a general-purpose, web-scale classifier—performed poorly with specificity, often returning context-free or culturally generic tags. Editors underscored the importance of mission-oriented ML: constrained, domain-specific models for tasks like plant species identification, text recognition in flags, or mass tagging of highly homogeneous corpora (e.g., firework colorations). The misalignment between the AI’s generalist architecture and community curation goals was a primary axis of tool dissatisfaction.

6. Absence of Collaborative, Iterative Tagging

Contrary to CAT’s workflow assumption that tagging is an individually finalized act, editorial practice within Commons is highly collaborative and iterative, with contributors expecting peer review and correction over time. The lack of interface affordances for shared, deliberative tagging (e.g., collaborative evaluation, justification, or incremental refinement) constituted a major process breakdown.

7. Disconnect from Search UX

Despite SDC’s theoretical support for advanced semantic queries, the realized search experience on Commons did not leverage new structured metadata capabilities. Editors often resorted to redundant, overly broad tag inclusion to improve retrievability, effectively reconstituting folksonomic duplication within the SDC layer. Participants highlighted the urgent need for redesigning search to semantically reason over SDC/Wikidata relationships instead of relying on rudimentary string-matching.

Implications for HCI, CSCW, and Open Knowledge AI

The study yields several implications for future tools and AI integration in decentralized, multilingual, community-governed knowledge systems:

• Consensus-Driven Design: Structured data initiatives require joint, transparent, and iterative alignment processes involving the editorial community, especially for foundational constructs like Depicts.
• Participatory AI/ML: Successful deployment of assistive AI should move toward participatory co-design, community-curated training datasets, and problem-specific classifiers, with public interfaces for contestation and negotiation over model outputs.
• Legacy System Integration: Rather than discarding established socio-technical infrastructure, AI-powered systems should ingest and reconcile knowledge from legacy tagging and category schemas.
• Multilingual and Cultural Specificity: Vision models must avoid Western-centric bias and flattening, instead combining pixel-based inference with existing multilingual, contributor-written metadata.
• Human-Centric Tag Validation: AI should explicitly support ambiguity, communicate model uncertainties, and provide collaborative mechanisms for peer review and sense-making, rather than simply seeking consensus-elimination or universal applicability.
• Search–Metadata Feedback Loops: AI-enhanced search and retrieval must close the feedback loop with tagging practices, aligning metadata policies and structured data architectures to downstream retrieval affordances.

Strong Results and Contradictory Claims

• CAT’s deactivation is a central empirical observation, resulting from sustained community rejection rather than technical infeasibility, highlighting the primacy of human factors in civic AI deployments.
• Editors repeatedly challenge the assumption that more tags, or more "generic" search, equates to increased value—contradicting objectives common in corporate, engagement-driven platforms.
• Direct content creation via AI assistance requires fundamentally distinct socio-technical and design considerations from moderation/coordination tasks, contesting assumptions from text-centric AI-for-moderation work.

Future Directions and Theoretical Contributions

Beyond practical adjustments to SDC/CAT, the study broadens the research horizon for human–AI collaboration in open, multilingual, knowledge commons. It demonstrates the need for sociotechnical frameworks capable of dynamic consensus, participatory machine learning, and human-in-the-loop ambiguity navigation. Addressing language, culture, and epistemic context becomes central when deploying AI in Wikimedia-scale, globally diverse platforms.

Conclusion

By thoroughly dissecting the CAT tool's deployment on Wikimedia Commons, the paper contributes substantial empirical evidence on the failure modes of generic, top-down AI interventions in community-curated, knowledge-centric ecosystems. The findings underscore that algorithmic augmentation of open knowledge work is only effective if rooted in community context, consensus, and participatory governance. The proposed human–AI collaboration paradigm must be domain- and context-aware, semantically integrated with legacy systems, and designed for ambiguity navigation—not simply automation or acceleration. The insights gained are broadly applicable to the intersection of AI, structured metadata, and open source sociotechnical systems.