Algorithmic Auditing: Participatory Quantitative Insights

• Algorithmic auditing is the systematic evaluation of automated decision systems for fairness, bias mitigation, transparency, and interpretability.
• The participatory audit workflow employs structured scoping, data collection, annotation, and reporting to enable non-expert and expert collaboration.
• Quantitative metrics and non-expert annotations validate outcomes by replicating expert findings while surfacing novel harm dimensions like age bias.

Algorithmic auditing is the systematic evaluation of automated decision systems (ADS) and AI/ML tools against defined criteria such as fairness, bias mitigation, transparency, social impact, and interpretability. While traditionally the domain of expert practitioners, recent research has demonstrated the feasibility of participatory and non-expert-led auditing, as exemplified in comprehensive studies involving adolescent auditors of AI systems (Morales-Navarro et al., 6 Aug 2025). Algorithmic auditing is now central to AI accountability and is increasingly recognized as a mechanism for not only surfacing technical flaws but also engaging broader communities in the critique and improvement of sociotechnical systems.

1. Process Architecture: End-to-End Participatory Audit Workflow

Morales-Navarro et al. (Morales-Navarro et al., 6 Aug 2025) provide an empirical template for participatory algorithmic audits, detailing a five-phase structure applied in a real-world workshop with teenagers auditing a generative model for TikTok’s Effect House.

1. Scoping & Hypothesis Formation: Audit teams formulate the central hypothesis, often through structured brainstorming and collaborative synthesis (e.g., identifying that a generative model reinforces race and gender occupation stereotypes). Visualization of the audit steps (hypothesis, data, testing, analysis, reporting) scaffolds the process for novice auditors.
2. Data Generation, Sampling, and Collection:
   • Auditors identify audit targets (e.g., 25 occupations) and design prompts (four templates per occupation).
   • Input diversity is ensured by curating a matrix of input profiles (built-in faces and CC-licensed celebrity headshots, balanced by race/gender).
   • Experiments are organized as a test matrix (25 occupations × 12 input faces × 4 prompts = 1,200 unique evaluations), with group collaboration via online tools (e.g., Miro board), and all generation parameters are standardized for reproducibility.
3. Analysis and Interpretation:
   • Sub-teams independently annotate outputs using a mix of strategies:
     • Descriptive change logs (e.g., “adds beard”).
     • Perceptual labeling (coarse binary or ternary categories such as masculine/feminine, older/younger).
     • Concrete proxies (wrinkles, gray hair as age indicators; facial hair for masculinity).
   • Iterative hypothesis testing is supported by parameter adjustment and test reruns.
4. Reporting:
   • Auditors identify stakeholder audiences (engineers, peer creators, end-users).
   • Results are communicated in peer-accessible formats (e.g., TikTok-style videos, slideshows), including quantitative summaries (proportion changed in key dimensions), rationales for occupation/test selection, and representative example pairs.

This workflow operationalizes algorithmic auditing as an inclusive, empirical, and collaborative practice. The method yields datasets and annotations that both enable expert triangulation and produce usable, peer-to-peer audit artifacts.

2. Analytical Strategies and Quantitative Metrics

Auditing practices in this setting converge on interpretable, relative-frequency metrics that can be formalized into group fairness measures:

• Change-in-Category Counts: For a given occupation and set of input profiles, the number or percent of outcomes classified in a protected category (e.g., “out of 12, 10 are masculine”).
• Proxy-Feature Counts: Use of observable features as group proxies—presence of facial hair (masculinity), makeup (femininity), wrinkles/gray hair (age).
• Perceptual Labels: Binary/ternary groupings (male/female, young/old, lighter/darker).

Mapping to formal fairness notions, let $A$ denote a sensitive attribute and $\hat{Y}=1$ the outcome (e.g., “output appears feminine”), then the demographic parity gap is

$$\Delta_{\mathrm{DP}} = \left| P(\hat{Y}=1 \mid A=a) - P(\hat{Y}=1 \mid A=b) \right|.$$

For example, if the occupation “nail technician” has $75\%$ feminine representations and “carpenter” just $8\%$, then $\Delta_{\mathrm{DP}} = 0.67$.

Quantitative metrics for age-shifting and racial representation are similarly operationalized as the empirical fraction of outputs that exhibit shifts (e.g., wrinkles added, race changed). Formally,

$$P(\text{wrinkles added}) = \frac{\#\{\text{outputs with wrinkles added}\}}{1200}$$

These metrics yield empirical, reproducible summaries that can be cross-compared with expert audits.

3. Comparison to Expert Provider Audits

Empirical alignment between participatory and expert-led audits is a central validation axis:

• For stereotypically gendered roles (e.g., “rapper,” “nail technician”), group-wise frequency statistics between teenagers and experts tightly match (e.g., teens: 96% “rapper” outputs were masculine; experts: ~94–100%).
• Both groups identify systematic up-aging for authority figures (e.g., “president” images older/more masculine).
• Expert re-coding (Fleiss’s $\kappa \approx 0.65$) confirms that non-expert annotations are credible, and frequency counts are robust.

This triangulation demonstrates that, with structured scaffolding, participatory audits can replicate core empirical findings of professional audits and may extend the audit lens to new harm dimensions not usually surfaced in expert practice.

4. Methodological Insights: Non-Expert Appropriation and Novel Harm Dimensions

Participatory audits differ from professional ones not only in expertise but also in perspective and criteria formation:

• Audit Criteria Development: Auditors incorporate lived experiences and new hypothesized harms (e.g., teens foregrounded age bias and occupational representation).
• Balancing Social/Technical Considerations: Technical protocol (standardized spreadsheet, parameter control) ensures compact data collection and management. Simultaneously, open debate and reflection elicit nuanced social interpretations—including explicit interrogation of auditors’ own stereotypes.
• Surfacing New Axes of Harm: Participatory groups identified age as a salient and underexplored bias, analyzing wrinkles and gray hair frequency by occupation—dimensions not regularly included in professional audits.

These methodological novelties reinforce the argument that non-expert involvement injects both construct validity and alternative critical perspectives into the auditing canon (Morales-Navarro et al., 6 Aug 2025).

5. Implications for Auditing Frameworks, AI Literacy, and Ecosystem Design

Participatory audits suggest concrete directions for algorithmic auditing research and governance ecosystems:

• Audit Tooling: Embedding audit modes (batch runners, annotation widgets, automated metric calculators) into ubiquitous digital platforms can facilitate non-expert audits at scale.
• K–12 AI Literacy: Algorithmic auditing is tractable as a high-school project module, connecting technical skills with critical inquiry and peer-driven reporting. Formats such as TikTok audit explainers leverage youth fluency and engagement.
• Research and Governance Expansion: Regular participation of non-experts in audits validates that “everyday” or crowdsourced auditing is viable beyond expert enclaves. The engagement surfaces new questions (e.g., age bias) and broadens the concept of algorithmic harm.

A broader implication is that participatory auditing enables reflective loops: auditors both confirm existing harms and originate new audit dimensions, thereby driving iterative improvement of audit methodologies and AI systems themselves.

6. Synthesis: Participatory Auditing within the Algorithmic Audit Ecosystem

This participatory case study confirms that full-cycle, scaffolded audits executed by non-experts are methodologically sound and capable of surfacing, quantifying, and communicating representational harms in black-box AI systems (Morales-Navarro et al., 6 Aug 2025). With structured five-step processes, collaborative data collection, divergence-aware annotation, and peer-to-peer reporting, the participatory paradigm complements and enhances professional auditing. It both expands the auditable set of harms and empowers broader publics to engage in the empirical critique of sociotechnical systems. As algorithmic decision-making diffuses into daily life, scalable and inclusive auditing frameworks become necessary for both rigorous accountability and robust AI literacy.