Participation in the age of foundation models (2405.19479v1)

Abstract: Growing interest and investment in the capabilities of foundation models has positioned such systems to impact a wide array of public services. Alongside these opportunities is the risk that these systems reify existing power imbalances and cause disproportionate harm to marginalized communities. Participatory approaches hold promise to instead lend agency and decision-making power to marginalized stakeholders. But existing approaches in participatory AI/ML are typically deeply grounded in context - how do we apply these approaches to foundation models, which are, by design, disconnected from context? Our paper interrogates this question. First, we examine existing attempts at incorporating participation into foundation models. We highlight the tension between participation and scale, demonstrating that it is intractable for impacted communities to meaningfully shape a foundation model that is intended to be universally applicable. In response, we develop a blueprint for participatory foundation models that identifies more local, application-oriented opportunities for meaningful participation. In addition to the "foundation" layer, our framework proposes the "subfloor'' layer, in which stakeholders develop shared technical infrastructure, norms and governance for a grounded domain, and the "surface'' layer, in which affected communities shape the use of a foundation model for a specific downstream task. The intermediate "subfloor'' layer scopes the range of potential harms to consider, and affords communities more concrete avenues for deliberation and intervention. At the same time, it avoids duplicative effort by scaling input across relevant use cases. Through three case studies in clinical care, financial services, and journalism, we illustrate how this multi-layer model can create more meaningful opportunities for participation than solely intervening at the foundation layer.

The Participatory Ceiling in Foundation Models

The paper entitled "Participation in the Age of Foundation Models" by Harini Suresh et al., presented at the ACM Conference on Fairness, Accountability, and Transparency (FAccT '24), explores the intersection of participatory ML approaches and the deployment of foundation models. This analysis explores how to empower historically marginalized communities which are often disproportionately affected by power imbalances and potential harms induced by these models.

Participatory Machine Learning and Foundation Models

Participatory ML has gained attention as a pathway to reduce biases and redistribute power by incorporating diverse stakeholder perspectives. Traditionally, this approach has been employed in specific applications where the context, stakeholders, and tasks are well-defined. However, the universal applicability and vast scope of foundation models bring new challenges to participatory approaches.

Foundation models represent a shift in ML paradigms, aimed at providing generalizable solutions across a myriad of tasks. Such models, trained on extensive datasets often scraped from the internet, are primarily developed and controlled by large tech companies. This centralization of control presents significant barriers to meaningful participation from affected communities.

Analysis Using the Parameters of Participation

The paper evaluates existing participatory efforts in foundation model development using the "Parameters of Participation" framework by Delgado et al. This framework examines the goals, stakes, scope, and forms of participation along a spectrum ranging from consultation to ownership. Several mechanisms were reviewed, including reinforcement learning with human feedback (RLHF), the development of rulesets and policies, and red teaming.

• RLHF: Primarily involves gathering feedback from crowd workers to refine models through preference modeling. This method usually only achieves the consultation mode of participation as it focuses on improving model performance without engaging deeper with impacted communities.
• Rulesets and Policies: This includes efforts like Anthropic's Collective Constitutional AI, where public polling is used to determine values that guide the model’s behavior. These efforts often remain at the inclusion mode due to limited stakes and forms of engagement, as participants do not effectively shape foundational decisions.
• Red Teaming: Involves domain experts identifying vulnerabilities in models. While useful, these initiatives are mostly confined to consultative roles without extending into collaboration or ownership modes due to their limited impact on decision-making.

The Participatory Ceiling and Centralization of Control

The paper identifies a "participatory ceiling," a conceptual threshold limiting the efficacy of participatory efforts in universally applicable foundation models. This ceiling exists primarily due to two factors:

1. Corporate Control and Incentives: Foundation model developers, often large tech companies, lack sufficient incentives to share control or engage deeply with affected communities. These companies are driven by profit motives and risk aversion, limiting transparency and meaningful stakeholder engagement.
2. Context-Agnostic Models: Foundation models are designed to be generalizable, making it challenging to address specific, context-sensitive issues through participatory means. Participation necessitates context-specificity to be meaningful, which is at odds with the universal applicability of foundation models.

Blueprint for Participatory Foundation Models

To overcome these challenges, the paper proposes a three-layer framework to facilitate meaningful participation:

1. Foundation Layer: The basic, universally applicable model developed by tech companies or other institutions. This layer should ensure foundational utility and be amenable to inputs from subsequent layers.
2. Subfloor Layer: This intermediate layer focuses on domain-specific technical infrastructure, norms, and governance. For instance, in the healthcare domain, this could involve patient advocacy groups pooling data to audit speech-to-text models for inclusivity and fairness.
3. Surface Layer: This layer pertains to specific downstream applications, where localized participatory methods can be used to co-create solutions tailored to a particular context, such as a clinic deploying a transcription tool for patient interactions.

Case Studies

The authors provide case studies to illustrate the application of this blueprint:

• Clinical Care: By engaging patient advocacy groups to collect dialect-specific data, the speech-to-text model can be fine-tuned for inclusivity, and localized use cases can benefit from domain-specific audits and guidelines.
• Journalism: Addressing copyright concerns, entities like newsrooms can form collectives at the subfloor layer to assert collective rights and negotiate fair use agreements with foundation model developers.
• Financial Services: Establishing robust reporting and recourse mechanisms at the subfloor layer can help better capture discriminatory patterns and inform more equitable fraud detection systems.

Conclusion

The proposed framework aims to break the participatory ceiling through domain- and application-specific engagements that collectively shape foundation models while recognizing local expertise and needs. This stratification enables more robust and meaningful participation, balancing the scale and generalizability of foundation models with the necessity for contextual sensitivity and community empowerment.

The paper underscores the importance of continuing to explore mechanisms that allow for equitable and just governance of AI technologies. It calls for further research into scoping subfloors, fostering accountability, and ensuring that participatory efforts genuinely redistribute power to those most affected by technological advancements.