Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt (2505.05197v1)

Abstract: AI systems are increasingly placed in positions where their decisions have real consequences, e.g., moderating online spaces, conducting research, and advising on policy. Ensuring they operate in a safe and ethically acceptable fashion is thus critical. However, most solutions have been a form of one-size-fits-all "alignment". We are worried that such systems, which overlook enduring moral diversity, will spark resistance, erode trust, and destabilize our institutions. This paper traces the underlying problem to an often-unstated Axiom of Rational Convergence: the idea that under ideal conditions, rational agents will converge in the limit of conversation on a single ethics. Treating that premise as both optional and doubtful, we propose what we call the appropriateness framework: an alternative approach grounded in conflict theory, cultural evolution, multi-agent systems, and institutional economics. The appropriateness framework treats persistent disagreement as the normal case and designs for it by applying four principles: (1) contextual grounding, (2) community customization, (3) continual adaptation, and (4) polycentric governance. We argue here that adopting these design principles is a good way to shift the main alignment metaphor from moral unification to a more productive metaphor of conflict management, and that taking this step is both desirable and urgent.

• The paper challenges the assumption of universal rational convergence by advocating a pluralistic, quilt-like approach to managing diverse human values.
• It introduces design principles for AI, emphasizing contextual grounding, community customization, continual adaptation, and polycentric governance.
• The study illustrates how embracing divergent moral frameworks and social technologies can lead to robust, adaptable AI systems.

The paper "Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt" (2505.05197) challenges a common implicit assumption in AI safety and ethics, which the authors term the "Axiom of Rational Convergence." This axiom posits that under ideal conditions (sufficient time, information, reasoning ability), rational agents would converge on a single, correct set of beliefs, values, or ethics. The paper argues that this assumption is questionable and unhelpful for building AI systems intended to operate within diverse human societies, which exhibit persistent and fundamental moral and value disagreements.

Instead of pursuing a "clearer vision of something true and deep" – a single, universal ethics for AI alignment – the authors propose adopting the metaphor of "sewing together a very large, elaborate, polychrome quilt." This view sees human societies as a patchwork of diverse communities with enduringly divergent values, norms, and worldviews. Stability and cooperation are achieved not through value convergence, but through practical "social technologies" like conventions, norms, institutions, and negotiation practices that manage conflict and enable coexistence despite fundamental differences.

The paper contrasts this perspective, which aligns with a philosophical "Conflict Theory" view of society, with the dominant "Alignment as Mistake Theory." Mistake Theory views disagreements as errors arising from correctable flaws (biases, lack of information) that would vanish under ideal rationality. The authors argue that applying Mistake Theory to AI alignment, by seeking a single, universally "correct" objective function or value set (like Coherent Extrapolated Volition [yudkowsky2004coherent]), overlooks the deep-seated and often irreconcilable nature of human disagreements. This pursuit of a universal alignment risks being socially unstable, potentially coercive, and neglects the practical work required for AI to navigate the existing landscape of pluralism.

The authors identify several potential blind spots arising from an over-reliance on the "Mistake Theory" approach:

1. Universalizing Reasoning: Assuming a single, optimal reasoning process for AI across all domains ignores how standards of justification and evidence vary across different human contexts and communities [kuhn1962structure].
2. Fragility of Long Abstract Arguments: Safety arguments relying on lengthy, abstract logical chains can be sensitive to initial assumptions, difficult to verify, and prone to motivated reasoning, becoming less reliable as they move away from empirical grounding.
3. Navigating Heterogeneous Preferences: Even for shared goals like mitigating existential risk (X-risk), achieving collective action requires overcoming significant challenges like the "start-up problem" (agreeing on specific, potentially costly strategies when values and priorities conflict) and the "free-rider problem" (ensuring actors contribute to global safety efforts when they can benefit from others' contributions) [marwell1993critical, heckathorn1996dynamics]. These are problems of conflict management and coordination among diverse actors, not just finding a technical solution based on a single value.
4. Methodological Solipsism: Mistake Theory can lead to a focus on individual-level epistemology (how I can reason better) rather than group-level dynamics (how we can coexist and coordinate despite differences), potentially leading to naive policy recommendations regarding power, pluralism, and conflict management.

The paper argues that stable human coexistence relies on "social technologies" [ostrom1990governing, hampshire1999justice, henrich2020weirdest] – conventions, norms, and institutions – which act as the "stitches" that bind the social quilt. These mechanisms manage conflict non-violently (e.g., courts, negotiation) and enable coordination (e.g., traffic laws, property rights) despite radical differences in underlying beliefs or values. Effectiveness stems not from universal agreement on values, but from perceived legitimacy, coercion, self-interest, or inertia. The key is understanding appropriateness, which is context-dependent and often rooted in "thick" (culturally specific) rather than "thin" (abstract, universal) moralities [walzer1994thick].

For AI systems to function effectively and safely in this "patchwork quilt" of human society, they must learn to navigate context-dependent appropriateness. Failures are seen as context-inappropriate behavior, not misalignment with a universal ideal. This necessitates a move away from monolithic AI design towards a more pluralistic and decentralized ecosystem.

The paper proposes four interconnected design principles for AI grounded in the appropriateness framework:

1. Contextual Grounding: AI systems need access to rich, real-world contextual information beyond just the immediate prompt (e.g., location, social roles, recent events, cultural practices). This allows the AI to understand the specific "patch" of the quilt it is operating within.

   function generate_appropriate_response(prompt, user_profile, environment_data): # environment_data includes location, time, recent news, group norms, etc. contextualized_input = combine(prompt, user_profile, environment_data) raw_response = language_model.generate(contextualized_input) return apply_context_specific_filters(raw_response, user_profile, environment_data)

   Implementing this requires technical solutions for accessing and responsibly using sensitive contextual data, emphasizing privacy-preserving techniques [nissenbaum2004privacy].
2. Community Customization: Allowing different communities or applications to customize the norms governing AI behavior to reflect their specific values and practices. This moves beyond a one-size-fits-all approach.

   function apply_context_specific_filters(response, user_profile, environment_data): community_norms = load_norms_for_community(user_profile.get('community_id')) app_rules = load_rules_for_application(environment_data.get('app_id')) # Filters/rules modify response based on community norms and app context filtered_response = enforce_norms(response, community_norms) final_response = apply_app_rules(filtered_response, app_rules) return final_response

3. Continual Adaptation: AI systems must learn and adapt to evolving social norms through ongoing feedback, similar to how humans learn via social sanctioning. This requires moving towards continuous learning rather than static training.

   function process_user_feedback(feedback_signal, interaction_context, generated_response): # Feedback_signal could be explicit rating, implicit reaction, or human moderation assessed_appropriateness = analyze_feedback(feedback_signal, interaction_context, generated_response) if assessed_appropriateness == INAPPROPRIATE: # Trigger learning update for the model or norm database update_norm_model(interaction_context, generated_response, feedback_signal) flag_for_review(interaction_context, generated_response) elif assessed_appropriateness == APPROPRIATE: # Reinforce positive behavior reinforce_norm_model(interaction_context, generated_response)

   Robust feedback mechanisms are crucial, and their design must consider potential manipulation by powerful agents.
4. Polycentric Governance: Decision-making and oversight for AI appropriateness should be distributed across multiple levels of authority (users, developers, platforms, regulators), mirroring human social governance structures [ostrom2010polycentric].

   # Layered rule system enforcement # Rules from different governance levels are applied sequentially or hierarchically user_settings -> community_rules -> app_developer_policies -> platform_policies -> legal_regulations

   This involves designing socio-technical systems where different stakeholders contribute to defining and enforcing appropriateness standards within their sphere.

Ultimately, the paper argues that the goal for AI is not to discover and implement a universal, objective morality, but to enable AI to participate constructively in the ongoing human project of "sewing the quilt" – managing disagreement, navigating diverse contexts, and contributing to collective flourishing within a pluralistic world. This shifts the focus from the "Astronomer" seeking a single, clear truth to the "Tailor" pragmatically stitching together diverse materials to create a functional and adaptable whole.