LEAD Policy: Framework for AI Consensus

• LEAD Policy is an institutional framework that aggregates and synthesizes scientific evidence into expert consensus to guide AI policy.
• It employs structured workflows, calibrated scoring, and iterative deliberation to ensure transparency and methodological rigor.
• Implemented at NeurIPS, the framework leverages a diverse community of experts to produce actionable, consensus-based summaries for policymakers.

A LEAD Policy (Leading Evidence Aggregation and Deliberation Policy) is an institutional framework for systematically generating, synthesizing, and aggregating scientific evidence into actionable expert consensus, with the explicit purpose of informing and shaping public policy on AI and related technologies. The core concept, as advanced by Bommasani in “NeurIPS should lead scientific consensus on AI policy” (Bommasani, 30 Sep 2025), is that a robust and transparent mechanism—mirroring the process of global scientific bodies such as the IPCC—should be established within major scientific organizations such as NeurIPS to close the current evidence–consensus gap in AI policy. This policy provides both a conceptual architecture (roles, workflows, and governance processes) and operational blueprints (consensus scoring, deliberation timelines, outputs) for institutionalizing consensus formation on policies requiring input from the AI research community.

1. The Evidence–Consensus Gap in AI Policy

Current AI policy arenas are characterized by a paradox: a surfeit of published primary research, yet no trusted process to synthesize and summarize that research into expert consensus. Policymakers face twin deficits:

• Evidence Synthesis Deficit: While meta-analyses and broad surveys (e.g., the International Scientific Report on AI) are sporadically produced, and ad hoc synthesis appears in special issues or white papers, these are neither institutionalized nor consistently trusted. There is no established mechanism within major venues (e.g., NeurIPS) for active, ongoing synthesis targeted at actionable policy questions.
• Consensus Formation Void: Unlike public health or climate science—where structures like the IPCC and WHO convene thousands of experts in transparent, multi-stage dialogue to generate consensus statements (“the consensus of experts is X with likelihood Y”)—AI policy lacks any such scalable, transparent architecture. Policymakers are thus left without rigorous, consensus-based summaries of what the field regards as established fact or with what confidence, leading to environments in which cherry-picking and politicization are prevalent (Bommasani, 30 Sep 2025).

2. Justification for NeurIPS as a Consensus Host

The proposal’s central institutional design decision—embedding the LEAD Policy mechanism within NeurIPS—rests on two pillars:

• Internal Legitimacy: NeurIPS convenes not only the world’s largest community of ML and AI researchers (>16,000 attendees), but also maintains broad disciplinary diversity (across deep learning, statistics, cognitive science, and related domains). Any consensus-building process rooted in this community, and governed by nomination and peer participation, inherits both subject-matter authority and community buy-in.
• External Credibility: NeurIPS is regarded as the highest-impact machine-learning conference with an established reputation for transparency, openness, and methodological rigor, as evidenced by its h5-index and media visibility. Policy statements endorsed by such a venue have immediate weight among governments, industry, and the press. Unlike recent UN initiatives or corporate summits, NeurIPS is not directly beholden to political or commercial interests—central for independence in consensus formation (Bommasani, 30 Sep 2025).

Other institutions (e.g., UN panels, local AI summits, FAccT) lack the combination of scientific legitimacy, scale, and independence. The IPCC and public-health GRADE processes are cited as analogues, but require years of institutional trust not available to newly-formed bodies.

3. Structural Elements and Workflow of the LEAD Policy Process

The LEAD Policy is structured as a multi-layered, iterative consensus pipeline, adapted from climate-science governance:

Element	Function	Example Implementation
Working Group (“Bureau”)	Oversee and coordinate all stages of consensus process	15–25 elected experts
Consensus Track	Venue for method/consensus-building submissions	Special NeurIPS track
In-Session Deliberation	Structured community debates, live consensus polling	SPM-format debate sessions
Modular Subgroups	Parallel “Working Groups” on key policy primitives	E.g., thresholds, metrics

Workflow (annual cycle) (Bommasani, 30 Sep 2025):

• Nominations and composition: Call for evidence proposals and nominations to the working group six months pre-conference.
• Scoping and drafting: Subgroups define and iteratively refine specific “policy primitives” (e.g., definitions of evaluation rigor, compute thresholds, red-teaming cost metrics).
• Community engagement: Meta-surveys before/after conference to quantify broad agreement; live debates on unsettled issues at NeurIPS; public posting of Summaries for Policymakers (SPMs).
• Drafting and review: Statements are iteratively written, reviewed (including line-editing by government representatives with scientific veto), and externally validated (e.g., by IAC-like panels).
• Publication: Final consensus statements are published every 3–5 years or as-needed, with explicit scoring of the degree and certainty of consensus.

4. Formal Mechanisms for Consensus Formation and Scoring

The consensus process is formalized through explicit scoring protocols that calibrate both agreement and uncertainty in the community:

Given a statement $x$, each expert $e$ submits a calibrated judgment $r_e(x)$ (e.g., $-1$ for complete disagreement, $+1$ for strong agreement). Each expert is weighted by $\omega_e$ (uniform or expertise-based), and the aggregate consensus score is defined as: $$C(x)\;=\;\frac{\sum_{e\in E} \omega_e\,r_e(x)}{\sum_{e\in E} \omega_e}$$ A statement is considered “likely” if $C(x)\ge\tau$ for some threshold $\tau$ (e.g., $\tau=0.66$ for >66% consensus), and other IPCC-style calibrated language may be mapped to intervals in $C(x)$ (“very likely,” “medium confidence,” etc.). Weights can be adjusted for expertise or variance, as in inverse-variance weighting or domain-specified criteria (Bommasani, 30 Sep 2025).

5. Governance, Transparency, and Best Practices

Central to LEAD Policy implementation is a strict adherence to procedural transparency, independence, and reproducibility—each drawn from established scientific governance models:

• Conflict-of-interest (COI) management: Declarations are published for all participants; external audits by advisory panels safeguard against bias.
• Open peer review: Draft consensus statements and SPMs are subject to broad community comment and formal written feedback; all revisions are archived.
• Iterative deliberation: Multiple rounds of revision and open debate are scheduled, emphasizing deliberative agreement rather than majority-rule voting.
• Calibrated language and uncertainty quantification: Every key summary is explicitly tagged with standardized probability/confidence terms.
• Operational cadence: Carefully scheduled drafting, comment, debate, and ratification cycles yield predictable, reproducible outputs for policymakers.

The seven-pillar best-practices blueprint includes: (1) community mobilization via open nomination/election; (2) scientific rigor in evidence inclusion; (3) modular consensus structure (parallel working groups); (4) iterative consensus cycles with peer review; (5) calibrated, standardized language for probability/uncertainty; (6) strict transparency and independence (public COI, external audit); (7) publication of Policy Summaries with scientist veto on all statements (Bommasani, 30 Sep 2025).

6. Addressing Objections and Institutionalization Challenges

Two main objections—deep divisions within the research community and the argument that “NeurIPS is not a policy venue”—are addressed by direct analogy to prior scientific consensus bodies. The climate science domain began with considerable disagreement, but consensus scaled over repeated, structured engagement. Policy-focused activity at NeurIPS (e.g., ethics reviews, reproducibility checklists, regulation workshops) already demonstrates the field’s openness to codifying policy engagement as a core community function.

Consensus is not construed as unanimity, but as a calibrated, deliberative expression of what the evidence most robustly supports, and an honest quantification of areas of residual uncertainty or disagreement; e.g., reporting “medium confidence” or “33–66% consensus” on controversial points, rather than a monolithic assertion (Bommasani, 30 Sep 2025).

7. Operationalization, Outputs, and Long-Term Vision

A full LEAD Policy infrastructure produces:

• Periodic, peer-reviewed “Summaries for Policymakers” (SPMs), with calibrated language, issued on a predictable schedule and widely citable by governments, regulatory bodies, and media.
• Public dashboards and data archives of consensus scores over time.
• Transparent documentation of all deliberations, comments, and conflict-of-interest disclosures.
• Extensible governance templates that can be adopted by other AI research venues (e.g., ICML, FAccT) or international bodies as needed.

Originating at NeurIPS, a mature LEAD Policy framework is positioned to evolve into a global, cross-institutional “AI IPCC,” codifying evidence-based, legitimate AI policy for governments, industry, and civil society (Bommasani, 30 Sep 2025).