Vibe Econometrics and the Analysis Contract

Abstract: "Vibe coding" and "vibe analytics" have been framed as a democratization of technical capability. This paper argues that AI-assisted methodology more broadly, or what I call "vibe methodology," also democratizes the failure modes specific to each domain. When AI assists with methods whose validity depends on assumptions that cannot be verified from the output alone (a class I call "vibe inference"), the failure surface is structurally different: the output does not reliably signal invalidity, and when it does, recognizing the signal requires the expertise the workflow bypasses. I focus on "vibe econometrics," the subset of AI-assisted causal analysis where identification can be named faster than it can be audited. The claim of this paper is not that AI invents inferential failures that did not previously exist, but that it changes their incidence, observability, and persuasive force enough to create a practically distinct governance problem. This results in three failure modes: method-data mismatch, where AI bypasses expertise at execution; confidence laundering, where AI amplifies the credibility of formatted output; and invisible forking, which spans both. What is new is not the failure modes but AI's industrialization of their packaging. The barrier between naming a method and executing it has collapsed, and weak foundations, dressed as rigorous analysis, now reach audiences at a scale, speed, and polish that previously required expertise. I propose the Analysis Contract, a pre-commitment framework that adapts the logic of pre-analysis plans and the Causal Roadmap to the AI-assisted setting. The contract imposes three conditions before a causal claim is made: a method-data contract, a data audit, and a pre-commitment statement defining what would count as a disconfirming result. The framework generalizes across domains of vibe inference through domain-specific instantiation.

• The paper introduces vibe methodology to rigorously govern AI-assisted causal inference, presenting a taxonomy of failure modes that challenge proper method validity.
• It empirically illustrates risks such as method-data mismatch, confidence laundering, and invisible forking, which undermine traditional inferential checks.
• The Analysis Contract framework offers pre-commitment, diagnostic friction, and structured oversight to mitigate systemic flaws in AI-driven econometrics.

Vibe Econometrics and the Governance of AI-Assisted Causal Inference

Introduction

"Vibe Econometrics and the Analysis Contract" (2605.08071) interrogates the epistemic and governance challenges introduced by AI-assisted analytical workflows in applied econometrics, focusing specifically on the structural transformation of causal inference practice under LLM mediation. The work generalizes and formalizes the "vibe coding" and "vibe analytics" phenomena—AI-driven, prompt-based execution of domain-specific methodologies—into what is defined as "vibe methodology," and characterizes the risk profile of AI-accelerated inferential practice when method validity becomes decoupled from the workflow. The analysis is anchored by a taxonomy of three failure modes and the introduction of the "Analysis Contract," a rigorously articulated governance mechanism designed to reintroduce pre-commitment and diagnostic friction into the AI-augmented domain-agnostic causal inference pipeline.

Theoretical Framework: Vibe Inference and the New Failure Surface

The central insight of the paper is that while AI democratizes technical access, it concurrently industrializes and scales latent methodological failure modes. Traditional analytic workflows enforced friction—both in method execution and results reception—through programming barriers and domain expertise. LLMs collapse these frictions, making the presentation of formatted, credible, yet potentially invalid inferential claims trivially easy and fast for non-experts.

The concept of "vibe inference" denotes methods whose inferential validity depends on identification assumptions untestable from output alone. This is especially pernicious in causal econometrics, where the surface form of the output (statistically rigorous tables, confidence intervals, narrative summaries) fails to encode the critical assumption checks that prior workflows embedded, and where the underlying audience often lacks the capacity to interrogate the analytic foundation.

Taxonomy of Failure Modes in Vibe Econometrics

Three interrelated but analytically distinct failure modes are identified:

• Method-Data Mismatch: LLMs can execute sophisticated causal estimators (e.g., DiD, PSM, RDD) on arbitrary data structures without validating the critical identification assumptions. The output is mathematically correct but causally uninterpretable. AI lowers the technical floor for incurring these errors and makes them both more frequent and less visible by bypassing the assumption checks that were previously enforced by programming friction or expertise.
• Confidence Laundering: AI-generated outputs inherit the rhetorical and typographic cues of expert analysis. This confers spurious legitimacy on fundamentally invalid analysis via "format amplification" and "sycophancy," capitalizing on empirically-documented human tendencies to over-rely on confidently presented algorithmic output [Steyvers et al., 2024; Rathi et al., 2025]. Critically, these confidence effects are not mitigated by audit trails; the problem is not malicious intent or outright fabrication, but the systematic overrepresentation of weakly supported claims in a form engineered to elicit trust.
• Invisible Forking: The LLM interaction paradigm (natural language, rapid iteration, absence of persistent memory or logging) encourages, and by default enables, post hoc specification search ("forking paths"), with the procedural history of analytic choices lost. This is an AI-accelerated version of Gelman and Loken’s "garden of forking paths" [Gelman & Loken, 2013], but now operating at orders-of-magnitude greater speed and scale, without the implicit documentation left by code revision or script history.

These failure modes are shown to be exacerbated, not solved, by post hoc disclosure or standard reproducibility tools, with auditability suffering under both organizational incentives and current review practices.

The Analysis Contract: A Pre-Commitment Governance Framework

To counteract the epistemic and institutional risks outlined, the Analysis Contract is introduced. It operationalizes three necessary and jointly sufficient process conditions, integrating lessons from pre-analysis plans (PAPs), statistical analysis plans (SAPs), and the Causal Roadmap for observational studies [Petersen & van der Laan, 2014; Munafò et al., 2017; Dang et al., 2023].

The three conditions:

1. Method-Data Contract: A domain- and method-specific checklist of identification assumptions is written and reviewed prior to AI-based execution. For DiD, this includes explicit documentation of units, aggregation, treatment/control definitions, trend check feasibility, etc. Violations are classified as stop, flag, or branch points for analysis adaptation.
2. Data Audit: Data are explicitly audited with respect to the above contract, verifying provenance, granularity, coding consistency, baseline equivalence, and the feasibility of plotting the critical identifying diagnostic. The core intervention is making assumption-testing explicit and visual, not implicit or buried in automated output.
3. Pre-Commitment Statement: The analyst, before observing outcomes, defines the primary specification, enumerates exclusion/falsification criteria, and stipulates transparent reporting conventions. This includes independent disclosure of any conflicts of interest and a commitment to report all attempted specifications, paralleling contemporary open science best practices.

The Analysis Contract mechanistically targets each failure mode: Conditions 1 and 2 directly block method-data mismatch, Condition 3 constrains forking and preempts confidence laundering by making ex ante commitments salient and auditable.

Comparative Context and Epistemic Implications

The Analysis Contract is conceptually aligned with the established literature on SAPs/PAPs but extends, adapts, and operationalizes these instruments for environments lacking institutional registration infrastructure, high technical expertise, or adversarial review. The theoretical shift is from code-level friction to structured meta-task pre-commitment, providing menu-driven friction--not as a bureaucracy, but as a disciplined record for both internal review and external audit.

A crucial empirical observation is that AI-assisted analysis lowers the cost of analysis but not the cost of analytic error. Structural incentives in organizations (favoring confirmatory evidence, reward for "successful" analysis) are not meaningfully offset by disclosure or voluntary review; the framework anticipates and foregrounds the persistence of motivated reasoning and the fragility of self-administered commitments [Kunda, 1990].

At sufficient scale, systematic deployment of LLM-mediated causal inference can create aggregate knowledge degradation. Such workflows threaten to erode evidentiary and scrutiny norms in both organizational and scientific contexts, as suggested by recent work on recursive model training collapse [Shumailov et al., 2024] and incentive-knowledge feedback breakdown [Acemoglu, Kong & Ozdaglar, 2026].

Practical Implementation and Future Directions

The paper provides method- and domain-specific templates (e.g., DiD in healthcare claims) as Appendices, emphasizing the need for context-sensitive operationalization rather than universal checklists. Review intensity is argued to be proportional to consequence—ranging from lightweight peer audits to formal adversarial review or independent methods oversight for high-stakes decisions.

The Analysis Contract is not a panacea but an essential procedural intervention. The proposed framework facilitates organizational self-assessment (e.g., training, analytics standards, or compliance guidelines) and builds a pre-emptive audit trail to avoid ex post rationalization.

Empirical validation is deferred to a planned companion paper, where LLMs will be challenged with canonical datasets that have explicit ground-truth identification status.

Conclusion

"Vibe Econometrics and the Analysis Contract" synthesizes a compelling and technically rigorous account of the epistemic risks engendered by LLM-based analytic workflows in causal inference. By introducing and formalizing the Analysis Contract, it supplies a minimal procedural governance architecture aimed at preserving inferential dignity in the face of automated scholarly and organizational analysis. While the framework does not and cannot guarantee the validity of all causal claims—particularly in incentive-rich or expertise-thin environments—it provides an explicit, reproducible scaffold for critical assumption-checking and accountability before causal inferences are made. This is a decisive contribution for the practical governance of AI in applied econometrics and serves as a blueprint for future field-level empirical assessments and the development of domain-adapted analytic standards.