Racing to Ruin
Abstract: We study R&D competition in the shadow of disaster: advancing the technology frontier raises the risk of permanently ending all firms' payoffs. Under perfect monitoring and common knowledge of rationality, the equilibrium frontier is bounded below by the optimal stopping time of a monopolist, and above by that of a representative firm that persistently but mistakenly believes its rival is about to stop. We then analyze how the frontier is shaped by transparency (speed of monitoring) and trust (belief in the rationality of rival firms).
Summary
- The paper demonstrates that aggressive R&D competition can escalate technological advancement to levels that significantly heighten catastrophic risk.
- It introduces equilibrium boundaries showing firms' strategic stopping points under both perfect and imperfect transparency conditions.
- The analysis underscores the importance of transparency and trust in mitigating risks and informing regulatory policies for emerging technologies.
An Analysis of "Racing to Ruin"
This essay provides a detailed examination of the paper "Racing to Ruin" by Drew Fudenberg and Andrew Koh, which investigates the dynamics of R&D competition among firms under the specter of existential risk. The study models how firms' efforts to push the technological frontier can inadvertently increase the probability of catastrophic outcomes, subsequently impacting equilibrium strategies in competitive scenarios.
Introduction and Model Overview
The paper examines a situation where each firm's advancement on the technological frontier increases existential risk, essentially raising the chance that further technological developments could lead to a market collapse. The authors propose a model involving duopolists who face a trade-off between advancing technology and increasing disaster risk. This model mirrors dynamics that could apply to crucial global players, such as nation states, particularly in technology races between countries like the US and China.
Under perfect monitoring, where firms observe each other’s technology levels and stopping times, the paper discusses conditions leading to different equilibria in the technology race. The authors delineate bounds for when firms stop this race based on whether they choose to cooperate or compete.
Key Results and Theoretical Implications
Equilibrium Boundaries
The research identifies two key thresholds regarding technology advancement:
- Lower Bound (Monopolist Stopping Point): This represents the technological level at which a monopolist, without competitors, would choose to cease further development due to rising disaster risks.
- Upper Bound (Representative Firm's Maximum): This is the point beyond which even a firm that optimistically assumes its competitor to keep racing finds it rational to cease development due to elevated existential risk.
In equilibrium, under perfect transparency and rationality assumptions, firm behavior converges between these bounds. The actual stopping time is influenced by transparency and trust between competing firms which affects coordination capabilities.
Competition under Imperfect Conditions
The paper's model also explores scenarios with imperfect information, where communication delays obstruct immediate reactions to a competitor's stoppage. Here, firms may overextend the technological frontier, racing further than they would under perfect conditions. This is particularly pronounced when firms cannot trust that their competitors will act rationally or when detection of a rival's stopping is lagged.
The equilibrium analysis in cases of imperfect transparency shows a marked increase in potential for continued technological escalation, leading to a higher risk of disaster.
Practical Implications and Future Outlook
The conclusions imply profound considerations for regulatory frameworks on technology advancements, especially for transformative and possibly dangerous innovations like artificial intelligence. Increased transparency in firms' actions and intentions can potentially temper the race dynamics that lead to excessive risk-taking.
Figure 1: Score-contest thresholds; parameters: , , ; illustrating technological advancement bounds under different parametric conditions.
Moreover, trust among competing entities significantly impacts equilibrium outcomes. Coordination failures, driven by distrust, almost inevitably lead to further escalation and increased risks of catastrophic failures. Promoting frameworks that enhance transparency and trust can mitigate these adverse effects.
Figure 2: Trust and transparency influencing technology race dynamics and failure probabilities.
Conclusion
"Racing to Ruin" offers an insightful theoretical framework delineating how technological races might unfold in the shadow of disaster risks. It punctuates the need for careful consideration of transparency and trust in competitive environments, especially when technological risks carry significant existential implications. The results serve as a clarion call for policymakers and industry leaders to foster environments where cooperative strategies can be developed to manage and potentially mitigate the tensions inherent in competitive technological advancements.
Whiteboard
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview
This paper studies a race between two companies to build powerful new technology (think advanced AI). Pushing the technology forward makes each company more successful in the short run, but it also raises the chance of a permanent disaster that ends everyone’s profits forever. The authors ask: Will companies slow down to avoid disaster, or will they keep racing? And how do watching each other (transparency) and trusting each other (beliefs about how sensible the rival is) change what happens?
A simple way to picture it: two climbers race up a mountain to win a prize. The higher they go, the bigger their prize—but the risk of an avalanche grows with altitude. If either climber stops, the avalanche risk stops growing. Should they keep going or stop together before it’s too dangerous?
The Main Questions
To make the ideas concrete, the paper focuses on these questions:
- If both companies can perfectly see what the other is doing, how far does the race go before they stop?
- What changes when there’s a delay in noticing the other company has stopped (less transparency)?
- What changes if you’re not fully sure the other company is rational and might never stop (less trust)?
- How do these factors push the “technology frontier” closer to safety or closer to disaster?
How They Study It
The authors build a simple “game” that plays out over time:
- Two firms can keep developing their tech or stop permanently. Developing increases their own profits and hurts their rival’s profits (classic competition).
- The higher the most advanced firm climbs, the higher the chance per moment of a disaster that kills everyone’s future profits (this chance is called the “hazard rate”).
- If a firm stops, it locks in its current level of tech and no longer adds to the disaster risk. (Think: the avalanche risk only increases if someone is still climbing higher.)
- The firms may or may not immediately see when the other has stopped. “Perfect transparency” means you see it right away. “Imperfect transparency” means you find out after a random delay.
- “Trust” means you believe the other firm is rational and wants to avoid needless risk. “Low trust” means you think there’s a chance they’re “crazy” and will never stop.
To understand where the race will likely end, they compare three imagined decision-makers:
- A single company with no rival (a monopolist). It balances more profit against more risk and picks a safe stopping point.
- Two competing companies that can perfectly see each other and know the other is rational.
- A “representative firm” that wrongly believes the rival will freeze at the current level (this makes it overly optimistic about the benefits of racing ahead).
These comparisons create simple “markers” (lower and upper bounds) for how far the real race will go.
What They Find
Here are the main results, presented in plain terms:
- Perfect watching helps coordination and sets a ceiling:
- If both firms can instantly see each other’s decisions and both are rational, the race never goes beyond a certain ceiling. That ceiling is the stopping point of the “overly optimistic” firm that thinks its rival will freeze now.
- The race also won’t end before a lower floor: the stopping point a single firm (a monopolist) would choose. Why? Because the last mover who’s still climbing has at least as much reason to stop late as a monopolist does.
- So, with perfect transparency and rational rivals, the stopping point always lies between:
- Lower bound: the monopolist’s safe stop.
- Upper bound: the over-optimistic firm’s stop.
- Sometimes these two points are the same, so the outcome is tightly pinned down.
- Less transparency (slow or delayed detection) pushes the frontier outward and can cause endless racing:
- If it takes a long time to learn the other firm has stopped, then stopping first is risky: you sit still while your rival keeps gaining and the disaster risk keeps climbing until they notice you stopped.
- When detection is very slow, “racing forever” can become self-sustaining. Nobody wants to be the first to stop, so no one stops at all—disaster becomes almost certain.
- When detection is pretty fast (but not perfect), firms still stop in finite time, but they are tempted to “stop second.” This can make the race overshoot the ideal ceiling a bit. The paper shows this extra overshoot shrinks as detection gets faster, roughly like 1 divided by the news speed.
- Less trust (you suspect your rival might never stop) creates three zones:
- Low trust: every likely outcome is a race to ruin. Disaster almost surely happens.
- Medium trust: both “stop now” and “race to ruin” can happen, depending on expectations.
- High trust: the chance that two rational firms race forever becomes very small.
- Transparency is double-edged: speeding up detection first makes it attractive to wait for proof the other stopped (which can kill the “stop early” outcome), but if detection gets fast enough, stopping becomes self-enforcing again.
Why this matters:
- Even when both firms would prefer to slow down together, strategic worries—like “I don’t want to stop first,” “I don’t know if they stopped yet,” or “maybe they’ll never stop”—can push them into dangerous territory.
- Good monitoring and higher trust help firms coordinate to stop before it’s too risky.
Why This Is Important
- For powerful, potentially dangerous technologies (like advanced AI), unregulated competition can drive everyone to take too much risk.
- Better transparency—fast, reliable ways to see when others slow down or stop—can prevent “races to ruin.”
- Building trust—through credible commitments, audits, shared safety thresholds, and reputation—reduces the fear that you’ll be the sucker who stops first while your rival sprints ahead.
- Policymakers can design rules that make safe stopping easier and free-riding harder, for example:
- Fast reporting and verification of safety pauses.
- Penalties for secretly continuing.
- Shared, publicly known “red lines” that automatically trigger a pause.
Takeaway
In a high-stakes tech race, competition alone can carry everyone too far, risking disaster. If firms can see each other clearly and trust each other to act sensibly, they can coordinate to stop at a safer point. But when detection is slow or trust is low, the race can spiral into “racing to ruin.” Smart transparency tools and trust-building rules can shift outcomes from danger to safety.
Practical Applications
Immediate Applications
Below are concrete ways practitioners can act on the paper’s results now, with sector links and key dependencies noted.
- Sector-wide “mutual-pause” channels to make stopping instantly observable
- What: Set up authenticated, real-time “stop” broadcast channels (APIs, Slack bridges, signed emails, webhooks) among frontier organizations so a unilateral pause is detected with minimal lag (high ω).
- Sectors: AI, biotech, defense/dual-use robotics.
- Tools/products: Mutual-Pause Protocol (cryptographic stop messages with immediate receipts), shared kill-switch notifications tied to release gates.
- Why it maps: With faster detection, the expected overshoot beyond the safe ceiling τR shrinks to O(1/ω); perfect/immediate observability implements early stopping.
- Assumptions/dependencies: Willingness to join; identity/attestation of senders; uptime SLAs; legal safe harbors for pause signaling.
- Regulatory transparency requirements that reduce detection lags
- What: Mandate near-real-time reporting of capability milestones, incident reports, and “pause” notices to a regulator or licensed monitor; require telemetry or compute-attestation that can trigger alerts.
- Sectors: AI governance (compute oversight), biosecurity (gene synthesis screening logs), critical software.
- Tools/products: Transparency-as-a-Service providers; regulator dashboards; secure telemetry collectors; compute notarization on hardware TEEs.
- Why it maps: Imperfect monitoring can make “race forever” self-enforcing; reducing lag restores cooperative stopping equilibria.
- Assumptions/dependencies: Legal authority; privacy/IP protections; API and schema standards; hardware/Cloud vendor cooperation.
- Governance certifications to raise “trust” (belief in rationality) among competitors
- What: Independent audits/certifications for safety governance (board oversight, incident response, staged release), plus publication of audit summaries.
- Sectors: AI, biotech, advanced robotics; procurement (govt/enterprise buyers).
- Tools/products: Safety culture scorecards; audit badges; assurance frameworks aligned with NIST/ISO.
- Why it maps: With higher trust, equilibria shift from racing-to-ruin to early stopping; backfires of transparency at intermediate trust are less likely.
- Assumptions/dependencies: Credible auditors; limited greenwashing; repeated interactions.
- Contracts that reward “stop-first” and penalize “continue-after-peer-stops”
- What: Pre-specified bilateral/multilateral contracts with escrowed deposits: bonuses for verifiable early stops after agreed triggers; liquidated damages if a party continues after another’s stop signal.
- Sectors: AI labs, defense primes, pharma co-development, large research consortia.
- Tools/products: Smart-contract escrows; third-party arbiters with verified logs; trigger definitions tied to KPIs.
- Why it maps: Makes “stop-first” a best response even with modest detection lags; internalizes the hazard externality.
- Assumptions/dependencies: Verifiability of “stop” and “continue”; enforceable jurisdiction; antitrust compliance.
- Internal “frontier risk dashboards” with operational stop rules
- What: Compute and track the two indices D_i log π(t,t) (neck-and-neck race incentive) and D_i log π(t,0) (monopoly incentive) alongside λ(t) (hazard) to identify the bracket [underline τ, τR] and define pre-committed stop thresholds.
- Sectors: Any high-stakes scaling effort (AI, bio, autonomous systems).
- Tools/products: Risk-calculator modules; policy engines to freeze training/deployment when indices cross; red-team feedback feeding λ(t).
- Why it maps: The paper pins when stopping is privately and jointly optimal; operationalizes thresholds.
- Assumptions/dependencies: Measurable proxies for π and λ; executive buy-in; integration with CI/CD or training pipelines.
- Policy levers that shrink τR by reducing “winner-take-most” rents
- What: Encourage interoperability and standardization; require staged, partial releases; limit exclusive data advantages; promote prize-splitting mechanisms that blunt marginal gains from being slightly ahead.
- Sectors: AI platforms, app ecosystems, cloud marketplaces.
- Why it maps: τR rises with the diagonal index D_i log π(t,t), which increases with rivalry intensity; dampening comparative-advantage rents moves the ceiling inward.
- Assumptions/dependencies: Careful antitrust and IP design; avoid stifling beneficial competition.
- Liability and insurance pricing that raise effective λ(t)
- What: Differential liability, capital requirements, and insurance premiums that scale with measured frontier capability (proxy for t), including strict liability for catastrophic externalities.
- Sectors: AI, biotech, critical infrastructure software.
- Why it maps: Both the floor underline τ and the ceiling τR move earlier when λ increases; stronger perceived tail risk deters racing.
- Assumptions/dependencies: Insurability of tail risk; actuarial proxies; legislative frameworks.
- Experimentation and measurement programs to estimate λ(t) and payoff indices
- What: Red-team programs tied to calibrated risk ladders; capability evals to estimate marginal payoff gradients; publish anonymized aggregate curves.
- Sectors: Academia–industry partnerships in AI/bio/robotics.
- Tools/products: Standardized eval suites; open curves for D_i log π and λ; sandboxed testbeds.
- Why it maps: Reliable inputs are needed for threshold calculations and for credible cross-firm coordination.
- Assumptions/dependencies: Shared metrics; data-sharing agreements; privacy-preserving aggregation.
- Simulation tools for “noisy monitoring” coordination stress tests
- What: Agent-based simulators that vary ω (detection lag), trust priors, and contract terms to stress-test governance proposals.
- Sectors: Regulators, standards bodies, corporate risk teams.
- Why it maps: Model shows regimes where transparency helps/hurts and where racing equilibria persist; simulations guide policy design.
- Assumptions/dependencies: Calibrations; scenario diversity; model risk disclosure.
- Team-level norms in open-source and research communities
- What: Project charters with “pause-on-red-flags” norms, public stop announcements, and mirrored issue trackers to reduce detection lag.
- Sectors: Open-source AI/biology tooling; academic consortia.
- Why it maps: Practical translation of high-ω observability to distributed teams.
- Assumptions/dependencies: Maintainer adherence; code-of-conduct enforcement; minimal legal risk for public pauses.
Long-Term Applications
These applications likely require further research, scaling, international alignment, or new infrastructure.
- Treaty-grade verification architectures for instant pause observability
- What: International systems for cryptographically authenticated, tamper-evident stop signals; hardware-rooted compute attestation; near-real-time regulator mirrors.
- Sectors: AI safety treaties, bio nonproliferation, autonomous weapons control.
- Potential workflows: Hardware TEEs → attest training/experiment states; notarized logs → regulators/peers; automated halt propagation to facilities.
- Dependencies: Hardware standards; cross-border legal agreements; resilience against spoofing; governance for false positives.
- Global “capability-based” release gates tied to model-evaluated λ(t)
- What: Binding, cross-firm stage gates where progressing beyond capability tier k requires safety evidence that pushes D_i log π below λ or demonstrates risk mitigation; automatic moratorium triggers.
- Sectors: AI foundation models, gene-editing platforms, autonomy stacks.
- Tools/products: Tiered capability taxonomy; independent evaluators; universal “red flag” triggers.
- Dependencies: Consensus on capability tiers; scalable evaluation; enforcement and dispute resolution.
- Market designs that make “stop-first” dominant
- What: Safety prize mechanisms and procurement scoring that over-reward demonstrable early stops or shared safety R&D; reputational markets anchored to verified pauses.
- Sectors: Public procurement, philanthropic challenges, ESG-linked finance.
- Why it maps: Reshapes payoff function π so that the best response favors early stopping even with noisy monitoring and mixed trust.
- Dependencies: Verifiable metrics; gaming resistance; sustained funding.
- Antitrust and IP reforms to structurally lower D_i log π(t,t)
- What: Policies that reduce steepness of winner-take-most competitions (data portability, compulsory licensing after thresholds, limits on exclusive compute access).
- Sectors: Digital markets, compute providers, data-rich platforms.
- Impact: Lowers τR sustainably across technologies prone to arms races.
- Dependencies: Legal reforms; innovation incentives balance; global harmonization.
- Global safety bonds and catastrophe reinsurance pools
- What: Large, pooled “safety bonds” that are forfeited upon verified dangerous continuation past agreed thresholds; reinsurance for catastrophic externalities priced to λ(t).
- Sectors: AI/bio ecosystems; sovereign backstops.
- Dependencies: Capitalization; moral hazard controls; adjudication institutions.
- Standardized estimation pipelines for π and λ across domains
- What: Longitudinal programs to co-estimate payoff gradients and hazard functions from incidents, near-misses, and evaluation suites; public benchmarks.
- Sectors: Academia, standard bodies (NIST/ISO), regulators.
- Tools/products: Reference models for λ(t); domain transfer methods; uncertainty quantification.
- Dependencies: Data access; privacy tech; community adoption.
- Education and training for “coordination under existential risk”
- What: Curricula for executives, policymakers, and engineers on the strategic dynamics identified (upper/lower bounds, transparency–trust trade-offs, attrition risks).
- Sectors: Business schools, public policy schools, professional societies.
- Outputs: Playbooks; tabletop exercises; certification courses.
- Dependencies: Case libraries; cross-sector faculty; sustained demand.
- Extensions to multi-firm ecosystems and platform-mediated governance
- What: Apply n-firm versions with heterogeneous actors (including “crazy types”) to cloud platforms or app stores that can enforce pauses via platform rules.
- Sectors: Cloud/AI platforms, package registries, app stores.
- Tools/products: Platform-level kill-switches; developer policy clauses; graduated enforcement.
- Dependencies: Platform leverage; fairness and due process; resistant forks.
- Adaptive transparency policies accounting for “double-edged” effects
- What: Regulators deploy dialable transparency (ω) coupled with trust-building programs; use model-based triggers to raise/lower reporting speed to keep early-stopping equilibria viable.
- Sectors: Tech regulation broadly.
- Why it maps: At intermediate trust, more transparency can temporarily destroy cooperative equilibria; adaptive designs mitigate this.
- Dependencies: Real-time policy analytics; stakeholder feedback loops; statutory flexibility.
- Cross-domain application: finance, energy, and climate interventions
- Examples:
- High-frequency trading risk races: exchange-level circuit breakers and instant halt observability to prevent leverage “frontier” overshoot.
- Geoengineering R&D: verification and pause triggers tied to measured planetary risk indices.
- Nuclear/fusion scaling: capability-linked oversight that raises effective λ(t) with staged deployments.
- Dependencies: Domain-specific indices; sector regulators; international norms.
Notes on Core Assumptions/Dependencies (common across items)
- Measurability: Organizations can estimate or proxy π (marginal payoff from being ahead) and λ (risk that scales with frontier capability).
- Observability: Feasible, authenticated channels exist to make “stop” decisions visible to rivals quickly.
- Enforceability: Contracts, audits, and penalties are legally credible across jurisdictions.
- Actor types: Most key actors are rational and responsive to incentives; residual “crazy types” are rare but must be accounted for via designs robust to their presence.
- Antitrust/IP: Coordination devices are designed with competition law in mind and avoid collusion on prices/outputs, focusing narrowly on safety-related pauses.
- Data and privacy: Transparency systems protect trade secrets and sensitive data while proving relevant facts (e.g., via cryptographic attestations).
Collections
Sign up for free to add this paper to one or more collections.