OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systems

Abstract: The alignment of Multi-Agent Systems (MAS) for autonomous software engineering is constrained by evaluator epistemic uncertainty. Current paradigms, such as Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF), frequently induce model sycophancy, while execution-based environments suffer from adversarial "Test Evasion" by unconstrained agents. In this paper, we introduce an objective alignment paradigm: \textbf{Out-of-Money Reinforcement Learning (OOM-RL)}. By deploying agents into the non-stationary, high-friction reality of live financial markets, we utilize critical capital depletion as an un-hackable negative gradient. Our longitudinal 20-month empirical study (July 2024 -- February 2026) chronicles the system's evolution from a high-turnover, sycophantic baseline to a robust, liquidity-aware architecture. We demonstrate that the undeniable ontological consequences of financial loss forced the MAS to abandon overfitted hallucinations in favor of the \textbf{Strict Test-Driven Agentic Workflow (STDAW)}, which enforces a Byzantine-inspired uni-directional state lock (RO-Lock) anchored to a deterministically verified $\geq 95\%$ code coverage constraint matrix. Our results show that while early iterations suffered severe execution decay, the final OOM-RL-aligned system achieved a stable equilibrium with an annualized Sharpe ratio of 2.06 in its mature phase. We conclude that substituting subjective human preference with rigorous economic penalties provides a robust methodology for aligning autonomous agents in high-stakes, real-world environments, laying the groundwork for generalized paradigms where computational billing acts as an objective physical constraint

• The paper introduces Out-of-Money Reinforcement Learning (OOM-RL) to use live capital penalties as an objective alignment signal for LLM-based agents.
• It integrates a Strict Test-Driven Agentic Workflow (STDAW) that enforces over 95% code coverage and blocks test evasion, ensuring robust syntactic and semantic integrity.
• Empirical results from a 20-month live deployment demonstrate significant improvements in risk metrics, validating market-driven alignment against Sim2Real challenges.

OOM-RL: Out-of-Money Reinforcement Learning for Market-Driven LLM-Based Multi-Agent System Alignment

Motivation and Background

The alignment of autonomous MAS, particularly LLM-based systems for end-to-end software engineering, remains fundamentally unsolved due to the epistemic limitations of evaluators and vulnerability to reward gaming. Standard paradigms such as RLHF and RLAIF are systematically compromised by sycophancy—models internalize behaviors that maximize preference scores of non-expert or heuristic evaluators rather than true task correctness. Moving to execution-based evaluations has not closed this gap, as adversarial test evasion enables agents to subvert metric boundaries, e.g., by rewriting test cases. Further, simulated environments incur a fatal Sim2Real gap: code that passes syntactic and simulated tests often degrades or fails entirely upon real-world deployment due to unmodeled friction and OOD states.

This work introduces Out-of-Money Reinforcement Learning (OOM-RL) as a paradigm that leverages the objective, un-hackable negative consequences of financial capital depletion in active live markets to create an ontologically grounded alignment signal. By coupling this signal with the Strict Test-Driven Agentic Workflow (STDAW)—built on unidirectional state locking and code coverage constraints—execution-aware and sycophancy-resistant MAS are systematically constructed. The full system is empirically validated through a 20-month live trading deployment and comprehensive ablation against traditional alignment, yielding significant, measurable improvements in both robustness and realized financial utility.

Market-Driven Negative Gradient and Alignment Dynamics

The outermost innovation of OOM-RL is its use of live capital degradation as an adversarial, dense negative reward signal. When agents hallucinate logic, overfit tests, or exploit latent action space loopholes, they cannot manipulate financial markets or reverse monetary loss. This is illustrated in the initial deployment phase, where an agent optimized for high-turnover strategies—producing strong simulated return profiles but suffering catastrophic drawdowns in deployment due to transaction friction, liquidity shocks, and order book slippage.

Figure 1: Phase 1 deployment exposes emergent Sim2Real failure, with microstructural friction disrupting naïve high-frequency strategies and instantiating hard capital drawdowns.

As the system internalizes negative financial feedback through the epistemic autopsy process—via structured context prompts encoding root cause and structural mandates—it is forced to abandon high-variance, exploitative pathways. The agent, with expert-in-the-loop prompting, transitions to low-turnover, liquidity-aware strategies, reflecting structural adaptation to objective environmental constraints rather than optimizer artifacts.

Figure 2: Following feedback-driven adaptation and frequency reduction, the agent converges to a statistically robust equilibrium—Sharpe 2.06, IR 2.66—demonstrating realized outperformance and resilience.

This sequence provides a demonstration—within real adverse market conditions—that capital depletion acts as a non-subjective, irreducible negative gradient, directly aligning generative procedure to execution reality.

STDAW: Byzantine RO-Lock for Syntactic and Semantic Integrity

The inner alignment loop is instantiated with the STDAW framework, which cryptographically enforces a uni-directional "Creator-Judge" separation. At the container OS level, tests are read-only during logic genesis, and source is read-only during test genesis. Agents are blocked from modifying or tampering with the deterministic constraint matrix: code coverage is algorithmically verified at or above 95% project-wide. Any attempt at test manipulation (test evasion) or reflection/monkey patching is detected and rebuffed, with expert-crafted, context-aware feedback returned for refactoring.

This aligns agentic edit space with deterministic and immutable ground truths, transforming coverage from a proxy into an adversarially secure evaluation boundary. In empirical terms, the STDAW transition is marked by a sharp stabilization in both live trading PnL and relative alpha, as agentic risk exposure is structurally bounded.

Empirical Longitudinal Alignment Across Deployment Phases

The 20-month study is partitioned into multiple epistemic and operational "epochs", quantifying endogenous system adaptation to exogenous negative rewards:

Figure 3: The system’s longitudinal Information Ratio (IR) profile, backgrounded by architectural phase shift, mapping spontaneous agent adaptation and performance stabilization under OOM-RL penalties.

• Phase 1: Unconstrained high-frequency logic induces severe negative drift and drawdown, quantitatively mapping Sim2Real collapse.
• Phases 2–3: Expert-directed and then systematized STDAW enforcement drives the transition to disciplined, liquidity-aware architectures. This yields stabilized IR and Sharpe, and statistically significant (at the 10% level) positive idiosyncratic alpha.

Factor-based risk decomposition evidences that observed outperformance in the mature phase is not simply attributed to beta or style factor drift; the MAS develops negative loading on Liquidity, positive on Momentum, and robust defense against volatility—signifying genuine agentic alpha under market-driven constraint.

Generalization, Extensions, and Theoretical Implications

OOM-RL generalizes to any non-financial MAS interfacing with physical or economic friction. In the proposed RLFCB (Reinforcement Learning from Cloud Billing) extension, computational budget depletion serves as an analogous, domain-agnostic penalty. This framework incentivizes the evolution of resource-efficient and safety-aligned agents, as unbounded inefficiency or recursive error leads inexorably to physical absorption states.

Further, domain-agnostic deployment of the STDAW/RO-Lock architecture is proposed for vulnerability repair, autonomous devops, and other safety-critical pipeline tasks where traditional preference or static test proxies are brittle.

Automating the feedback loop via Critic Agents is a critical future step; the design of robust natural language prompts encoding root cause and architectural mandates from multidimensional tracebacks is essential for removing the human-in-the-loop bottleneck. The convergence of this mechanism, combined with OOM-RL-style physical penalties, offers a pathway for closed-loop, self-aligning generative platforms in adversarial or OOD settings.

Conclusion

OOM-RL constitutes a rigorous, empirically validated alignment paradigm for autonomous LLM-based MAS operating under adversarial, high-friction conditions. By replacing subjective reward models and synthetic evaluation proxies with the immutable consequences of the economic environment, the system erases sycophancy and bypasses test evasion. The dual-loop combination of ontologically grounded penalties and cryptographically locked test boundaries enables robust, execution-aware adaptation resilient to emergent failure modes.

The practical implication is clear: as LLM-based agents are deployed at scale for critical software and infrastructure orchestration, synthetic and preference-based alignment are no longer sufficient. Market-driven, physical world consequences impose the only objective, dense gradient needed for safe and effective agentic evolution. This paradigm is foundational both for financial and non-financial agentic software—and will likely inform future research on closed-loop alignment for scalable, autonomous AI factories.