$\texttt{YC-Bench}$: Benchmarking AI Agents for Long-Term Planning and Consistent Execution

Abstract: As LLM agents tackle increasingly complex tasks, a critical question is whether they can maintain strategic coherence over long horizons: planning under uncertainty, learning from delayed feedback, and adapting when early mistakes compound. We introduce $\texttt{YC-Bench}$, a benchmark that evaluates these capabilities by tasking an agent with running a simulated startup over a one-year horizon spanning hundreds of turns. The agent must manage employees, select task contracts, and maintain profitability in a partially observable environment where adversarial clients and growing payroll create compounding consequences for poor decisions. We evaluate 12 models, both proprietary and open source, across 3 seeds each. Only three models consistently surpass the starting capital of \$200K, with Claude Opus 4.6 achieving the highest average final funds at \$1.27 M, followed by GLM-5 at \$1.21 M at 11$\times$ lower inference cost. Scratchpad usage, the sole mechanism for persisting information across context truncation, is the strongest predictor of success, and adversarial client detection is the primary failure mode, accounting for $47\%$ of bankruptcies. Our analysis reveals that frontier models still fail through distinct failure modes such as over-parallelization, demonstrating the capability gaps for long-horizon performance. $\texttt{YC-Bench}$ is open-source, reproducible, and configurable.

• The paper introduces YC-Bench to assess LLM agents' long-horizon coherence in strategic planning and execution over a simulated year-long startup operation.
• It employs a persistent scratchpad and structured CLI commands to manage memory, risk, and resource allocation under POMDP conditions.
• Empirical analysis shows that only a few models sustain profitability, highlighting the critical role of strategic scratchpad usage and frequent tool actions.

YC-Bench: A Benchmark for Long-Term Coherence in LLM Agents

Benchmark Design and Motivation

YC-Bench addresses a critical evaluation gap for LLM agents: the capacity for sustained strategic coherence across hundreds of compounding steps in realistic, partially observable environments. The benchmark simulates one year of startup operation, requiring agents to navigate dynamic task selection, risk management, resource allocation, and financial optimization in the presence of adversarial clients and delayed rewards.

The environment formalizes the challenge as a POMDP, combining deterministic but unknown transition and observation dynamics. Agents interact through a CLI interface, issuing explicit commands for observation, task selection, employee assignment, client management, and memory maintenance—specifically via a persistent scratchpad, which serves as the only cross-turn information persistence mechanism in the presence of context truncation. The design is adversarial: a significant fraction of clients inflate work requirements after task acceptance, requiring robust inference of reliability from episodic task outcomes.

Figure 1: Overview of YC-Bench—agent interaction via structured CLI commands, state tracking, and hidden adversarial client dynamics.

Empirical Analysis: Model Performance and Behavioral Insights

YC-Bench evaluates twelve LLM models (both proprietary and open-source), benchmarking their ability to maximize final company funds under the constraints of the environment. The results identify substantial gaps in long-horizon agentic abilities that are not captured by standard benchmarks such as AgentBench or SWE-Bench.

Of twelve models, only five turn a profit, and just three (Claude Opus 4.6, GLM-5, GPT-5.4) achieve substantial capital growth (>5× starting funds). These models distinguish themselves by early client trust concentration, strategic avoidance of adversarial clients, and frequent use of the scratchpad for memoization of key facts and decision rules. Conversely, most models distribute effort indiscriminately, suffer avoidable task failures, or go bankrupt due to payroll escalation and repeated engagement with unreliable clients.

Figure 2: Funds trajectory over time for 12 models, showing divergence and bankruptcy events; only three models sustain substantial profit across runs.

Scratchpad usage emerges as the dominant predictor of long-term success: top models record decision heuristics, client reliability profiles, and operational strategies persistently, enabling adaptive behavior despite context truncation. The three high-performing models execute 3–4× more tool actions per run than their counterparts, with the largest gaps in task inspection and scratchpad updates, leading to superior trust-building and task throughput.

Figure 3: Tool usage analysis—top models significantly outpace others in command execution, inspection, and persistent memory writing.

Failure Modes and Long-Horizon Coherence Analysis

Analysis of failed tasks reveals three primary modes: (1) repeated engagement with adversarial clients due to poor memory and policy persistence, (2) suboptimal employee assignment (incorrect productivity estimation, excessive task concurrency), and (3) over-committed employees leading to payroll-driven financial collapse. Identifying adversarial clients—who represent ~35% of the market—is particularly challenging; models that ignore or fail to blacklist unreliable clients accept adversarial tasks at or above market prevalence, resulting in cascading failures and negative feedback loops.

Figure 4: Adversarial client engagement rate; dashed line denotes natural prevalence, with most models overexposed relative to top performers.

Behavioral spectrum analysis demonstrates that even high-scoring models occasionally violate their own strategic guidelines, evidencing a reasoning–execution gap. Lower-performing models exhibit rigid action loops, lack reflection, or fail to act even when diagnostic information is correctly recorded. This exposes the lack of unified deliberation–execution capabilities in most current LLM agents, confirming that long-horizon coherence is a pipeline of perceive → record → retrieve → act, with failure points at every stage.

Figure 5: Representative decision moments; panels highlight diverse reasoning–execution lapses across model classes.

Implications, Limitations, and Directions for Future Research

YC-Bench illuminates persistent gaps in long-horizon agentic reasoning, memory management, and adaptive choice-making for modern LLMs. Model size and API cost do not consistently correlate with performance; cost-efficient open-source models can outperform proprietary baselines despite lower absolute returns. Practical deployment architectures for agentic LLMs must prioritize robust memory interfaces (e.g., persistent external scratchpads), dynamic risk inference schemes, and persistent strategy adaptation for production readiness. The environment’s deterministic structure, limited employee roster, and explicit numerical outputs simplify some aspects; future work should introduce stochasticity, dynamic workforce composition, and more naturalistic observation modalities to further stress test agentic coherence and adaptation.

The benchmark’s POMDP structure, persistent adversarial pressure, and compounding financial constraints present nontrivial sequential decision-making challenges not tractable with short-term heuristics, simple retrieval, or context-only memory solutions. Empirical evidence suggests a need for architectures integrating deliberate memory, policy refinement, and explicit execution mapping between reasoning and action.

Conclusion

YC-Bench establishes a rigorous new standard for evaluating the long-horizon coherence, planning, and execution consistency of LLM agents (2604.01212). The empirical findings—only three models consistently maintain profitability, and the strongest predictor of success is persistent, strategic memory usage—underscore significant gaps in agentic robustness and the need for improved architectures, inference strategies, and evaluation schemes. The benchmark is open-source and extensible, inviting community-driven model stress-testing on compounding sequenced tasks and real-world adversarial dynamics. Future research can leverage YC-Bench to drive advances in memory systems, policy adaptation, and unified deliberative agent architectures.