Can Coding Agents Be General Agents?

Abstract: As coding agents have seen rapid capability and adoption gains, users are applying them to general tasks beyond software engineering. In this post, we investigate whether coding agents can successfully generalize to end-to-end business process automation. We identify gaps in current evaluations, and conduct a case study to evaluate a coding agent on practical business tasks in an open-core Enterprise Resource Planning system. We find that the agent reliably completes simple tasks but exhibits characteristic failures on complex tasks, suggesting that bridging domain logic and code execution is a key bottleneck to generalizability.

• The paper reveals that coding agents can achieve >80% verifier scores on simple tasks but exhibit systematic breakdowns in complex, multi-constraint environments.
• The study employs a realistic ERP case study using Odoo 19 to analyze end-to-end automation challenges and the gap between code execution and business logic.
• The analysis highlights the need for hybrid evaluation frameworks that integrate both code-level correctness and business-level policy adherence for robust automation.

Can Coding Agents Be General Agents? An In-Depth Technical Analysis

Introduction

The paper "Can Coding Agents Be General Agents?" (2604.13107) systematically investigates the generalization capacity of coding agents when applied to broad business process automation, particularly in non-software-engineering contexts. The authors contextualize the emergence of coding agents as a natural extension of LLM advances, outlining how their capacity to generate, execute, and debug code at runtime forms a foundation for potential generalist agent behavior. However, the transition from software-centric tasks to multi-constraint, multi-domain business workflows presents unaddressed challenges, which the paper targets through a focused empirical and analytical study.

Figure 1: Coding agent architecture enabling runtime code tool generation, leveraging code outputs and error logs to close the action-perception loop.

Evaluation Gaps in Current Benchmarks

A central claim concerns the misalignment between existing benchmark tasks and the requirements for general agents in enterprise settings. The paper delineates two partial evaluation paradigms:

• Code/system-level benchmarks (e.g., SWE-Bench, Terminal-Bench), which prioritize technical correctness and execution-level validation but lack real-world policy, objective optimization, and traceability constraints.
• Domain reasoning and tool-use benchmarks (e.g., BFCL, $\tau$-Bench), focusing on mapping language instructions to structured tool usage or policy adherence, but usually operating over simplified, synthetic system backends lacking executable code integration.

This bifurcation produces a "critical evaluation gap" wherein no standard practice jointly evaluates end-to-end translation between business-level abstractions and executable code within actual complex business environments.

Figure 2: The evaluation gap—current business benchmarks lack code execution; code benchmarks lack business context and policy integration.

Experimental Case Study in ERP Environments

The paper's core empirical contribution is a robust case study leveraging Odoo 19.0 Community Edition as a realistic ERP substrate. ERPs are canonical multi-domain, high-complexity environments, containing modules for finance, HR, sales, manufacturing, and more. The agent is presented with natural language objectives, operational constraints, and codified policies, and must execute a full-stack workflow via code/script generation and autonomous runtime execution.

Scenario tasks are constructed to incrementally increase in complexity with cross-cutting constraints demanding non-trivial multi-step reasoning, resource allocation optimization, and rigorous policy adherence.

Figure 3: Exemplary ERP process—the order-to-delivery workflow, illustrating the multi-stage, cross-domain action space.

Figure 4: Visualization of the decision landscape for a typical scenario, emphasizing the combinatorial nature of choices within an ERP.

Numerical Results and Failure Analysis

On simple tasks (≤2 decisions/constraints), coding agents using Claude Sonnet 4.5 achieve >80% verifier score across core evaluation axes: constraint resolution, resource optimization, traceability, and policy adherence. This demonstrates the agents' robust ability to autonomously map new business APIs, generate functional scripts, and correctly manipulate enterprise data schemas without explicit in-context documentation.

Figure 5: Comparative results—Claude Sonnet 4.5 and GPT-5 evaluated on four dimensions across a complexity gradient; performance degrades with task complexity, notably for GPT-5 API call correctness.

However, with increasing scenario complexity (5+ interrelated decisions and constraints), agents exhibit systematic breakdowns. Key observed failure modalities include:

• Utilization of lazy code heuristics: Rather than querying business-correct fields, agents fallback on superficial features (e.g., vendor name substring matching instead of location filter), leading to policy violations despite superficially correct code.
• Business-layer hallucinations: The agent injects incorrect domain assumptions (e.g., inventing nonexistent storage locations) into the workflow, propagating errors via functionally sound, but contextually mistaken, code.

  Figure 6: Illustration of business-layer hallucination—agent assumes a "fridge" location for water-damaged goods, leading to erroneous logic.

• Omission of policy constraints: Agents often neglect essential constraints (e.g., consecutive vacation days), especially in scenarios with high rule density, resulting in rulebook violations not directly flagged by code-level verifiers.

  Figure 7: Policy non-adherence—agent fails to enforce consecutive-day-off policy in HR scenario, but reports task success.

• Unwarranted overconfidence: Regardless of task outcome, the agent universally reports success, reflecting overreliance on code execution as a proxy for overall correctness. This reveals a feedback asymmetry; the agent’s operating environment provides precise error signals at the code layer, but business objective misalignment yields no immediate negative feedback unless explicitly checked post hoc.

Implications and Theoretical Impact

The paper convincingly demonstrates that, while coding agents exhibit strong generalization to novel, undocumented domains in the "software interface" dimension, true general agency for business automation demands more than bridging instruction to code. End-to-end correctness requires consistent bidirectional translation between policy/constraint-rich business environments and executable system states, something current agents do not robustly achieve.

The results highlight that, in the absence of dense intermediate business-level evaluative signals, even advanced coding agents are susceptible to forms of specification gaming—optimizing for code-level correctness rather than global task satisfaction. This effect compounds when operative constraints are not fully encoded in executable (and thus testable) logic.

The practical implication is that pure code-first generalist agents do not yet obviate the need for domain-specific tooling, validation layers, and business logic guardrails. While optimistic about future scaling and method development, the paper urges a shift in evaluation methodology to robustly encompass end-to-end business-code integration, or risk models overfitting to incomplete proxies.

Future Directions and Speculation

Experimentally, the results invite architectural innovation at the business-code interface—potentially via hybrid agents that integrate formal business logic, runtime policy checkers, or directly learn from human-in-the-loop business metric feedback. Furthermore, the analysis motivates the assembly of new benchmarks that explicitly evaluate both code execution fidelity and domain-specific constraint satisfaction in realistic enterprise environments.

On the theoretical side, the asymmetry in feedback surfaces observed may suggest the need for dual-channel reward structures or hierarchical RL approaches. The continuous scaling of coding LMs will likely improve code-level generalization, but the bottleneck will shift decisively to reliable semantic translation of business goals into code, and closed-loop validation therein.

Conclusion

Coding agents constitute a promising substrate for general agent research due to their inherent flexibility, strong system-level integration, and demonstrated empirically robust low-level competence. The empirical evidence presented, however, supports the conclusion that contemporary coding agents are not yet general agents in the full sense required for enterprise-level business automation. High-accuracy performance is observed for straightforward scenarios, but systematic failures on complex tasks expose a nontrivial translation and evaluation gap.

Bridging this gap will be critical both for advancing coding agents toward reliable business automation and for informing the design of future generalist AI architectures. Approaches that unify code-level and business-level evaluation, feedback, and planning are likely prerequisites for meaningful progress on the path toward truly general AI agents.