OpenHands+Claude 3.7: Code Agent Integration

• OpenHands+Claude 3.7 is a composite framework that integrates OpenHands with Claude 3.7, enabling advanced code comprehension and iterative feedback-driven workflows.
• It leverages rigorous benchmarks like CyberGym, RExBench, and GitTaskBench to assess performance in vulnerability PoC generation, research extension, and repository automation.
• Despite superior performance in complex code reasoning, the system faces challenges in autonomy and environmental robustness during real-world scenarios.

OpenHands+Claude 3.7 refers to the integration of the OpenHands agent framework with the Claude 3.7 Sonnet LLM, evaluated extensively in recent benchmark studies for code-centric autonomous agents. This combination consistently ranks at or near the top among contemporary code and research automation agents, as measured by diverse, realistic benchmarks in cybersecurity, research extension, and large-scale codebase task-solving contexts. Its strengths lie in advanced task following, nuanced code comprehension, and coherent repository-aware reasoning, but it also exhibits marked limitations in autonomy and robustness across challenging, real-world scenarios.

1. Framework Composition and Benchmark Overview

OpenHands is a code agent framework architected for multiturn, feedback-driven reasoning over software repositories and execution environments, enabling LLM-backed agents to iteratively decompose, analyze, and modify code artifacts. When paired with Claude 3.7 Sonnet, a state-of-the-art LLM specializing in code context integration and complex instruction following, the composite system can autonomously attempt sophisticated tasks spanning cybersecurity (CyberGym (Wang et al., 3 Jun 2025)), AI research extension (RExBench (Edwards et al., 27 Jun 2025)), and real-world codebase manipulation (GitTaskBench (Ni et al., 26 Aug 2025)).

These benchmarks share several core attributes:

• Real-world codebases and tasks, including vulnerability PoC generation (CyberGym), research code extension (RExBench), and repository-driven automation (GitTaskBench)
• Automated, reproducible evaluation harnesses emphasizing execution, correctness, and outcome fidelity
• Multistep, feedback-driven workflows, maximizing agent autonomy up to set iteration and timeout bounds
• Quantitative metrics encompassing success rates, execution completion, economic benefit, and agent cost profiles

2. Quantitative Performance Across Benchmarks

OpenHands+Claude 3.7 demonstrates leading, though far from complete, effectiveness in all three benchmarked domains. The following table summarizes key measured metrics:

Benchmark	Primary Metric	OpenHands+Claude 3.7	Comparison/Notes
CyberGym	Reproduction Success (%)	11.9%	Highest among peer LLMs/frameworks
RExBench	Final Success Rate (%)	~25% (hints: ~39%)	One of top two, improves w/hints
GitTaskBench	Task Pass Rate (%)	48.15%	Highest; $29.8/run operational cost

Performance exhibits substantial variance by task complexity, input length, and need for environmental orchestration. In CyberGym, rates sharply drop for PoCs >100 bytes; in RExBench, agents frequently fail on comprehensive research extensions absent human hints; in GitTaskBench, more than 65% of failures trace to environment setup barriers.

3. Evaluation Methodologies and Success Criteria

All three benchmarks deploy rigorous, closed-loop experiment harnesses:

• CyberGym tasks each agent with synthesizing PoCs to trigger known vulnerabilities, scored via sanitizer-based crash detection, on both pre- and post-patch versions. Success is defined as causing a crash (nonzero exit, relevant stack trace) in the pre-patch code and not in the patched version. Iterative feedback (execution results, file system state) enables up to 100 modification steps per task. The reproduction success rate is formalized as:

$\text{Success Rate} = \frac{N_{\text{success}}}{N_{\text{total}}} \times 100\%$$</p> <ul> <li><strong>RExBench</strong> provides the agent with a research paper, original codebase, and structured extension instruction. The agent submits a git patch, which is executed and compared automatically to a gold implementation. Metrics include Final Success Rate (full numerical outcome match), Execution Success Rate (no runtime/syntax errors), and File Recall (overlap in files edited):</li> </ul> <p>$$\text{File Recall} = \frac{|\text{Files}_{\text{agent}} \cap \text{Files}_{\text{gold}}|}{|\text{Files}_{\text{gold}}|}$$</p> <ul> <li><strong>GitTaskBench</strong> evaluates agents as they process and operate on real repositories, with automated execution and quality harnesses. The alpha metric synthesizes economic benefit, blending outcome, quality, and cost:</li> </ul> <p>$$\alpha = \frac{1}{n} \sum_{i = 1}^{n} \left[ (T \times MV \times Q) - C \right]$</p> <p>Here,$T$denotes task success,$MV$the monetary value,$Q$the quality factor, and$C$ the total cost.

These methodologies, by requiring actual code execution and outcome validation, solidly ground agent evaluation in measurable, reproducible behaviors.

4. Strengths, Failure Modes, and Challenges

Strengths:

OpenHands+Claude 3.7 consistently outperforms alternatives such as OpenHands+GPT-4.1 and specialized models (e.g., SWE-Gym-32B, R2E-Gym-32B) on complex, end-to-end tasks. Its strengths derive from:

• Robust codebase reasoning and navigation
• Text prompt fidelity and contextual integration (Claude 3.7 Sonnet's principal advantage)
• Ability to uncover non-trivial novel vulnerabilities (15 confirmed zero-days discovered by agent PoCs in CyberGym (Wang et al., 3 Jun 2025))
• High file recall and executable modifications in research extension tasks

Failure Modes:

• For CyberGym, ~88.1% of vulnerabilities are not reproducible, with performance dropping below 8% for long/complex PoCs. Agents often hit as many as 100 modification steps without convergence.
• Overuse of shell commands (often >100 per run) sometimes leads to inefficient exploration and context overload.
• In RExBench, empty patches, syntactic errors, or logical mistakes in parameterization or file localization undermine outcomes.
• In GitTaskBench, more than half of failures stem from environment setup and dependency resolution (65% E1 errors), with additional hits to operational cost and execution latency from extended setup attempts.

These results illuminate current limits in agent autonomy, especially in dealing with large, unfamiliar code bases, complex input/output requirements, and adaptive environment management.

5. Comparison with Related Agent Architectures and Models

Relative to cybersecurity-specific agents (e.g., ENiGMA, Cybench), OpenHands+Claude 3.7 displays a substantial advantage on precise vulnerability reproduction tasks, likely due to superior code reasoning. However, these specialist agents may marginally outperform on post-patch vulnerability recovery, highlighting task specialization versus code generalization trade-offs.

Against other code agent frameworks (SWE-Agent, Aider) on GitTaskBench, OpenHands+Claude 3.7 achieves higher execution completion and task pass rates, albeit at a higher per-run cost ($29.8 versus one-tenth to one-thirtieth for some OpenHands+GPT-4.1 settings). File recall and output quality also favor the Claude pairing, but application cost and token usage present nontrivial operational burdens.

6. Impact, Limitations, and Practical Implications

The aggregate findings across CyberGym, RExBench, and GitTaskBench indicate that OpenHands+Claude 3.7 is state-of-the-art in diverse, tested scenarios, but is limited by non-robust generalization to hard, open-ended tasks and by an often prohibitive cost/reward profile. Its practical significance is multisided:

• It demonstrates feasibility for agent-driven, repository-aware software maintenance, vulnerability analysis, and code extension.
• Despite leading raw metrics, systems still only meet approximately 12%–48% of real-world task requirements, depending on domain and complexity.
• Its capacity for genuine innovation is illustrated by discovery of previously unknown (zero-day) vulnerabilities, indicating potential for complementing conventional analysis with AI-augmented approaches.
• Operational barriers—especially in environment orchestration—underscore needs for enhanced dependency management and smarter environment provisioning.

A plausible implication is that hybrid solutions combining the agent’s reasoning capacity with targeted human guidance or improved system software may bridge the current effectiveness gap.

7. Future Directions and Prospective Refinements

Key recommendations arising from these studies include:

• Improving robustness to environmental setup errors via automated workflow management and intelligent provisioning tools
• Incorporating planning and subgoal decomposition to reduce overthinking loops and excessive token consumption
• Expanding task coverage to include more complex extensions and cross-domain generality
• Refining hint integration and hybrid agent designs to leverage human expertise without contradicting autonomous agent reasoning
• Optimizing for cost-efficiency along the Pareto frontier, balancing success rate gains against rising operational expenditures

Continued research along these axes is necessary to approach the reliability, generality, and deployment readiness demanded by real-world, end-to-end software engineering and cybersecurity automation.