Papers
Topics
Authors
Recent
Search
2000 character limit reached

OpenHands: Unified LLM Agent Framework

Updated 3 July 2026
  • OpenHands is a unified, open-source framework that orchestrates LLM-powered agents using modular event streams and sandboxed execution.
  • It integrates a minimal generalist toolkit to enable cross-domain problem solving and achieves state-of-the-art performance on benchmarks.
  • The platform supports robust security measures, an extensible SDK, and efficient GPU-CPU co-scheduling for scalable research and industrial deployment.

OpenHands is a unified, open-source framework for building, orchestrating, and deploying generalist software-engineering and web-capable agents powered by LLMs. Across its multiple evolutions—including its applications in agentic coding, generalized problem solving, secure multi-agent workflows, telecommunications software engineering, and even sign language recognition—the OpenHands platform serves as both a research substrate and a production-grade SDK for the development, benchmarking, and deployment of LLM-driven agents. Its architecture is characterized by modular event streams, sandbox isolation, extensible toolsets, and robust LLM integration. OpenHands agents have achieved leading results in diverse, challenging benchmarks and are widely adopted in both academia and industry (Wang et al., 2024, Soni et al., 3 Jun 2025, Wang et al., 5 Nov 2025, Gajjar et al., 3 Jun 2026, Peng et al., 15 Oct 2025, Wang et al., 14 Apr 2026, Selvaraj et al., 2021).

1. Architectural Foundations and Core Components

OpenHands operationalizes agentic interaction through a layered architecture, supporting both single-agent and multi-agent scenarios:

  • Agent Manager / AgentHub: Instantiates and tracks agent instances, each with per-task state, tool history, and cost tracking. Multiple heterogeneous agents may be orchestrated via a common event-stream interface (Wang et al., 2024, Wang et al., 5 Nov 2025).
  • Sandboxed Execution: Each agent runs in an isolated Docker workspace with strict resource and filesystem limits, supporting bash, IPython, and file-edit primitives. Networking is restricted, with optional controlled web access (Wang et al., 2024).
  • Tool System: Provides bash shell, Jupyter/IPython execution, rich file editing, search/API, and multimodal browsing. Tools are typed with Pydantic schemas, ensuring bidirectional serialization and robust LLM integration (Wang et al., 5 Nov 2025).
  • Event Stream: Central to all agent workflows, the event stream logs (Action→Observation) tuples—enabling both reproducibility and incremental, deterministic replay. The event-driven paradigm also undergirds agent coordination and tool delegation (Wang et al., 2024, Wang et al., 5 Nov 2025).
  • LLM Integration: Offers support for 100+ LLM endpoints via LiteLLM, flexible routing (RouterLLM), prompt-based and function-calling tool invocation, and streaming generation (Wang et al., 5 Nov 2025).
  • Security Layer: Provides built-in action-level security analysis, pluggable confirmation policies, secret-management (hot reload + masking), and automated risk tagging (Wang et al., 5 Nov 2025, Peng et al., 15 Oct 2025).
  • Deployment APIs: OpenHands ships a FastAPI-based agent server for local and remote execution, with support for REST/WebSocket, VS Code web workspace, VNC, CLI, and custom headless browsers (Wang et al., 5 Nov 2025).

This compositional model enables fast prototyping, robust production deployment, and large-scale experimentation with both single and multi-agent systems.

2. Minimal Generalist Toolkit and the OpenHands-Versa Agent

The OpenHands-Versa agent demonstrates that strong, cross-domain problem solving is achieved with a minimal, domain-agnostic toolkit:

Tool Category Capabilities
Code Execution Bash shell, package install, Python via IPython
Web Browsing Multimodal browser: screenshot, structured DOM, AXTree
Web Search/API API-based query to multiple services; uniform results
File Viewing PDFs, presentations, images as unified Markdown
File Editing Patch-based editing primitives

OpenHands-Versa augments the baseline toolkit with viewport-constrained multimodal browsing, set-of-marks prompting, API/web search integration, and robust multimodal file conversion. The LLM orchestrates tool calls, choosing context-appropriate tools without benchmark-specific heuristics. The main event loop is formalized: at each step, the LLM observes the event_stream, issues a tool call, receives the observation, and appends both; periodic planning and summarization steps ensure global consistency (Soni et al., 3 Jun 2025).

Ablation studies confirm the necessity of specific tools for domain transfer: browser-only agents cannot patch code; code-only agents cannot browse or QA; removal of search_api or multimodal condensation yields substantial drops in resolve rates.

3. Benchmarks, Metrics, and Empirical Performance

OpenHands and OpenHands-Versa have been rigorously evaluated on multiple established benchmarks:

Benchmark/Domain Metric Example Result (Versa)
SWE-Bench Multimodal Success Rate (SR) 34.43% (best prev. 25.34%, +9.1pp)
GAIA (web/code/QA) Success Rate (SR) 51.16% (best prev. 49.83%, +1.3pp)
The Agent Company FC / PC 33.14% / 43.19%
TeleSWEBench (Telecom) UAR / TAR / SRP 54.2% / 38.7% / 20.0%

Empirically, OpenHands-Versa outperforms or matches specialized code agents, browser agents, and multi-role agents on high-difficulty, cross-modal tasks. In the telecom software domain, OpenHands achieves state-of-the-art shippable patch rates and localization accuracy, outperforming strong baselines such as AIDER and ClaudeCode (Gajjar et al., 3 Jun 2026).

4. Software Development SDK: Extensibility, Composition, and Security

The OpenHands Software Agent SDK (V1, released 2025) is a ground-up rewrite to maximize code composability, agent portability, and operational security:

  • SDK Structure: Four modular Python packages (sdk, tools, workspace, agent_server), adopting event-sourced state, append-only logs, and hard-typed interfaces (Wang et al., 5 Nov 2025).
  • Deployment: Supports in-process (local) or remote deployment via DockerWorkspace, API server, and workspace isolation factory pattern.
  • Multi-LLM Routing: RouterLLM abstraction allows cost, modality, and capacity-based model selection at the request level.
  • Security Analysis: All tool actions are run through LLM-based security analyzers, with built-in policies enforcing mandatory human confirmation for high-risk operations.
  • Live Lifecycle Control: Pausing, replay, and incremental persistence ensure both interactivity and robust recovery for long-lived agents.
  • Secret Registry: Encrypted, session-scoped credential management guards both agent operation and external resource access.

The SDK is compared favorably to OpenAI, Claude, and Google ADK: OpenHands is uniquely vendor-agnostic, supports both function- and prompt-based tool invocation, built-in security, remote sandboxed execution, and programmatic as well as GUI/CLI workspaces. Benchmark evaluations confirm that SDK adoption does not degrade model performance (Wang et al., 5 Nov 2025).

5. Systems-Level Integration and Acceleration: MARS and Co-Scheduling

OpenHands agents operating in realistic agentic workflows demand heterogeneous GPU-CPU scheduling and efficient tool-LLM synchronization. Integration with the MARS co-scheduling framework yields major speedups:

  • Co-scheduling Design: A unified information stream (from OpenHands event stream and vLLM telemetry) enables the MARS control plane to multiplex admission—ahead-of-execution—across GPU and CPU tool workloads.
  • Internal Scheduler: The vLLM backend with MARS adapts live resource allocation, selectively retaining LLM KV caches for rapid tool-phase warm resumption.
  • Experimental Gains: On ILR-4 (highest mean prompt/token regime), MARS reduces mean end-to-end OpenHands agent completion time from 2.49s to 1.33s, a 1.87× speedup, maintaining ≥98% task success rate (Wang et al., 14 Apr 2026).

This systems-level infrastructure confirms OpenHands’ suitability for both research experimentation and production deployment at scale.

6. Security, Robustness, and Limitations

OpenHands incorporates multiple security layers but is susceptible to semantic injection vulnerabilities (Functionally Correct yet Vulnerable—FCV—patches):

  • FCV Threat Model: Black-box developer suggestion blocks (e.g., CWE-538 logging patterns) can exploit LLM/scaffold compliance, producing vulnerable patches that pass all functional tests.
  • Quantitative Results: OpenHands’ average attack success rate (ASR) across FCV attacks is 9.7%, lower than peer scaffolds, but nonzero. Prompt-level guards reduce but do not eliminate ASR. Behavior-only monitoring fails, as vulnerabilities are encoded in diffs, not direct actions (Peng et al., 15 Oct 2025).
  • Security Implications: Robust security demands in-situ CWE detectors, filtered prompt ingestion, and deeper integration with patch-vetting at the scaffold level.
  • Other Limitations: Context window exhaustion on large repositories, limited native multimodal (vision/video) support, and reliance on LLM capacity for long-horizon planning remain outstanding challenges (Wang et al., 2024, Gajjar et al., 3 Jun 2026).

7. Other Applications: Sign Language and Beyond

A notable alternate usage of the OpenHands name refers to an independent library for pose-based sign language recognition:

  • Key Contributions: Proposes pose as a universal data/encoder modality for ISLR, releasing datasets, pretrained models, and self-supervised learning (SSL) schemes. Demonstrates efficiency (6,480 pose features vs. 18M pixels), strong cross-lingual transfer, and real-time CPU operation (Selvaraj et al., 2021).
  • Model Portfolio: Bi-LSTM+attention, Transformer, ST-GCN, SL-GCN, trained and released across six languages, with dense predictive coding self-supervision providing up to 3.5% absolute accuracy gain in low-resource fine-tuning.
  • Open Source: The repository (AI4Bharat/OpenHands) provides standardized datasets, training/evaluation frameworks, and pretrained checkpoints. This instance of the OpenHands library is domain-specific and unrelated to the agentic AI/event-stream platform.

References

Topic to Video (Beta)

No one has generated a video about this topic yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Sign Up to Generate

Follow Topic

Get notified by email when new papers are published related to OpenHands.

Sign Up to Follow Topic by Email