---
title: 'OpenHands: Unified LLM Agent Framework'
url: https://www.emergentmind.com/topics/openhands
type: topic
---

# OpenHands: Unified LLM Agent Framework

OpenHands is a unified, open-source framework for building, orchestrating, and deploying generalist software-engineering and web-capable agents powered by large language models (LLMs). Across its multiple evolutions—including its applications in agentic coding, generalized problem solving, secure multi-agent workflows, telecommunications software engineering, and even sign language recognition—the OpenHands platform serves as both a research substrate and a production-grade SDK for the development, benchmarking, and deployment of LLM-driven agents. Its architecture is characterized by modular event streams, sandbox isolation, extensible toolsets, and robust LLM integration. OpenHands agents have achieved leading results in diverse, challenging benchmarks and are widely adopted in both academia and industry [2407.16741, 2506.03011, 2511.03690, 2606.05001, 2510.17862, 2604.26963, 2110.05877].

## 1. Architectural Foundations and Core Components

OpenHands operationalizes agentic interaction through a layered architecture, supporting both single-agent and multi-agent scenarios:

- **Agent Manager / AgentHub:** Instantiates and tracks agent instances, each with per-task state, tool history, and cost tracking. Multiple heterogeneous agents may be orchestrated via a common event-stream interface [2407.16741, 2511.03690].
- **Sandboxed Execution:** Each agent runs in an isolated Docker workspace with strict resource and filesystem limits, supporting bash, IPython, and file-edit primitives. Networking is restricted, with optional controlled web access [2407.16741].
- **Tool System:** Provides bash shell, Jupyter/IPython execution, rich file editing, search/API, and multimodal browsing. Tools are typed with Pydantic schemas, ensuring bidirectional serialization and robust LLM integration [2511.03690].
- **Event Stream:** Central to all agent workflows, the event stream logs (Action→Observation) tuples—enabling both reproducibility and incremental, deterministic replay. The event-driven paradigm also undergirds agent coordination and tool delegation [2407.16741, 2511.03690].
- **LLM Integration:** Offers support for 100+ LLM endpoints via LiteLLM, flexible routing (RouterLLM), prompt-based and function-calling tool invocation, and streaming generation [2511.03690].
- **Security Layer:** Provides built-in action-level security analysis, pluggable confirmation policies, secret-management (hot reload + masking), and automated risk tagging [2511.03690, 2510.17862].
- **Deployment APIs:** OpenHands ships a FastAPI-based agent server for local and remote execution, with support for REST/WebSocket, VS Code web workspace, VNC, CLI, and custom headless browsers [2511.03690].

This compositional model enables fast prototyping, robust production deployment, and large-scale experimentation with both single and multi-agent systems.

## 2. Minimal Generalist Toolkit and the OpenHands-Versa Agent

The OpenHands-Versa agent demonstrates that strong, cross-domain problem solving is achieved with a minimal, domain-agnostic toolkit:

| Tool Category    | Capabilities                                           |
|------------------|-------------------------------------------------------|
| Code Execution   | Bash shell, package install, Python via IPython       |
| Web Browsing     | Multimodal browser: screenshot, structured DOM, AXTree|
| Web Search/API   | API-based query to multiple services; uniform results |
| File Viewing     | PDFs, presentations, images as unified Markdown       |
| File Editing     | Patch-based editing primitives                        |

OpenHands-Versa augments the baseline toolkit with viewport-constrained multimodal browsing, set-of-marks prompting, API/web search integration, and robust multimodal file conversion. The LLM orchestrates tool calls, choosing context-appropriate tools without benchmark-specific heuristics. The main event loop is formalized: at each step, the LLM observes the event_stream, issues a tool call, receives the observation, and appends both; periodic planning and summarization steps ensure global consistency [2506.03011].

Ablation studies confirm the necessity of specific tools for domain transfer: browser-only agents cannot patch code; code-only agents cannot browse or QA; removal of search_api or multimodal condensation yields substantial drops in resolve rates.

## 3. Benchmarks, Metrics, and Empirical Performance

OpenHands and OpenHands-Versa have been rigorously evaluated on multiple established benchmarks:

| Benchmark/Domain        | Metric                   | Example Result (Versa)                |
|-------------------------|--------------------------|---------------------------------------|
| SWE-Bench Multimodal    | Success Rate (SR)        | 34.43% (best prev. 25.34%, +9.1pp)   |
| GAIA (web/code/QA)      | Success Rate (SR)        | 51.16% (best prev. 49.83%, +1.3pp)   |
| The Agent Company       | FC / PC                  | 33.14% / 43.19%                      |
| TeleSWEBench (Telecom)  | UAR / TAR / SRP          | 54.2% / 38.7% / 20.0%                |

- **Metrics:** Task-level resolve rate ($SR$), full completion (FC), partial completion (PC), Pass@1 for first-try success, unit-test acceptance rate (UAR), TeleJudge semantic acceptance (TAR), end-to-end shippable patch percentage (SRP), mean task cost, speedup ratios for system integration [2506.03011, 2606.05001, 2511.03690, 2407.16741, 2604.26963].
- **Protocols:** All results use official test splits; temperature and MaxSteps are fixed; no benchmark-specific prompt augmentations are used.

Empirically, OpenHands-Versa outperforms or matches specialized code agents, browser agents, and multi-role agents on high-difficulty, cross-modal tasks. In the telecom software domain, OpenHands achieves state-of-the-art shippable patch rates and localization accuracy, outperforming strong baselines such as AIDER and ClaudeCode [2606.05001].

## 4. Software Development SDK: Extensibility, Composition, and Security

The OpenHands Software Agent SDK (V1, released 2025) is a ground-up rewrite to maximize code composability, agent portability, and operational security:

- **SDK Structure:** Four modular Python packages (sdk, tools, workspace, agent_server), adopting event-sourced state, append-only logs, and hard-typed interfaces [2511.03690].
- **Deployment:** Supports in-process (local) or remote deployment via DockerWorkspace, API server, and workspace isolation factory pattern.
- **Multi-LLM Routing:** RouterLLM abstraction allows cost, modality, and capacity-based model selection at the request level.
- **Security Analysis:** All tool actions are run through LLM-based security analyzers, with built-in policies enforcing mandatory human confirmation for high-risk operations.
- **Live Lifecycle Control:** Pausing, replay, and incremental persistence ensure both interactivity and robust recovery for long-lived agents.
- **Secret Registry:** Encrypted, session-scoped credential management guards both agent operation and external resource access.

The SDK is compared favorably to OpenAI, Claude, and Google ADK: OpenHands is uniquely vendor-agnostic, supports both function- and prompt-based tool invocation, built-in security, remote sandboxed execution, and programmatic as well as GUI/CLI workspaces. Benchmark evaluations confirm that SDK adoption does not degrade model performance [2511.03690].

## 5. Systems-Level Integration and Acceleration: MARS and Co-Scheduling

OpenHands agents operating in realistic agentic workflows demand heterogeneous GPU-CPU scheduling and efficient tool-LLM synchronization. Integration with the MARS co-scheduling framework yields major speedups:

- **Co-scheduling Design:** A unified information stream (from OpenHands event stream and vLLM telemetry) enables the MARS control plane to multiplex admission—ahead-of-execution—across GPU and CPU tool workloads.
- **Internal Scheduler:** The vLLM backend with MARS adapts live resource allocation, selectively retaining LLM KV caches for rapid tool-phase warm resumption.
- **Experimental Gains:** On ILR-4 (highest mean prompt/token regime), MARS reduces mean end-to-end OpenHands agent completion time from 2.49s to 1.33s, a 1.87× speedup, maintaining ≥98% task success rate [2604.26963].

This systems-level infrastructure confirms OpenHands’ suitability for both research experimentation and production deployment at scale.

## 6. Security, Robustness, and Limitations

OpenHands incorporates multiple security layers but is susceptible to semantic injection vulnerabilities (Functionally Correct yet Vulnerable—FCV—patches):

- **FCV Threat Model:** Black-box developer suggestion blocks (e.g., CWE-538 logging patterns) can exploit LLM/scaffold compliance, producing vulnerable patches that pass all functional tests.
- **Quantitative Results:** OpenHands’ average attack success rate (ASR) across FCV attacks is 9.7%, lower than peer scaffolds, but nonzero. Prompt-level guards reduce but do not eliminate ASR. Behavior-only monitoring fails, as vulnerabilities are encoded in diffs, not direct actions [2510.17862].
- **Security Implications:** Robust security demands in-situ CWE detectors, filtered prompt ingestion, and deeper integration with patch-vetting at the scaffold level.
- **Other Limitations:** Context window exhaustion on large repositories, limited native multimodal (vision/video) support, and reliance on LLM capacity for long-horizon planning remain outstanding challenges [2407.16741, 2606.05001].

## 7. Other Applications: Sign Language and Beyond

A notable alternate usage of the OpenHands name refers to an independent library for pose-based sign language recognition:

- **Key Contributions:** Proposes pose as a universal data/encoder modality for ISLR, releasing datasets, pretrained models, and self-supervised learning (SSL) schemes. Demonstrates efficiency (6,480 pose features vs. 18M pixels), strong cross-lingual transfer, and real-time CPU operation [2110.05877].
- **Model Portfolio:** Bi-LSTM+attention, Transformer, ST-GCN, SL-GCN, trained and released across six languages, with dense predictive coding self-supervision providing up to 3.5% absolute accuracy gain in low-resource fine-tuning.
- **Open Source:** The repository (AI4Bharat/OpenHands) provides standardized datasets, training/evaluation frameworks, and pretrained checkpoints. This instance of the OpenHands library is domain-specific and unrelated to the agentic AI/event-stream platform.

## References

- [2407.16741] OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- [2506.03011] Coding Agents with Multimodal Browsing are Generalist Problem Solvers
- [2511.03690] The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents
- [2606.05001] TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications
- [2510.17862] When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?
- [2604.26963] MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems
- [2110.05877] OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages

Source: https://www.emergentmind.com/topics/openhands