IBM CUGA: Hierarchical Generalist Agent

• IBM CUGA is a hierarchical generalist agent defined by a layered planner–executor framework that optimizes utility and ensures auditability in diverse enterprise environments.
• Benchmark evaluations on WebArena and AppWorld demonstrate CUGA's superior success rates and operational efficiency compared to state-of-the-art models.
• Pilot deployments in business process outsourcing settings highlight significant reductions in response latency, enhanced reproducibility, and substantial development cost savings.

IBM Computer-Using Generalist Agent (CUGA) is a hierarchical, generalist agentic system architected for robust, auditably safe, and scalable deployment in heterogeneous enterprise environments, including APIs, web user interfaces, command-line interfaces, and file systems. CUGA is distinguished by its layered planner–executor control stack and was evaluated both on state-of-the-art academic benchmarks and in a production-aligned Business-Process-Outsourcing (BPO) pilot. Its design, benchmark performance, and integration requirements address enterprise-grade demands for scale, audit trails, safety, and operational governance (Shlomov et al., 27 Oct 2025).

1. System Architecture and Formal Model

CUGA is structured as a hierarchical planner–executor. Agentic task completion is formally modeled as a constrained utility maximization problem: for user instruction $I$, initial state $S_0$, and available sub-agents $\mathcal{A}$ (category: API, Browser, CLI, Code), the planner emits a sequence of sub-task goals $P = (g_1,\dots,g_K)$. For each $g_k$, a specialized executor $\pi_{\mathrm{exec}}^{(a)}$ (with $a\in\mathcal{A}$) produces a trajectory $\tau_k$. The aggregate execution trace is denoted $\Tau = (\tau_1,\dots,\tau_K)$. A reward function $R(\Tau)\in[0,1]$ encodes success; resource cost is $S_0$0 (with $S_0$1 the $S_0$2-th low-level action, $S_0$3 its cost). The planner–executor solves

$S_0$4

where $S_0$5 governs cost-performance tradeoff.

CUGA’s architecture implements three nested layers:

• Chat/Context Layer: Maintains conversational context and manages immediate Q&A (ChatAgent).
• Outer Loop (Task Planning): Agents for complexity classification (TaskAnalyzerAgent), sub-goal decomposition (TaskDecompositionAgent), persistent task ledger maintenance (PlanControllerAgent), gatekeeping for HITL (Human-In-The-Loop), and scheduling retries.
• Inner Loop (Sub-task Execution): Specialized sub-agents receive atomic sub-goals, employ schema-grounded prompting and reflection-based error handling, and maintain exhaustive provenance logs for every tool invocation.

The API sub-agent stack incorporates tool selection (ShortlisterAgent) using OpenAPI schema registries, code synthesis (APICodePlannerAgent + CodeAgent), execution in resource-constrained sandboxes, and reflective validation. Browser sub-agents (used for benchmarks) include DOM-based planning (BrowserPlannerAgent), Playwright-based actuation (ActionAgent), and DOM-to-Markdown extraction (QAAgent).

2. Benchmark Performance: WebArena and AppWorld

CUGA was evaluated on two prominent academic settings:

Benchmark	Level	CUGA Metric	Next Best (model)
WebArena	Overall SR (%)	61.7	58.1 (Operator [OpenAI '25])
AppWorld	TGC (%) / SGC (%)	Test-Normal: 73.2 / 62.5	72.6 / 53.6 (Qwen2.5)
		Test-Challenge: 57.6 / 48.2	47.2 / 28.8 (Qwen2.5)

• WebArena: Comprises 812 real-world web tasks across 6 domains and a multi-app setting. CUGA's overall Success Rate (SR) is 61.7%. Subdomain-specific SRs include Reddit (75.5%) and GitLab (61.7%). CUGA leads the published academic leaderboard.
• AppWorld: Consists of tasks across 9 applications (457 APIs). On the Normal set, CUGA achieves Task Goal Completion (TGC) of 73.2% and Scenario Goal Completion (SGC) of 62.5%, both improvements over prior systems (Shlomov et al., 27 Oct 2025).

This benchmark validation demonstrates that CUGA delivers strong generalization across unseen tools and workflows, exceeding the accuracy and compositional flexibility of other large-scale agents in their respective test splits.

3. Domain-Specific Evaluation: BPO-TA Benchmark

To bridge academic and enterprise production settings, the BPO-TA benchmark was introduced: 26 analytic decision-support tasks spanning 13 redacted, read-only endpoints typical in talent acquisition.

• Task Catalog: Includes lookups, cross-API joins, iterative comparisons, provenance reporting, and deliberate tasks intended to trigger graceful failure (unsupported queries).
• Data schema: Minimal OpenAPI per endpoint; schema-grounded prompts strictly enforce canonical definitions (e.g., for SLA, hiring conversions).
• Execution requirements: Every agent-produced output is accompanied by a provenance log $S_0$6endpoint, parameters, timestamp$S_0$7, with a deterministic parser validating low-level outputs before API invocation.

Selected tasks involve joining across endpoints (e.g., merging candidate volume with recommendation summaries), aggregative loops (e.g., skill impact iteration), and strictly returning “feature not supported” responses for unsupported queries.

4. Pilot Deployment in Enterprise Production

CUGA was deployed as a browser extension within a professional recruitment dashboard for a major BPO operation.

• Integration points: API/Tool Hub for endpoint management, browser extension interception for contextual cues (e.g., requisition IDs), and a central logging service for audit and regression.
• System requirements: Fast onboarding of >13 APIs (hours, not weeks), persistent task ledger with JSON-schema validation, all interactions riding through a configurable HITL gate, and strict read-only mode for data governance.
• User experience: Context-aware; agents pre-fill context-sensitive parameters by reading the visible UI state.

Baseline and pilot results are summarized as:

Metric	Manual Baseline	CUGA Pilot
Avg. time-to-answer	~20 min	~2–5 min
Reproducibility	~60 %	~95 %
Responses w/ provenance	~40 %	~92 %
Projected dev time saving	–	~90 % reduction
Projected dev cost saving	–	~50 % reduction

In benchmarked pilot runs, overall task accuracy across 26 tasks was 87%; valid first-try rate was 78%; 95% of responses included provenance; average latency per query was 11.2 seconds. Analyst-reported reproducibility was 4.6/5. Failures occurred mainly on unsupported cross-application queries.

5. Technical and Organizational Lessons

The deployment yielded several technical and organizational insights:

• Technical:
  • Minimal, well-curated OpenAPI schemas strengthen prompt reliability.
  • Schema-grounded prompting, error reflection, code sandboxing, and layering of interrupt/reflection nodes collectively reduced the system’s error rates by approximately one third.
  • Provenance capture with every call, persistent contact-ledger, and regression tests underpinned auditability and rapid debugging.
  • Centralized extensibility in the API/Tool Hub reduced per-API onboarding overhead from weeks to hours.
• Organizational:
  • Fine-grained, configurable HITL enforced human approval at critical workflow stages, accommodating enterprise governance.
  • Domain-specific benchmarks such as BPO-TA were essential for real-world adoption, highlighting the practical gap between academic success and enterprise-readiness.
  • Effective operationalization required deep collaboration between research and business operational teams.

Ablation experiments confirmed that reflective retries improved accuracy (+11 pp) and variable tracking bolstered reproducibility (+15 pp).

6. Next Steps and Roadmap

The ongoing development trajectory of CUGA targets:

1. Expanding HITL configuration with customizable autonomy controls.
2. Policy engine to enable enterprise-controlled write actions (future support for create/update API calls).
3. Static and dynamic reasoning short-circuits for cost–accuracy optimization.
4. Trajectory reuse via macro-action compendium extracted from successful execution traces.
5. Mixed-model deployment, with smaller models for sub-tasks and large LLMs for complex planning.
6. Expansion of evaluation to TauBench2 for validating compliance in multi-actor enterprise workflows.

Appendices provide comprehensive architectural diagrams, detailed BPO-TA task definitions, and benchmark breakdowns (Shlomov et al., 27 Oct 2025).

7. Significance and Implications

The CUGA system evidences that production-oriented generalist agents can be made to satisfy stringent enterprise requirements while preserving high task accuracy and operational transparency. Empirical results indicate substantial reductions in workflow latency, developer effort, and baseline cost, with robustness to task variety and domain shift. The systematic integration of provenance, layered planning, and human-in-the-loop affordances addresses long-standing obstacles for agentic automation in regulated environments.

A plausible implication is that research-grade agentic architectures, when augmented with domain-specific benchmarks, schema-grounded prompting, and operationally aligned governance features, are viable pathways for bridging the gap from academic benchmarks to enterprise-impactful deployment.