HARBOR: A Harness Framework for Agentic Robot Reinforcement Learning

Abstract: Reinforcement learning (RL) has become a powerful paradigm for robot learning, particularly in sim-to-real settings, but its broader adoption remains limited by the engineering pipeline surrounding the algorithms. Building tasks, shaping rewards, and tuning hyperparameters require substantial expert effort, making RL workflows costly and difficult to scale. We introduce HARBOR, an agentic framework that frames robot RL automation as a harness-engineering problem: given a simulator codebase and a task specification, it automates the workflow from environment setup to policy training in simulation. HARBOR decomposes such high-level objectives into bounded stages executed by specialized agents through standardized commands, persistent artifacts, executable gates, and reusable knowledge, and scales iteration via decentralized parallel trials and experience learning across runs. We evaluate HARBOR across 6 benchmarks and 16 tasks in total, spanning manipulation, locomotion, and bimanual dexterous control. We demonstrate that HARBOR automates the simulation RL workflow end-to-end, designs rewards, tunes algorithms to match or improve over default configurations, and reduces engineering effort at practical token and wall-clock cost; the resulting policies can also be transferred to real robots.

• The paper introduces HARBOR, a framework that automates robot RL pipelines by leveraging modular agents, standardized commands, persistent artifacts, and gate checks.
• It demonstrates significant efficiency gains through parallel tuning and sample optimization, achieving 88.5%-100% simulation success across diverse tasks.
• The approach enables scalable, audit-friendly deployment of RL policies while reducing engineering overhead and improving sim-to-real transfer capabilities.

HARBOR: An Artifact-Centric Harness Framework for Agentic Robot RL Automation

Framing Robot RL Workflow Automation as Harness Engineering

HARBOR operationalizes the automation of robot RL pipelines as a harness engineering problem, leveraging structured abstractions that facilitate long-horizon, verifiable, and robust workflow orchestration. In RL-based robot learning, the engineering bottleneck stems from task construction, reward synthesis, domain randomization, and hyperparameter tuning—processes that are traditionally labor-intensive and error-prone. HARBOR bridges this gap by introducing a modular harness architecture consisting of specialized agents, standardized commands, persistent artifacts, verifiable gates, and reusable knowledge, engineered to exploit the stable MDP interfaces and executable feedback that typify robot RL ecosystems.

The framework formalizes the RL harness as $$\mathcal{H}_{RL} = (\mathcal{H}_A, \mathcal{C}, \mathcal{M}, \mathcal{G}, \mathcal{K})$$, synergizing context-isolated agents (for stage-local task execution), composable commands, artifact persistence (for workflow state externalization), gate-based executability checks, and continual learning from prior traces. This results in bounded automation stages where design mistakes are gated and contained, preventing cascading errors and facilitating rapid debugging and auditability.

Figure 1: HARBOR's agentic architecture coordinates stage-local agents, commands, artifacts, gates, and knowledge, enabling end-to-end robot RL workflow automation from simulator setup to policy training.

Gate-Checked Execution and Parallel Tuning Protocols

HARBOR's execution protocol is defined by gate-checked stage advancement. Each workflow stage—dependency setup, task generation, reward synthesis, algorithmic integration, domain randomization, and hyperparameter tuning—is handled by a localized agent operating on bounded artifacts and prior knowledge. Gate checks validate interface behavior: e.g., verifying valid action-target mappings or confirming semantic correctness in rollout behaviors. Failures trigger repair attempts, with unresolved cases deferred for human intervention.

Reward and algorithm tuning exploit centralized control and decentralized parallel execution. The main agent manages trial history and dispatches sub-agents to execute isolated runs, aggregating structured feedback for post-hoc experience distillation. This protocol not only enhances throughput but also enables asynchronous scheduling and cluster-based scalability, yielding significantly improved sample efficiency and wall-clock reduction in iterative stages.

Figure 2: Parallel reward tuning—multiple sub-agents execute isolated trials, feedback is aggregated, and experience is updated centrally for iterative improvement.

End-to-End RL Workflow Generality and Real-World Validation

HARBOR's evaluations span six benchmarks and sixteen tasks, stress-testing its generality across manipulation, locomotion, and bimanual dexterous control, based on four diverse simulators: IsaacLab, ManiSkill, Genesis, and MJLab. The system demonstrates robust artifact-centric, simulator-agnostic workflow automation across all stages—including dependency extraction, task registration, reward generation, and policy training. Strong numerical results include $88.5\%$–$100\%$ simulation success rates in IsaacLab for complex tasks such as three-cube stacking, with effective sim-to-real transfer confirmed for manipulation and dexterous tasks (though lower success is observed in MJLab due to simulator-specific physics instability).

HARBOR's reward design substantially outperforms LLM-based baselines Eureka and REvolve, maintaining high success even in long-horizon composition and articulated-object manipulation tasks. The experiential memory enables rapid reuse of successful reward structures and heuristics, yielding an $8\times$ speedup on reward optimization for stack-cube when leveraging accumulated knowledge.

Figure 3: End-to-end policy learning across four tasks and four simulators, with real-world robot validation.

RL Algorithm Tuning and Sample Efficiency Gains

HARBOR automates algorithmic integration and hyperparameter tuning for PPO, SAC, and TD3, improving performance across multiple benchmarks (IsaacLab, Bi-DexHands, Loco-MuJoCo). Parallel tuning is constrained by a bounded wall-clock, and policy stabilization techniques are automatically applied. Tuned configurations elevate sample efficiency and final returns over defaults in $11$–$12$ task settings. Notably, HARBOR-tuned SAC solves challenging bimanual dexterous tasks (e.g., ShadowHandDoorOpenIn) where official baselines fail.

Figure 4: HARBOR's tuning produces solid learning curves surpassing default configurations across three benchmarks and four tasks each.

Figure 5: Learning curves for 16 simulator-task settings show consistent policy improvement and high task success rates post-training.

Ablation and Efficiency Analysis

Ablation studies on ManiSkill tasks indicate HARBOR's pillar mechanisms—centralized-control/decentralized-execution, gate validation, and experience reuse—are responsible for Pareto-optimality in reliability and computational cost. Removing isolation or gate validation results in longer wall-clock times, higher token consumption, and masked defects. Absence of accumulated experience markedly lowers success rates, especially in generative and tuning stages. The harness architecture is faster and cheaper than vanilla LLM agents, with reliability maintained even when instantiated with weaker base models.

Implications and Future Directions

HARBOR represents a scalable approach to RL workflow automation in robotics, substantively lowering engineering barriers for sim-to-real policy deployment. Its harness abstraction facilitates modular extensibility—supporting custom user workflows, auditability, and adaptation to new simulators or algorithmic paradigms. The persistent experience and agentic decomposition enable increasing efficiency and reliability as more runs accumulate, pointing toward continual improvement with increased usage.

Practically, HARBOR can democratize RL-based robot learning, enabling rapid experimentation and deployment for both single-task and scalable generalist policies across configurations. Theoretically, its harness-centric view may generalize to other long-horizon agentic applications, laying groundwork for verifiable and robust automation interfaces in complex environments.

Remaining limitations include dependence on the available tools and prior knowledge; sim-to-real deployment steps still require manual intervention. As broader adoption expands the artifact and experiential corpus, HARBOR's reliability and automation scope are expected to improve. Extensions toward vision-language-action models and deeper integration with real-robot hardware and perception pipelines are natural future directions.

Conclusion

HARBOR instantiates artifact-centric harness engineering for robot RL, facilitating end-to-end workflow automation, autonomous reward and algorithm tuning, parallel experience-driven efficiency, and robust sim-to-real policy transfer. Its modular architecture achieves strong reliability and efficiency, providing practical and theoretical advances in automated robot learning workflows (2606.08610).