EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems

Abstract: Recent progress in embodied AI has produced a growing ecosystem of robot policies, foundation models, and modular runtimes. However, current evaluation remains dominated by task success metrics such as completion rate or manipulation accuracy. These metrics leave a critical gap: they do not measure whether embodied systems are governable -- whether they respect capability boundaries, enforce policies, recover safely, maintain audit trails, and respond to human oversight. We present EmbodiedGovBench, a benchmark for governance-oriented evaluation of embodied agent systems. Rather than asking only whether a robot can complete a task, EmbodiedGovBench evaluates whether the system remains controllable, policy-bounded, recoverable, auditable, and evolution-safe under realistic perturbations. The benchmark covers seven governance dimensions: unauthorized capability invocation, runtime drift robustness, recovery success, policy portability, version upgrade safety, human override responsiveness, and audit completeness. We define a benchmark structure spanning single-robot and fleet settings, with scenario templates, perturbation operators, governance metrics, and baseline evaluation protocols. We describe how the benchmark can be instantiated over embodied capability runtimes with modular interfaces and contract-aware upgrade workflows. Our analysis suggests that embodied governance should become a first-class evaluation target. EmbodiedGovBench provides the initial measurement framework for that shift.

• The paper demonstrates that task performance can be decoupled from operational governability, as shown by the trade-off between high task success and lower GovScores.
• The paper introduces seven governance dimensions and specific metrics, such as unauthorized capability invocation and recovery robustness, to stress-test AI systems.
• The paper underscores the need for explicit governance mechanisms to ensure auditability, human override responsiveness, and safe system upgrades in dynamic deployments.

EmbodiedGovBench: Governance-Oriented Benchmarking in Embodied Agent Systems

Motivation and Problem Formulation

Embodied AI systems, spanning robotic policies, foundation models, and modular runtimes, have experienced significant advancements in task capabilities. Yet, prevailing evaluation protocols focus almost exclusively on metrics of task performance—completion rates, path efficiency, manipulation accuracy—while neglecting critical properties required for operational governance: controllability, policy compliance, recovery, upgrade safety, auditability, and human override responsiveness. This gap is highly consequential as embodied deployments increasingly operate in dynamic, multi-policy, upgradable, and multi-principal environments. Traditional benchmarks inadequately expose scenarios where systems execute unauthorized capabilities, silently degrade, mishandle failure recovery, or produce incomplete operational traces—all failure modes invisible to task performance metrics.

EmbodiedGovBench introduces a governance-oriented evaluation paradigm in embodied agent systems, directly targeting whether the operational substrate remains governable under realistic capability and runtime perturbations. The benchmark prioritizes multi-dimensional governance assessment over pure skill measurement, reflecting the non-trivial distinction between competent and governable robot operation.

Benchmark Architecture and Governance Dimensions

EmbodiedGovBench is structured around seven governance dimensions, chosen for their operational salience and practical impact in real-world deployments:

1. Unauthorized Capability Invocation: Evaluation of agents' ability to avoid invoking capabilities that are present but conditionally forbidden, e.g., trust-bound actions or restricted by dynamic policy context.
2. Runtime Drift Robustness: Measurement of adherence to governance constraints under degraded states—sensor failures, resource depletion, or state divergence—focusing on detection and adaptation rather than blind continuation.
3. Recovery Success: Assessment of failure recovery mechanisms, emphasizing containment, escalation correctness, and avoidance of policy violations during error correction.
4. Policy Portability: Ensuring policy-bounded behavior persists across simulation-to-real or environment transitions, detecting latent failures in portability.
5. Version Upgrade Safety: Verifying behavior remains valid and constraints preserved after module or capability evolution, including forward/backward compatibility and rollback handling.
6. Human Override Latency: Quantifying the latency and pathway robustness for real-time human intervention, as opposed to deferred or unobservable override.
7. Audit Completeness: Evaluating the system’s ability to produce operational traces supporting error attribution, governance review, and compliance auditing.

Benchmark scenario templates inject perturbations targeting these dimensions across single-robot and fleet tracks, producing both isolated and compound governance stressors. The protocol is stress-oriented, eschewing nominal-only evaluations that systematically overestimate governability.

Metric Families and Scoring Framework

Metrics are organized into four principal families:

• Capability Governance ($G_{\text{cap}}$): Assessing enforcement of authorization boundaries, with metrics such as unauthorized invocation rates and trust-scope violations.
• Recovery Governance ($G_{\text{rec}}$): Judging the scope, appropriateness, and policy-consistency of failure recovery.
• Evolution Safety ($G_{\text{evo}}$): Measuring the robustness to module/capability upgrades, including post-upgrade policy re-validation and routing correctness.
• Operational Accountability ($G_{\text{acct}}$): Quantifying human oversight integration and audit trace completeness.

A composite GovScore aggregates these, with configurable weights reflecting deployment priorities (e.g., safety-critical weighting). Per-dimension scores are always reported to preserve diagnostic clarity and mitigate Goodhartian metric gaming.

Experimental Validation and Results

The benchmark was instantiated using 125 scenario instances in AI2-THOR, spanning five protocol families designed to probe capability legality, runtime drift, recovery, upgrade, and override dynamics. Four distinct systems were evaluated:

• A SayCan-style affordance baseline (System 0),
• A task-success-only planner (System 1),
• A governance-augmented planner (System 2),
• A Code-as-Policies code-generation baseline (System 3).

Key findings:

• Systems achieved similar task success (89.6%, 89.6%, 88.0%) yet exhibited significant divergence in GovScore (0.7227, 0.5770, 0.6357), reflecting substantial differences in practical governability.
• The governance-aware planner (System 2) led with a GovScore of 0.9437, but with a lower task success of 69.6%, exposing the governance–performance trade-off.
• Affordance-based and code-generation systems achieve partial governance compliance (System 0, $G_{\text{cap}}=0.87$) due to architectural side effects, not explicit governance mechanisms.
• Metrics such as bypass counts and degraded-state violations sharply discriminated code-generation systems, which can evade capability checks via dynamic code adaptation.
• Governance dilemma protocols ensured no system achieved a trivial perfect score due to unavoidable policy conflicts, review timeouts, or trace omissions.
• Weight-sensitivity analysis confirmed robustness: system ranking was stable under capability- or recovery-heavy profiles, confirming that discriminative power does not hinge on arbitrary weighting.

Benchmark Implications

Practical and Theoretical Impacts

EmbodiedGovBench concretizes governance evaluation as an operational requirement, not a peripheral safety check. Its adoption enables:

• Deployment-Readiness Assessment: Systems aligned with regulatory frameworks (e.g., EU AI Act [euaiact2024], ISO 13482 [iso13482]) require demonstrable governance controls; the benchmark offers direct measurement of these properties.
• Design Incentivization: By penalizing governance failures invisible to task-success metrics, the benchmark promotes architectures exposing explicit runtime structure, facilitating controlled evolution, review integration, and auditability.
• Fleet-Wide Verification: The inclusion of fleet protocols allows evaluation of delegation, trust, and audit across multi-agent deployments, directly addressing multi-principal governance and federated operation.

Methodological Advancements

EmbodiedGovBench operationalizes long-standing proposals on runtime assurance [desai2019soter], control barrier functions [ames2017cbf], and AI accountability [floridi2018ai4people, winfield2018ethical] in an executable, diagnostic-centric benchmarking framework. Its stress-oriented design distinguishes robust governance from superficial compliance and exposes emergent vulnerabilities common in code-generating and LLM-based architectures.

Limitations and Future Directions

While the presented experiments demonstrate significant discriminative capacity, key limitations include:

• Trace Discipline Dependency: Systems not exposing structured execution or audit traces cannot be exhaustively evaluated, which may bias comparisons towards modular, transparent architectures.
• Scenario Breadth: Current scenarios focus on five of seven dimensions; full validation in sim-to-real transitions and federated fleet contexts remains future work.
• External Validity: Correlation between high GovScore and real-world deployment outcomes remains to be systematically established.
• Potential for Metric Gaming: As with any benchmark, overfitting remains a risk, mitigated here by dilemma-driven episodes and multi-dimensional metric reporting.

Future work should include longitudinal deployment studies, multi-institutional replication, extension of the fleet protocols, and independent governance expert validation of scenario ground truths.

Conclusion

EmbodiedGovBench represents a paradigm shift in the evaluation of embodied AI systems. By moving beyond performance-centric assessment toward stress-tested, event-trace-based governance measurement, it fills a crucial gap in benchmarking frameworks. The experiments robustly demonstrate that task success is orthogonal to operational governability, and that meaningful distinctions emerge only through dedicated governance metrics. EmbodiedGovBench provides the methodological foundation and practical tools for the field to standardize on governability as a first-class evaluation target, accelerating the transition from competence to compliant, deployable, accountable embodied intelligence.

Reference:

"EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems" (2604.11174)