Closed-Loop Open-Ended Benchmarks

• Closed-loop open-ended benchmarks are evaluation protocols featuring evolving environments and iterative feedback loops that enable autonomous agents to adapt dynamically.
• They employ methodologies like online planning, model-predictive control, and high-fidelity simulations to mirror real-world conditions and emergent task complexities.
• These benchmarks advance research by measuring robustness and long-horizon performance using composite metrics such as Fluidity Index, driving scores, and recovery rates.

Closed-loop open-ended real-world benchmarks are a class of evaluation protocols, datasets, and simulation environments designed to assess the adaptability, robustness, and generalization capabilities of autonomous agents, machine learning models, and decision systems operating in dynamic and unconstrained real-world settings. These benchmarks move beyond static, fixed-goal, or open-loop tests by introducing feedback mechanisms between the agent and the environment, allowing actions to dynamically influence the future state of the environment and in turn the subsequent decisions of the agent. They are increasingly recognized as the gold standard for measuring performance in real-world autonomy, intelligence, and physically grounded reasoning.

1. Core Principles and Definitions

Closed-loop benchmarking refers to protocols in which the model or agent’s outputs directly influence the evolution of the environment during evaluation. In open-ended scenarios, there is no pre-specified set of goals or strictly bounded outcome space; tasks may be task-free (e.g., open-ended exploration) or encompass a wide spectrum of possible sub-tasks or failure conditions. Together, a closed-loop, open-ended real-world benchmark ("CLOE benchmark"—Editor's term) is defined by the following properties:

• Real-time or iterative feedback between agent and environment.
• Unconstrained or variable task goals, enabling emergent complexity and adaptation.
• Environment state that evolves in ways not fully known or controllable by the model a priori.
• Evaluation protocols and metrics that measure not just static performance, but adaptability, context switching, and long-horizon stability.

Contrast with open-loop and/or closed-ended benchmarks, which assess model predictions against fixed trajectories or answers without allowing the model’s predictions to influence the unfolding scenario (Caesar et al., 2021, Bouzidi et al., 8 May 2025, Ngoiya et al., 23 Oct 2025).

2. Technical Methodologies and Architectural Elements

Implementing closed-loop open-ended benchmarks requires:

• Iterative decision and feedback loops: At each time step, the agent receives observations, chooses actions, and these actions are applied to the environment, which then responds with new observations reflecting both the agent’s action and any exogenous changes (Agboh et al., 2021, Yan et al., 18 Nov 2024, Zhang et al., 4 Aug 2025).
• Online planning strategies: Candidate action sequences are proposed, simulated using internal or external models of environment dynamics (e.g., generative world models or simulators), and scored for downstream utility (e.g., task success, reward). This often employs algorithms such as model-predictive control, trajectory sampling, or revision policies operating over simulated rollouts (Zhang et al., 20 Oct 2025).
• Environment and scenario diversity: Real-world fidelity is achieved using either (i) digital twins of real physical environments created from sensor data and mapping (Yu et al., 28 Sep 2025), (ii) high-fidelity generative scene synthesis (Yan et al., 18 Nov 2024, You et al., 11 Dec 2024), or (iii) 3D physics environments supporting procedurally generated or user-specified scenes (Gan et al., 2021).
• Multi-level abstraction in action APIs: Environments provide standardized APIs mapping high-level policies to low-level actuators; Bench2ADVLM, for example, composes high-level vision-LLM outputs into mid-level and then hardware-level controls (Zhang et al., 4 Aug 2025).
• Support for both simulated and real-world ("hardware-in-the-loop") testing: Physical abstraction layers translate closed-loop control from simulation to deployed robots or vehicles (Zhang et al., 4 Aug 2025, Zheng et al., 12 Dec 2024).

3. Metrics of Adaptability, Robustness, and Long-Horizon Performance

Unlike static benchmarks measured with accuracy or L2 error, closed-loop open-ended real-world benchmarks employ composite metrics tuned to system-level properties:

• Fluidity Index (FI): Aggregates "accuracy adaptation" (AA) across context switches, measuring how well the model’s prediction changes align with real environmental shifts:

$FI(t) = \frac{\sum_{i=1}^n AA_i}{NC}$

where $$AA_i = 1 - \frac{|New\,Prediction_i - Old\,Prediction_i|}{Change\,in\,Initial\,Env\,State_i}$$ and $NC$ is the total number of environment changes (Ngoiya et al., 23 Oct 2025).

• Task success rate, longitudinal robustness: Evaluates whether the agent can achieve goals under varied scenarios and as the environment evolves (Zhang et al., 20 Oct 2025, Yan et al., 18 Nov 2024).
• Driving Score (DS): In autonomous driving, combines route completion with penalty terms for rule violations:

$$DS = \frac{1}{n_{total}} \sum_{i=1}^{n_{total}} Route\!-\!Completion_i \times \prod_j p_{i,j}$$

(Jia et al., 6 Jun 2024, Yu et al., 28 Sep 2025)

• Temporal consistency of predictions: Quantifies stability and smoothness of predictions supplied to downstream planners (Bouzidi et al., 8 May 2025).
• Sample efficiency and data cost: Captures how many interactions (or real-world resources) are needed to achieve competent performance, crucial for comparing reinforcement and imitation learning strategies (Uchendu et al., 4 Mar 2025, Gan et al., 2021).
• Recovery rate and failure handling: In manipulation and robotics, denotes the system’s ability to detect sub-goal failures and successfully re-plan (Zhi et al., 16 Apr 2024).

4. Representative Benchmarks and Key Domains

Modern CLOE benchmarks are now deployed in a wide range of domains:

Benchmark/Framework	Domain/Task	Key Features
NuPlan (Caesar et al., 2021)	Autonomous Driving	Closed-loop, real-world data, scenario metrics
Bench2Drive (Jia et al., 6 Jun 2024)	End-to-End Driving	Multi-ability, interactive skills, closed-loop
Gym4ReaL (Salaorni et al., 30 Jun 2025)	RL (Various, Realistic)	Multi-objective, partial obs., open-ended tasks
World-in-World (Zhang et al., 20 Oct 2025)	Embodied AI, Sim2Real	Unified online planning, action API, data scaling laws
DOE-1 (Zheng et al., 12 Dec 2024)	Driving (Multi-modal)	Unified closed-loop, autoregressive transformer
OPEn (Gan et al., 2021)	Physics/Generalization	Task-agnostic, active exploration
A2Perf (Uchendu et al., 4 Mar 2025)	Agents (Chip Design, Web Nav, Robotics)	Generalization, reliability, energy, data cost
ModelingBench (Qian et al., 21 May 2025)	Math Modeling (LLMs)	Open-ended problems, multi-agent workflows

This table highlights that CLOE benchmarks are being established across autonomous vehicles, robotics, RL control, language+vision, mathematical modeling, and open-world reasoning.

5. Advances in Adaptability and Second-Order Intelligence

CLOE benchmarks catalyze advances in agent adaptability by exposing the model to a continuum of context switches and requiring successive re-adaptation. Central notions include:

• First-order adaptability: Immediate adjustment to new environmental states.
• Second-order adaptability (digital replenishment): Sustained performance via self-replenishing computational and representational "current," i.e., models that not only respond to instantaneous change, but also learn to manage resources/tokens to maintain long-horizon responsiveness (Ngoiya et al., 23 Oct 2025).
• Emergent resilience and failure recovery: This architecture is exemplified in closed-loop manipulation (hybrid open/closed-loop execution with explicit robustness segmentation) (Agboh et al., 2021), as well as high-level planning in large world models (Zheng et al., 12 Dec 2024) and multi-agent collaborative LLM-based workflows (Qian et al., 21 May 2025).
• Provenance and interpretability: Integration of knowledge graphs and scenario generation modules to enable fact-based reasoning and identification of failure modes in real time (Sui et al., 10 Oct 2024, Zhang et al., 4 Aug 2025).

6. Empirical Insights and Future Directions

Empirical results across many CLOE benchmarks demonstrate that:

• Improvements in open-loop accuracy or visual fidelity do not necessarily yield gains in closed-loop task success; controllability and temporal consistency are often more crucial (Bouzidi et al., 8 May 2025, Zhang et al., 20 Oct 2025).
• Downsized models can deliver comparable or even superior closed-loop performance compared to large models, making them attractive for resource-constrained deployments (Bouzidi et al., 8 May 2025).
• Closed-loop evaluation surfaces nuanced performance differences in generalization, robustness, and adaptation to rare or adversarial conditions, which may be entirely missed in open-loop (log replay or static) benchmarks (Jia et al., 6 Jun 2024, Yu et al., 28 Sep 2025).
• Transferability and sample efficiency remain open research challenges, especially in open-ended, task-agnostic learning settings (Gan et al., 2021).

Ongoing and future directions aim to:

• Incorporate more realistic agent interactions (e.g., reactive behaviors for non-ego agents, generative scene augmentation) (Yu et al., 28 Sep 2025, You et al., 11 Dec 2024).
• Develop and adopt scaling laws for data and inference allocation to better inform model deployment (Zhang et al., 20 Oct 2025).
• Extend benchmarks to quantify even higher orders of adaptability, supporting evaluation of systems with emergent super-intelligence and proactive, resource-aware learning behaviors (Ngoiya et al., 23 Oct 2025).

7. Impact and Standardization in the Research Community

Closed-loop open-ended real-world benchmarks have become essential in comparative evaluation for embodied AI, RL, robotics, and interdisciplinary modeling domains. Their impact is reflected in:

• Increased focus on benchmarks that integrate realistic sensor streams, rich environmental variation, and support for hardware-in-the-loop validation (Yan et al., 18 Nov 2024, Zhang et al., 4 Aug 2025).
• The adoption of open-source, community-extendable evaluation frameworks supporting transparent, reproducible, and apples-to-apples comparison of algorithmic advances (Uchendu et al., 4 Mar 2025, You et al., 11 Dec 2024).
• A paradigm shift toward evaluating and training models for foundational properties required for real-world deployment: long-horizon generalization, fault tolerance, explainability, and sustained adaptability in unconstrained, feedback-rich environments.

These developments collectively mark CLOE benchmarks as not only necessary for next-generation model assessment, but as formative influences on model design, deployment strategy, and the overall progress toward generalizable machine intelligence.