- The paper introduces one episode driver, one resource-handle protocol, and one trace identity scheme to evaluate workloads from foundation-model decoding to whole-body control.
- DeepInsight matches reference scores while improving throughput by up to 1.29× over lm-evaluation-harness, 1.41× through asynchronous scheduling, and 3.31× with decoupled resource pools.
- The unified trace exposes cross-layer failures that isolated benchmarks miss, including boundary handoffs that caused 28.9% of failures in a 96-episode vehicle-guide study.
The problem: a spectrum no single harness spans
DeepInsight, developed by XPENG Robotics, addresses an evaluation-infrastructure gap specific to embodied humanoid systems. A Physical AI stack must be evaluated across operators that differ by more than three orders of magnitude in episode length—from a single foundation-model decoding step to thousands of physics ticks of whole-body control—while varying orthogonally along modality (text, image, audio, physics state), reward semantics (exact match, model-based judgment, trajectory-analytic termination), and resource profile (GPU-bound inference, I/O-bound sandboxes, compute-bound parallel simulation). Existing orchestrators each bake in regime-local assumptions: lm-evaluation-harness assumes short episodes and deterministic scorers; OpenCompass assumes predictable per-sample cost; HELM's abstraction is methodological rather than operational; VLMEvalKit and lmms-eval inherit the single-shot generation assumption; Inspect AI relaxes it but remains scoped to foundation-model evaluation; Isaac Lab provides parallel simulation but not model/sandbox/judge orchestration. No existing framework spans the full range.
The paper's central claim is that this heterogeneity can be carried by exactly three shared invariants—one episode driver, one resource-handle protocol, one trace identity scheme—without sacrificing benchmark fidelity, throughput, or cross-layer diagnosability. Notably, the authors do not claim unification by homogenization; the regimes remain heterogeneous behind narrow interfaces.
Architecture: three abstractions
The task abstraction exposes a two-method environment surface (reset(handle), step(handle, action)) with all transient episode state on a per-episode handle rather than on the environment. This makes environments stateless with respect to episodes, so one async runtime drives both one-step QA decodes and multi-hundred-step sandboxed rollouts through the same workers. Task declaration is split into suite configuration (what the task is) and template configuration (how to run it), merged at startup into a persisted snapshot used for replay. Judgment is an orthogonal axis: scorers are separately registered components reading the trace, so a new judge applies across the corpus by substitution. In the dominant case, a new benchmark enters as one configuration file.
The resource abstraction detaches expensive backends into two symmetric control planes—inference and sandbox—exposed to the orchestrator only via acquire(kind, episode_id, constraints) / release(handle). Stages receive independent concurrency budgets keyed to resource profile (a CPU-bound verifier and an I/O-bound code-test step occupy different queues even within the same role). The inference plane carries a heterogeneous engine fleet (vLLM, vLLM-Omni, SGLang) with client-side service discovery and a bounded-load consistent-hash policy for KV-cache affinity. The sandbox plane offers two deployment modes behind one handle: a pre-warmed container pool for stateless calls and per-episode OpenSandbox leases for stateful workloads, with lifecycle riding Kubernetes. Scaling knobs are orthogonal per bottleneck.
The result abstraction writes every event from every subsystem into one append-only structured record under a hierarchical identity tuple (run_id, suite_id, task_id, episode_id, epoch) plus a multi-rate position index (turn_id, step_id, tick) and causal lineage (event_id, parent_event_id). Cross-source analysis becomes a join; cross-layer causal diagnosis becomes a graph traversal over parent pointers. The trace—not the aggregate—is the durable artifact, and reporting runs as off-path consumers of the trace.
Evaluation where peers exist
System 2 is the only layer with mature peer frameworks, so quantitative comparison is confined there. All comparisons use identical hardware (single 8×A100 node), vLLM serving configuration, sampling profiles, and shared judge configuration (Qwen3-30B-A3B-Instruct-2507), isolating scheduler differences.
Accuracy alignment. DeepInsight reproduces reference-scale scores while covering the broadest task set. It is closest-to-reference on 5/8 text rows for Qwen3.6-27B, 4/9 for Qwen3-32B, 8.0/15 multimodal VQA rows against VLMEvalKit and lmms-eval, and 4/5 omni-modal rows against lmms-eval. Two offsets are conceded as protocol-level rather than infrastructural: LiveCodeBench depends on an unpublished reference sandbox/timeout, and Video-MME references include subtitles while all open runs use none.
Throughput. On peer-native suites, DeepInsight achieves 1.29× over lm-eval (whose synchronous batch barriers cost most on long, length-variable math workloads), 1.13× over Inspect AI (already asynchronous, but sharing one stage budget), only 1.03–1.04× over VLMEvalKit/lmms-eval (prefill-bound VQA leaves little idle time to reclaim), and 1.20× on the omni suite.
Attribution ablations. Internal ablations isolate the architectural causes. On AIME-2024 (480 episodes), replacing synchronous batch barriers with asynchronous refill cuts wall-clock from 152 to 108 minutes (1.41×) at unchanged accuracy, raising generation occupancy from 54% to 89% and LLM throughput from 1,547 to 2,178 tokens/s (+41%). On LiveCodeBench v6, collapsing stages into a single pool sized to the sandbox cap (14 workers) versus decoupled pools yields a 3.31× wall-clock difference (5h57m vs. 1h48m); the coupled condition throttles generation from its 128-way budget to 14 slots, collapsing LLM throughput from 2,385 to 731 tokens/s, and also forfeits incremental results entirely—the coupled run completes all 454 samples in one final burst after ~336 minutes. Horizontal scaling on a 27-suite workload shows near-linear speedup: 2.03× at two nodes, 4.00× at four, each within ~1.5% of linear, with failure rates below 0.1%. No peer framework supports multi-node execution, so this axis has no baseline.
Case studies beyond the peer frontier
System 1. Closed-loop simulation coverage (VLN-CE-style navigation, LIBERO-style manipulation, internal tasks) and subjective human-preference evaluation (blind pairwise rating of audio-conditioned motion generation) are recorded in the same result schema as objective metrics. The paper explicitly frames these as demonstrations of coverage and extensibility, not exact score reproduction under original benchmark environments.
System 0. Whole-body-control evaluation is framed as a release decision rather than a ranking. A controlled SR–MPJPE screen nominates WBC-RC-01 (97.90% SR, 1.915 cm MPJPE), but the behavior-level diagnostic checklist computed from registered trajectory statistics blocks it: hip-joint dynamics, contact-attitude stability, torso posture, and upper-body kinematics fail despite passing command tracking, gait symmetry, swing-foot clearance, and style scores. This judgment requires per-tick trajectory state retained and queryable on one trace—an aggregate-only pipeline cannot produce it.
Full-system composition. A vehicle-guide task composes System 2–1–0 under one trace identity over 96 episodes. End-to-end success is 60.4%, despite criterion-level marginals ranging from 70.8% (vehicle reached) to 94.8% (task intent recognized)—a clear composition gap between local completion rates and system outcome. Among the 38 non-success episodes, primary failure loci distribute as System 1 at 42.1% (16/38), boundary handoff at 28.9% (11/38), System 2 at 18.4%, and System 0 at 10.5%. The prominence of boundary failures—navigation reporting local success while violating downstream preconditions, generated motion exceeding controller feasibility—is precisely the class of defect invisible when layers are evaluated in separate harnesses, and constitutes the paper's strongest argument for unification.
Limitations and open questions
The paper is candid about scope. Quantitative depth at Systems 1 and 0 is deferred; those case studies demonstrate coverage, not competitive benchmarks. The full-system study uses one representative composed task family, with broader coverage left open. Every rollout reported is simulated, leaving the sim-to-real gap as the decisive unresolved uncertainty for hardware-bound policies. Alignment evidence has bounded exceptions (LiveCodeBench's unpublished sandbox protocol, Video-MME subtitle mismatch). The claim that "no smaller set [of abstractions] suffices" is argued structurally rather than proven exhaustively, and the cross-layer diagnosis payoff is demonstrated on a single composed task family rather than characterized statistically.
Conclusion
DeepInsight demonstrates that a heterogeneous Physical AI evaluation workload—from single-decode QA to thousand-tick whole-body control—can be carried on one runtime by three narrow invariants, with production evidence of reference fidelity, single-node speedups attributable to specific architectural mechanisms via controlled ablation, near-linear multi-node scaling, and a diagnostic capability (cross-layer failure localization on one trace) that federated per-segment harnesses structurally cannot reproduce. The principal open questions are wider embodied task-family coverage and extending the resource-handle protocol to physical robots so that sim-to-real gaps become measurable within the same trace identity.