Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation

Abstract: Fine-grained manipulation marks a regime where global scene context no longer suffices, and success hinges on the tight coupling of local attribute grounding, high-fidelity spatial perception, and constraint-respecting motor execution. However, current embodied AI benchmarks collapse these capacities into binary success rates, systematically inflating reported capabilities by up to 70% and masking the architectural bottlenecks that impede real-world deployment. We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes: understanding, perception, and controlled behavior. Built on a compositional task graph, MetaFine absorbs heterogeneous external benchmarks and reconstructs them into diagnostic scenarios of varying complexity under a unified protocol. Evaluating state-of-the-art vision-language-action (VLA) models through this lens exposes severe dimension-specific failures invisible to conventional metrics. Through targeted causal intervention, we identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision: improving it directly unlocks previously inaccessible manipulation capabilities without modifying downstream policies. MetaFine further supports hybrid real-sim validation, using limited paired real-world rollouts to calibrate scalable simulation-based estimates for more stable physical benchmarking. By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity. The MetaFine framework, benchmarks, and supporting resources will be publicly released at our project page: https://metafine.github.io/.

• The paper introduces the MetaFine framework, which decomposes manipulation into understanding, perception, and behavior, revealing overinflated binary success metrics.
• The paper shows that diagnostic evaluation using atomic skills and a hybrid real-sim pathway enhances the detection of granular failure modes.
• Empirical results demonstrate that architectural innovations like multi-scale cross-attention in visual encoders significantly boost manipulation precision and robustness.

Diagnostic Decomposition of Fine-Grained Manipulation: The MetaFine Framework

Introduction and Motivation

Fine-grained manipulation exposes limits of global-context-based approaches in embodied AI, where local attribute specification, high-fidelity perception, and constraint-respecting execution must be tightly coupled for success. Existing manipulation benchmarks rely primarily on coarse, binary success rates, leading to systematic inflation of capabilities—by up to 70% according to empirical analyses—and obfuscation of critical architectural bottlenecks. Such scalar metrics aggregate over failure modes attributable to deficiencies in semantic understanding, spatial perception, and low-level control, thereby providing little actionable insight for system improvement.

The MetaFine framework introduces a meta-evaluation architecture that disentangles manipulation competencies into three orthogonal axes: understanding (via semantic intervention), perception (via graded geometric and photometric perturbations), and controlled behavior (via atomic skill composition and trajectory diagnostics). This approach exposes capacity-specific failures otherwise invisible in aggregate evaluation and enables actionable diagnosis essential for progressing toward robust, physically dexterous embodied agents.

Figure 1: The MetaFine framework covers a three-dimensional diagnostic protocol via compositional task graphs, an extensible asset library, and a hybrid real-world validation substrate.

MetaFine Architecture and Protocol

MetaFine is both a protocol and an extensible ecosystem, operationalized through three primary components:

1. Compositional Task Graph: Manipulation tasks are specified as compositions of parameterized, atomic skills (e.g., Grasp Part, Rotate Along, Insert), with nodes binding perceptual, semantic, and physical constraints. This abstraction enables systematic scenario generation and granular decomposition of long-horizon tasks.
2. Asset Library: Currently comprising 431 part-segmented objects, 1,078 annotated parts, and 4,312 grasp poses, the asset library enables rapid task instantiation, supports diverse object categories, and provides fine-grained manipulation targets.
3. Three-Dimensional Diagnostic Protocol: The evaluation axis includes:
   • Understanding: Assessed via attribute-level instruction modifications and testing compositional/compound language ability.
   • Perception: Probed through parametric viewpoint and illumination perturbations at multiple severity levels; area under the success curve (AUSC) quantifies robustness.
   • Behavior: Dissected using stage-wise evaluation and trajectory smoothness, dissociating mode collapse, drift, and arrest from aggregate performance.

A critical innovation in MetaFine is support for hybrid real--sim evaluation using prediction-powered inference. 3D Gaussian Splatting-based scene transfer ensures that a small set of paired real rollouts can statistically calibrate scalable simulation estimates, yielding stable, transferable performance data. This hybrid pathway directly addresses the variance and accessibility challenges in physical benchmarking.

Figure 3: Overview of MetaFine, illustrating atomic skills, compositional task graphs, and hybrid real--sim evaluation via 3D scene transfer and PPI calibration.

Analytical Findings

Aggregate Metrics Conceal Failure Multiplicity

Binary success measures substantially overestimate the real capabilities of state-of-the-art policies. For instance, on "Grasp Part," top-line performances drop from 95% (coarse criteria) to 80% (part-level), while tasks with increasing spatial or constraint complexity see breakdowns to as low as 12% for "Rotate Along" when evaluated precisely. Under lighting or viewpoint perturbation, further dramatic drops are observed (e.g., from 80% to 15% under severe lighting in Grasp Part), evidencing a 70% inflation in apparent competency when only nominal metrics are used.

Figure 4: MetaFine exposes the inflation in aggregate success metrics and details the dimension-specific degradation under perturbations and robust diagnosis of failure profiles.

Understanding: Deficits in Semantic Grounding

Attribute-level instruction modifications expose brittle language grounding, with all assessed VLA policies achieving 0% under part-swapped instructions despite moderate headline performance in the base case. Some families (e.g., $\pi_0$, $\pi_{0.5}$) exhibit a 31–34% performance drop upon attribute substitution, but none translate language changes into correct spatial re-binding. Long-horizon, composite tasks further reveal that only models with unified language-action co-optimization retain contingency over unresolved sub-instructions—others "arrest" after the first subgoal.

Figure 5: Out-of-distribution evaluation shows loss patterns under photometric and geometric perturbation, and failed attribute-level generalization (semantic intervention).

Perception: Visual Encoder as Bottleneck

Encoder-level spatial fidelity is the limiting factor for fine-grained manipulation. For peg-in-hole, stage-wise analysis shows that visual encoders without multi-scale or cross-attention design produce diffuse activation and imprecise local correspondence, leading to sub-3% overall success rates with failure depth varying by architecture. Causal intervention replacing a single-scale encoder with a multi-scale cross-attention frontend (no change to the VLM or policy) increases grasping from 39% to 67% and alignment from 0% to 32%.

Figure 6: Encoder-focused architectural changes concentrate spatial attention and yield dramatic improvements in manipulation precision and robustness.

Photometric and geometric robustness dissociate: for instance, DP3 exhibits >84% robustness (AUSC) to lighting perturbation but inferior viewpoint invariance, suggesting architectural tuning must be axis-specific.

Behavior: Stability-Expressiveness Trade-off

Action decoding paradigm determines error propagation properties. Deterministic regression (OpenVLA-OFT) delivers spatially coherent but rigid, repetitive behavior—such policies are unable to recover from modest perceptual bias during fine alignment/insertion tasks. Stochastic approaches (flow matching, diffusion) offer correction diversity but, under ambiguous spatial features, accumulate drift, leading to trajectory instability and spatial loss over long horizons.

Figure 2: Stage-wise analysis identifies recurrent spatial drift (stochastic heads) and repetitive, non-adaptive behavior (deterministic regression), confirmed via hybrid real--sim calibrated rollouts.

Benchmark Unification and Ecosystem Role

MetaFine's adapters absorb heterogeneous benchmarks (RoboTwin, ManiSkill, LIBERO), standardizing evaluation through object/action unification and conversion of scenario structure into atomic skill graphs. This positions MetaFine as an evaluation substrate rather than another task collection, enabling diagnostic, cross-benchmark comparison and consistent competency attribution.

Figure 7: Benchmark adapters enable RoboTwin, ManiSkill, and LIBERO integration into a unified, diagnostic MetaFine representation, supporting direct comparisons and extending evaluation surface.

Implications and Future Directions

Practical Impact

• Benchmarking as Diagnosis: By exposing the origin of failure, MetaFine transforms evaluation from non-informative policy ranking into actionable system repair. Architectural interventions can directly target the axes identified as bottlenecks, facilitating rapid iteration and meaningful improvement.
• Calibrated Real-World Validation: Hybrid evaluation reconciles statistical stability with limited hardware access, producing reliable physical performance estimates while minimizing expensive real rollouts.

Theoretical Implications

• Co-Design Imperative: The empirical results emphasize that scaling isolated subsystems (e.g., LLM, visual encoder, action head) offers diminishing returns—robust fine-grained manipulation requires tight, cross-module co-design governed by constraints surfaced through diagnostic evaluation.
• Evaluation Standardization: Absorbing heterogeneous benchmarks into a compositional, competency-decomposable protocol enables the field to progress on shared baselines, eliminating ambiguities resulting from task/convention mismatch.

Forward Outlook

• Extensibility: MetaFine is designed as an open, community-driven infrastructure—atomic skills, objects, and task graphs are modular and intended for continuous expansion.
• Integration: As new manipulation paradigms emerge (e.g., tactile augmentation, multi-embodiment skills), MetaFine's protocol and adapters will support seamless evaluation, providing a stable reference for progress across the embodied AI landscape.

Conclusion

MetaFine refutes the sufficiency of binary, pass/fail benchmarks for fine-grained manipulation evaluation. By operationalizing a three-dimensional, diagnostic protocol grounded in atomic skill composition and robust to scenario scale and diversity, it reveals bottlenecks in state-of-the-art architectures, provides actionable guidance for system improvement, and standardizes the measurement of genuine robotic dexterity. Its hybrid real--sim pathway further delivers statistically robust physical validation, while its ecosystem design guarantees enduring extensibility and cross-benchmark comparability. MetaFine thereby sets a new agenda for competency-centric evaluation and design in fine-grained robotic manipulation.

Reference:

"Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation" (2605.19986)