A Cross-Model VLM-Judge Protocol for Single-Image 3D Mesh Quality (and Why Cheap Proxies Fall Short)

Abstract: Single-image-to-3D generators are improving quickly, but there is no agreed, human-free way to tell whether one generated mesh is better than another. Practitioners commonly rely on cheap automatic proxies (render-space CLIP similarity and mesh geometry-validity statistics), yet how well these track perceived quality is unestablished. We make two contributions. First, we propose and validate a reproducible VLM-judge evaluation protocol: a fixed 24-view headless render rig, two independent vision-language judge families, and a mandatory position-bias correction that queries both presentation orders and keeps only order-consistent verdicts. The two judge families agree substantially with each other (Cohen's kappa = 0.66), well above the chance-agreement floor. Second, using this protocol as the reference, we show the cheap proxies do not substitute for it. Geometry validity is only a weak signal on average (because, as we show, it is bimodal) and stays below our pre-registered target, while render-CLIP is at chance. A learned Bradley-Terry head collapses onto a single manifoldness statistic (giving render-CLIP a negative weight) and matches geometry-only exactly, so learning the feature weights buys nothing. The proxy is also bimodal: it is significantly above chance on contrasts with visible geometric defects but at chance on ambiguous contrasts, consistent with geometry validity tracking the judge only when the defect is visually salient. We therefore recommend the VLM-judge protocol as a reliable, reproducible evaluator under the conditions tested (two feed-forward generators on Google Scanned Objects, with a face-drop degradation regime) and advise against geometry/CLIP proxies as optimization targets.

• The paper introduces a position-bias-corrected cross-VLM protocol yielding high inter-judge agreement (κ=0.66) for mesh quality evaluation.
• The paper exploits a fixed 24-view rendering setup and rigorous statistical validation to reveal the limitations of conventional cheap proxies.
• The paper demonstrates that geometry and render-CLIP proxies fall short in ambiguous cases, recommending direct VLM judgments for improved optimization.

Cross-Model Vision-LLM Judging for Single-Image 3D Mesh Quality: Analysis and Limitations of Cheap Proxies

Introduction

This paper provides a rigorous and systematic investigation of automatic evaluation protocols for single-image-to-3D (SI2M) mesh generation, focusing on the reliability of cheap automatic proxies––such as geometry validity and CLIP-based render-space similarity––in contrast with an explicit, position-bias-corrected cross-VLM judge protocol. The proposed protocol leverages two independent open VLM judge families, employing a fixed 24-view render pipeline and strict position-bias correction, to yield reproducible, human-free quality judgments. The results demonstrate substantial cross-model agreement (Cohen’s $\kappa=0.66$) but also expose the inherent limitations and misleading nature of common low-cost proxies, especially on ambiguous cases where generator differences are not visually obvious.

Methodology

The evaluation framework comprises the following core components:

• Fixed-View Rendering Rig: Each mesh is rendered from 24 views with a headless rasterizer after normalization.
• Cross-VLM Judge Protocol: Quality is measured as a pairwise preference between alternative SI2M outputs, judged independently by two vision-LLMs: Qwen2.5-VL-7B-Instruct (oracle judge $X$) and InternVL3-8B (validation judge $Y$). Verdicts are taken as reliable only if consistent under both $A,B$ and $B,A$ ordering, effectively correcting for position bias.
• Cheap Automatic Proxies: Geometry validity aggregates five mesh statistics (watertightness, manifoldness, non-self-intersection, normal consistency, and render-CLIP similarity). These proxies are evaluated directly, combined in a fixed-weight fashion, or fit via a learned Bradley–Terry pairwise head.
• Statistical Validation: Agreement metrics are carefully calibrated using Wilson 95% confidence intervals and cluster bootstrapping at the object level.

Empirical Results

Cross-VLM Judge Reliability

Dual VLM judge agreement reached 0.83 on all dual-labeled pairs, yielding $\kappa=0.66$ after correcting for marginal baselines. This substantial agreement persists across all pair subgroups and is robust to various candidate contrasts, demonstrating effective mitigation of position bias—26% of uncorrected verdicts flipped with presentation order.

Failure of Cheap Proxies

Geometry validity proxies achieved 0.62 agreement (cluster-bootstrap $[0.55,0.69]$), significantly above chance but well below the target threshold ($\approx{\ge}0.75$); render-CLIP similarity was at 0.48, indistinguishable from chance. The Bradley–Terry learned reward collapsed to a single geometry statistic (manifoldness), yielding no measurable benefit from proxy feature combination.

Crucially, these proxies only exhibited predictive power when geometric defects were visually salient (e.g., clear open/holed meshes), aligning with typical cherry-picked demo scenarios.

Figure 1: Cheap proxies collapse on ambiguous subgroups while the VLM judge remains reliable; geometry-proxy accuracy (blue) is high for visible defects but drops to chance for mixed/ambiguous cases, while render-CLIP is consistently weak and judge agreement (green) persists.

Subgroup and Calibration Analysis

Geometry proxies performed well only on contrastive cases with visually obvious defects—cross-generator ($0.91$) and within-TripoSR degradation ($0.80$)—but were at chance (0.53) on cross-generator-mixed pairs where defects are subtle or absent. The deleterious implications are clear: proxies risk dramatically overestimating quality in regimes of subtlety, which are essential for fine-grained ranking or optimization.

Figure 2: The geometry proxy is weakly but correctly calibrated, with increased accuracy as the reward gap widens between candidates; dashed line is chance.

No meaningful signal was recoverable in ambiguous, subtle-quality comparisons. This bimodal error profile highlights the risk of relying on proxies during downstream generator alignment or preference optimization.

Implications and Limitations

The paper establishes the following critical implications:

• Reproducible, Human-Free Reference Standard: The VLM-judge protocol provides a scalable, consistent evaluator for SI2M mesh quality under the tested regime, facilitating system-level ranking and reward modeling—without requiring human annotation.
• Proxy Pitfalls for Automatic Reward Modeling: Geometry and render-CLIP metrics, while convenient, do not generalize beyond easily visible defects and risk misleading optimization pipelines. Specifically, automatic generator improvement via these proxies is inherently limited and potentially counterproductive.
• Practical Recommendation: For model alignment, direct use of de-biased VLM-judge reference is advisable over faulty proxies, especially in reinforcement or preference-based optimization frameworks.

However, the study is bounded by important caveats:

• VLM Judges as Oracle: The ground-truth reference is defined by current VLMs, subject to their model-specific weaknesses and imperfect alignment with true human quality perception.
• Narrow Regime: Results are specific to two feed-forward SI2M generators, a face-drop degradation regime, and Google Scanned Objects source images. Generalization to diverse generators, object classes, or richer corruption modes remains untested.
• Human Validation: No direct human spot-checking was performed; human-VLM divergence is an open issue, as is VLM reliability on more challenging or naturalistic data.

Future Directions

The findings motivate several directions for advancement:

• Deployment of richer, learned visual features or advanced VLM architectures as reference rewards.
• Expansion to broader object classes, complex corruption/failure modes (e.g., thin structure, topology errors), and domain transfer testing.
• Systematic human evaluation to calibrate VLM judge validity.
• Direct use of VLM-judge preferences in optimal generator alignment and reward model improvement, bypassing misleading proxies.

Conclusion

This study presents strong empirical evidence that, for SI2M mesh quality, position-bias-corrected cross-VLM judgment is a reliable and reproducible evaluator, with substantial protocol clarity and statistical rigor. Common cheap proxies—geometry validity and render-CLIP similarity—are weak and misleading, performing at chance on ambiguous comparisons and thus unsuitable for use as reference rewards in generator optimization. For automatic mesh quality assessment and model specialization, direct VLM-judge supervision forms a more reliable reference. Future work should explore expanded evaluation regimes, improved VLM and visual reward modeling, and bridge VLM-human agreement gaps.