State Beyond Appearance: Diagnosing and Improving State Consistency in Dial-Based Measurement Reading

Abstract: Multimodal LLMs (MLLMs) have achieved impressive progress on general multimodal tasks, yet they remain brittle on dial-based measurement reading. In this paper, we study this problem through controlled benchmarks and feature-space probing, and show that current MLLMs not only achieve unsatisfactory accuracy on dial-based readout, but also suffer sharp performance drops under viewpoint and illumination changes even when the underlying dial state remains fixed. Our probing analysis further reveals that same-state samples under appearance variation are not consistently clustered, while neighboring states fail to preserve the local structure implied by continuous dial values. These findings suggest that existing MLLMs largely ignore the intrinsic state geometry of dial measurement tasks and instead rely on superficial appearance cues. Motivated by this diagnosis, we propose TriSCA, a tri-level state-consistent alignment framework for dial-based measurement reading. Specifically, TriSCA consists of state-distance-aware representation alignment, metadata-grounded observation-to-state supervision, and state-aware objective alignment. Extensive ablation studies and evaluation experiments on controlled clock and gauge benchmarks, together with evaluation on an external real-world benchmark, demonstrate the effectiveness of our method.

• The paper introduces TriSCA, a tri-level alignment framework designed to enforce state consistency in dial-based measurement reading for MLLMs.
• It leverages a controlled synthetic pipeline and contrastive triplet loss alongside metadata grounding and RL objectives to decouple appearance from dial state.
• Experimental results demonstrate significant robustness improvements and finer discrimination of neighboring dial states under challenging visual perturbations.

State Consistency in Dial-Based Measurement Reading with TriSCA

Problem Diagnosis: Appearance Fragility of MLLMs on Dial Readout

Despite progress in MLLMs across vision-language tasks, current architectures exhibit fundamental weaknesses on dial-based measurement reading. Controlled benchmark construction and probing experiments reveal that state-of-the-art MLLMs display major fragility when the visual appearance varies due to viewpoint or illumination changes, even with constant underlying dial state. As shown by sharp accuracy degradation (Figure 1), these models fail to maintain robustness under non-canonical imaging conditions.

Figure 1: Performance of MLLMs drops abruptly under appearance shifts, indicating high susceptibility to viewpoint and illumination changes even when dial state remains fixed.

Further, feature-space analyses indicate that same-state samples scatter across the representation manifold when appearance varies, while neighboring dial states often cluster more tightly than warranted (Figure 2). This failure to enforce both cross-condition invariance and local sensitivity severely limits precise measurement reading.

Figure 2: Dial-based reading requires features that are invariant to appearance-only perturbations while retaining sensitivity to fine-grained state changes—a property not achieved by current MLLMs.

Controlled Benchmarking: Appearance-State Decoupling by Synthesis

To systematically diagnose state-level consistency, the work introduces a controlled synthetic pipeline to generate clocks and gauges with explicit disentanglement of dial state and appearance (viewpoint, lighting, blur, etc.). This approach enables clean construction of same-state but different-appearance pairs, and nearby-state but similar-appearance pairs (Figure 3). Sample assets span the full cross-product of states and nuisance visual factors, enabling both rigorous evaluation and fine-grained training interventions.

Figure 3: The controlled synthesis pipeline provides systematic appearance and state manipulation for both clocks and gauges, producing fine-resolution neighboring-state pairs and decoupling state from nuisance appearance variables.

Representation Analysis: Empirical Failure and Feature Geometry

Empirical probing of existing visual features—based on t-SNE visualizations and retrieval metrics—demonstrates that features extracted from strong MLLMs do not respect dial state geometry. Same-state samples fail to cluster compactly under appearance variation; subtle local changes in pointer position (i.e., neighboring dial states) are not accurately separated (Figure 4).

Figure 4: Pre-alignment feature representations lack compactness for same-state samples and do not enforce fine-grained separation for neighboring dial states; state-aware alignment improves both properties.

TriSCA: A Tri-Level State-Consistent Alignment Framework

To address appearance-induced representational failures, the work proposes TriSCA, a joint training procedure that operates across representation, reasoning, and objective levels:

• State-Distance-Aware Representation Alignment: Using a contrastive triplet loss, representation learning is explicitly supervised such that same-state samples from different appearances are drawn together while neighboring states are separated proportionally to their physical state difference.
• Metadata-Grounded Observation-to-State Reasoning: Training is supervised with templates that decompose dial readout into explicit indicator localization and scale mapping steps, grounding predictions in geometric evidence rather than global appearance.
• State-Aware Objective Alignment: Instead of binary correctness, reward signals are defined as continuous functions of state error (e.g., minute difference for clocks), optimized via Group Relative Policy Optimization (GRPO) for lightweight and stable policy updates.

With this stage-wise alignment, the TriSCA pipeline reshapes feature space, encourages locally coherent reasoning, and provides reward signals consistent with the geometry of the measurement task.

Experimental Results: Robustness and Transfer

On the controlled synthetic benchmark, TriSCA exhibits strong and consistent accuracy improvement across both clocks and gauges and across all levels of appearance perturbation. The performance gap is most pronounced in the hard setting with simultaneous viewpoint and illumination shifts, demonstrating substantial mitigation of performance collapse (Figure 5).

Figure 5: TriSCA slows the rate of accuracy degradation as appearance perturbations become more severe, outperforming baseline MLLMs particularly at higher difficulty levels.

Case studies validate that TriSCA not only reduces aggregate error, but also robustly stabilizes predictions across appearance changes and improves discrimination of nearby dial states (Figure 6).

Figure 6: Example cases illustrate improved state prediction consistency and higher local precision following TriSCA alignment.

Complementary evaluation on the MeasureBench external benchmark confirms that TriSCA brings substantial gains for dial readout (Dial category), with minimal impact on unrelated classes, demonstrating good transfer and specificity.

Ablation and Representation Analysis

Ablation analysis underscores the necessity of all three TriSCA stages: representational alignment forms the basis for larger gains, grounding in metadata-based supervision reduces shortcut reliance, and the state-aware RL objective maximizes benefit only when strong representation and reasoning have been established. Quantitative retrieval metrics (Recall@1, Silhouette score) corroborate that TriSCA induces a more state-consistent and discriminative representation space.

Implications and Future Directions

The findings expose a previously underexplored bottleneck of state consistency in visual measurement reading for MLLMs—a task regime where geometric state estimation, not semantic recognition, is essential. TriSCA demonstrates that explicit state-level alignment across feature, supervision, and objective spaces can substantially improve robustness to appearance nuisance factors and fine-grained state discrimination.

Practically, this holds promise for safety-critical applications requiring robust and precise meter reading under uncontrolled field conditions, such as robotic inspection and industrial monitoring. Theoretically, it expands multimodal learning beyond text-centric template tuning and advocates for geometry-aware, structured learning in continuous-state tasks.

Future work should extend state-consistent alignment strategies to broader instrument categories (linear scales, composite displays), incorporate more varied real-world degradations, and generalize to wider classes of physically-grounded structured estimation within MLLMs. Mechanistically, there remains scope for developing end-to-end training objectives that unify all three alignment levels.

Conclusion

This study introduces and addresses the critical shortcoming of representational state inconsistency in dial-based measurement reading for MLLMs. Through controlled synthesis, in-depth feature probing, and the tri-level TriSCA alignment framework, dial state estimation becomes more robust, locally precise, and generalizable across appearances and domains. While major progress is established, tackling the full spectrum of complex real-world meters, and general physically-grounded estimation, defines rich directions for future alignment-centric research.