Incorporate object orientation into VLA evaluation metrics for the Move near task

Develop and validate evaluation metrics for Visual Language Action models that explicitly incorporate object orientation in tasks where orientation is critical—such as the Move near task—by first identifying which tasks require orientation assessment and then integrating orientation into the uncertainty and quality metrics so that final orientation correctness is accurately captured.

Background

During correlation analysis, the authors found that several metrics failed to capture whether the final orientation of manipulated objects was appropriate in the Move near task, despite orientation being critical for safety and task acceptability in that setting.

They propose leveraging simulator access to object poses and orientations to extend metrics to account for orientation-sensitive tasks, but acknowledge that defining when and how to integrate orientation remains unresolved.