Reproducibility of UI-TARS 1.5 and similar GUI agents on AndroidWorld

Ascertain whether the reported performance of GUI interaction models such as UI-TARS 1.5 can be reproduced on the AndroidWorld benchmark under standardized evaluation conditions, and identify factors that prevent replication of the published results.

Background

In evaluating Android GUI agents, the authors aimed to use models like Qwen2.5-VL and UI-TARS. They report difficulty finding evaluation scripts compatible with AndroidWorld and an inability to replicate published performance, particularly for UI-TARS 1.5. Similar reproducibility concerns have been publicly raised by others.

Establishing reproducibility on AndroidWorld is important for fair comparison and for validating claims about agent capabilities in realistic device-control tasks. Standardized evaluation procedures would help determine whether reported metrics are achievable and under what conditions.