Latent State Estimation Helps UI Agents to Reason (2405.11120v1)

Abstract: A common problem for agents operating in real-world environments is that the response of an environment to their actions may be non-deterministic and observed through noise. This renders environmental state and progress towards completing a task latent. Despite recent impressive demonstrations of LLM's reasoning abilities on various benchmarks, whether LLMs can build estimates of latent state and leverage them for reasoning has not been explicitly studied. We investigate this problem in the real-world domain of autonomous UI agents. We establish that appropriately prompting LLMs in a zero-shot manner can be formally understood as forming point estimates of latent state in a textual space. In the context of autonomous UI agents we then show that LLMs used in this manner are more than $76\%$ accurate at inferring various aspects of latent state, such as performed (vs. commanded) actions and task progression. Using both public and internal benchmarks and three reasoning methods (zero-shot, CoT-SC & ReAct), we show that LLM-powered agents that explicitly estimate and reason about latent state are able to successfully complete up to 1.6x more tasks than those that do not.

• The paper shows that zero-shot LLM inference accurately estimates latent states with 77%-97% accuracy, often surpassing expert performance.
• The method improved UI agent task success rates from 28% to nearly 46% by leveraging latent state estimates for better action decisions.
• The approach enables agents to better determine stopping criteria and adjust to real-time UI errors, paving the way for broader real-world applications.

The Role of Latent State in Autonomous UI Agents

Introduction

When we think of AI agents interacting with a user interface (UI)—like those that help automate tasks on websites or applications—a variety of challenges arise. One key challenge is dealing with latent state. Simply put, latent state represents details about the environment or task progress that aren't directly observable; they need to be inferred from noisy, sometimes incomplete, data.

Although LLMs have shown impressive abilities in reasoning and various benchmarks, exploring their capability to estimate latent state and use it for reasoning in real-world scenarios hasn't been explicitly studied. This paper dives into this issue specifically within the domain of autonomous UI agents. Let's break down their approach and findings.

Why Latent State Matters

Autonomous UI agents interact with interfaces, performing actions like clicking, typing, and scrolling, to achieve goals expressed in natural language. However, UIs present a noisy and partial representation of the current state:

• Noisy Descriptions: UI screen descriptions can be incomplete or include unnecessary elements that clutter the representation.
• Action Uncertainty: Actions commanded by the agent might not always align with actions performed due to errors (e.g., clicking on the wrong button).

Given these challenges, important aspects like the current UI screen, action outcomes, and task progress become latent.

Estimating Latent State with LLMs

The researchers proposed a method where LLMs estimate various aspects of latent state by processing textual prompts in a zero-shot manner. This means the model uses pre-existing knowledge without additional training data specific to the tasks at hand. Specifically, they attempted to estimate five key aspects for UI agents:

1. Previous Actions
2. Screen Summaries
3. Task Progression
4. Previous Mistakes
5. Task Completion

Their approach involved creating structured prompts for the LLM to infer these latent aspects step-by-step.

Evaluating the Technique

To evaluate their approach, the researchers tested the LLM-powered agents on three benchmarks: PixelHelp, Android In The Wild, and Android-50, which collectively covered 135 tasks from 48 different applications. The evaluation was performed online, meaning the agents interacted with live environments rather than pre-recorded data.

Key Findings

1. Accuracy of Latent State Estimation: The LLMs were notably accurate in estimating the various aspects of latent state—with accuracies ranging from about 77% to 97%. In some cases, they even outperformed human experts.
2. Improved Task Performance: Incorporating latent state estimates substantially improved the agents' ability to complete tasks. For example, the task success rate of agents using zero-shot reasoning increased from 28% to nearly 46% when latent state estimates were included.
3. Handling Stopping Criteria: Agents that used latent state estimates were better at knowing when to stop working on a task, reducing the percentage of tasks where they stopped prematurely by impressive margins.

Implications and Future Directions

These findings have several interesting implications:

• Broader Application: While the paper focused on UI agents, the methodology could be adapted for other environments where latent state matters, such as traffic management systems or robotic process automation.
• Refinement and Extension: Future research could explore improving grounding performance (ensuring actions are performed as intended) and experimenting with different reasoning methods, possibly leading to even better task success rates.
• Language-Based Models: The paper suggests that LLMs understanding language can provide substantial value in estimating latent states, a task typically approached through traditional statistical methods or specialized training data.

Conclusion

The paper gives us an enlightening glimpse into how LLMs can engage with complex real-world environments by estimating and reasoning about latent states. While there's still room for improvement, the results are promising for the development of more robust and capable AI agents in the future.

Overall, this research offers a pragmatic approach to tackling some of the fundamental challenges faced by autonomous AI agents, highlighting the potential of LLMs in real-world applications.