Modeling Distinct Human Interaction in Web Agents

Abstract: Despite rapid progress in autonomous web agents, human involvement remains essential for shaping preferences and correcting agent behavior as tasks unfold. However, current agentic systems lack a principled understanding of when and why humans intervene, often proceeding autonomously past critical decision points or requesting unnecessary confirmation. In this work, we introduce the task of modeling human intervention to support collaborative web task execution. We collect CowCorpus, a dataset of 400 real-user web navigation trajectories containing over 4,200 interleaved human and agent actions. We identify four distinct patterns of user interaction with agents -- hands-off supervision, hands-on oversight, collaborative task-solving, and full user takeover. Leveraging these insights, we train LMs to anticipate when users are likely to intervene based on their interaction styles, yielding a 61.4-63.4% improvement in intervention prediction accuracy over base LMs. Finally, we deploy these intervention-aware models in live web navigation agents and evaluate them in a user study, finding a 26.5% increase in user-rated agent usefulness. Together, our results show structured modeling of human intervention leads to more adaptive, collaborative agents.

• The paper introduces a novel framework casting human intervention in web agents as a sequence prediction task, improving intervention accuracy by over 61%.
• It leverages multimodal context, including screenshots and accessibility trees, to accurately model intervention triggers and style-conditioned responses.
• Deployment in a real-world plugin shows a 26.5% increase in user-rated usefulness, demonstrating practical benefits in reducing unnecessary interruptions.

Modeling Distinct Human Interaction in Web Agents: An Expert Review

Motivation and Problem Definition

This paper systematically addresses the limitations of current web agents in collaborative settings, where human intervention is indispensable for error correction, preference refinement, and control transfer during web task execution. Existing agentic systems typically lack principled models for anticipating when and why users intervene, resulting in either over-autonomy (proceeding past critical user preferences or errors) or excessive confirmations. This work formulates the modeling of human intervention as a stepwise sequence prediction problem, grounded in multimodal web context and agent action proposals. The core research question is whether anticipating human intervention based on interaction style can yield more adaptive and effective collaborative agents.

Figure 1: CowCorpus, a dataset of 400 human-agent collaborative web trajectories capturing intervention events and motivations for modeling intervention-aware agents.

CowCorpus: Dataset Construction and Behavioral Analysis

The authors introduce CowCorpus, comprising 400 real-user web navigation trajectories (totaling over 4,200 action steps). The dataset covers both benchmarked tasks (derived from Mind2Web) and unconstrained free-form user tasks. For each trajectory, CowCorpus logs step-level annotations distinguishing agent and human actions, including intervention events—pauses, resumes, and overrides—that precisely mark user-initiated control transfer.

CowCorpus enables a fine-grained analysis of intervention motivations, which are categorized as:

• Error Correction and Recovery: Addressing agent mistakes or premature actions, breaking repetitive agent loops, and restarting flawed execution.
• Preference Misalignment: Intervening when agent actions diverge from implicit or underspecified user preferences.
• Assistive Takeover: Compensating for agent failures in handling complex UI elements or missing resources, or proactively avoiding unrecoverable errors.

Temporal features are defined to characterize intervention frequency, intensity, normalized position within trajectories, and explicit handback rates (fraction of interventions after which agent regains control). Using $k=4$ clustering on these features, the authors reveal four distinct user-agent interaction patterns:

Figure 2: Four distinct interaction clusters—Takeover, Hands-on, Hands-off, and Collaborative—with differing temporal dynamics, control transfer, and intervention frequency.

Modeling Human Intervention: Sequence Prediction and Style Conditioning

Human intervention is cast as a stepwise binary classification, with a multimodal prompt encoding past trajectory, current observation (screenshot, accessibility tree), and agent action proposal $\hat{a}_t$. The predictive model estimates $p(y_t = 1 \mid o_t, \hat{a}_t, \tau_{t-1})$.

Modeling approaches include:

• General Intervention-Aware Models: Fine-tuned large multimodal models (LMMs), transcending fully autonomous execution.
• Style-Conditioned Models: Models specialized for the distinct user clusters (excluding Hands-off, where intervention is absent).

Evaluation utilizes step accuracy, F1 scores, and the Perfect Timing Score (PTS), which penalizes mistimed intervention predictions relative to ground-truth intervention steps. PTS reliably reflects the temporal accuracy of predictions under varying penalty strengths.

Empirical results show that fine-tuned open-weight models (Gemma 27B SFT, LLaVA-8B SFT) achieve state-of-the-art PTS (0.303, 0.201), outperforming closed-source generalist models (PTS up to 0.293). Fine-tuning with CowCorpus enables smaller models to master intervention timing nuances where proprietary LMs fail, demonstrating a 61.4–63.4% improvement in intervention prediction accuracy over base LMs.

Figure 3: Heatmap of PTS scores across style-conditioned models and clusters, with diagonal dominance indicating cluster specialization yields superior temporal prediction except where data is sparse.

Deployment and User Study: Practical Impact

The intervention-aware models are deployed within PlowPilot, a Chrome extension built atop CowPilot. This system prompts users only at moments of predicted high intervention likelihood, reducing unnecessary interruptions and oversight burden.

A follow-up user study with cluster-assigned models demonstrates a 26.5% increase in user-rated agent usefulness compared to existing collaborative agents, with positive effects spanning intervention effort, benefit of automation, preference alignment, and perceived agent control.

Figure 4: User responses show a 26.5% higher rating for intervention-aware agents relative to baseline collaborative agents.

Ablation and Robustness Insights

Ablation studies show:

• Few-shot prompting yields inconsistent gains; in some cases, introducing examples reduces temporal prediction precision.
• Explicit reasoning in prompts degrades PTS, suggesting real-world intervention decisions are intuitive, not easily captured by stepwise justification.
• Inclusion of human action history and multimodal inputs (screenshots + AXTree) are critical for accurate intervention modeling, improving step accuracy and macro F1.
• PTS robustness: Model ranking by temporal accuracy is stable across penalty strengths ($\alpha$), with fine-tuned models less sensitive to penalty variations.

Figure 5: PTS ranking consistency across temporal penalty values, confirming reliability as a comparative metric.

Implications and Future Directions

The results show that deliberate modeling of human-agent interaction style and intervention timing significantly boosts agent adaptivity and user satisfaction. This research advocates for a shift away from exclusively optimizing agent autonomy. Instead, collaborative agents must dynamically adapt their interaction protocols—neither override nor over-confirm—with respect to temporally structured human behavioral signals.

Practically, future collaborative web agents should:

• Deploy style-adaptive intervention prediction, incorporating user trajectory histories for personalization.
• Integrate uncertainty estimation to identify decision points where user input is most valuable, reducing unnecessary oversight.
• Leverage multimodal context for robust, fine-grained intervention modeling.

Theoretically, this work highlights that human-agent interaction patterns are not only actionable but should be formalized within agent policy. As web agents are increasingly deployed in real-world settings, principled behavioral modeling is critical for fostering trust, safety, and effective partnership.

Conclusion

This paper demonstrates that distinct human interaction styles and intervention patterns in web agents can be modeled and predicted with high accuracy using stepwise multimodal sequence classifiers, substantially outperforming standard LLMs. Deployment of such models into real-world agents significantly increases user-perceived usefulness, emphasizing that principled human-in-the-loop modeling is essential for next-generation collaborative AI agents. The methodologies and dataset introduced will inform further research on adaptive agents capable of structured collaboration, personalization, and temporally-sensitive user engagement.