MisActBench: Benchmark for Misaligned Actions

• MisActBench is a benchmark that systematically evaluates action-level misalignments in computer-use agents using human-annotated trajectories.
• It addresses both adversarial and intrinsic misalignment causes, enabling reproducible research on detecting deviations in digital workflows.
• The benchmark employs detailed metrics such as precision, recall, and F1 score to assess the performance of CUA alignment guardrails.

MisActBench is a benchmark for misaligned action detection in computer-use agents (CUAs), empirically characterizing and evaluating action-level alignment failures arising during the execution of digital workflows. Designed to address the lack of systematic, action-level datasets and evaluations in the CUA domain, MisActBench provides a large collection of realistic trajectories in which each action is annotated by human experts for alignment with user intent. The benchmark enables reproducible research on both adversarially induced and intrinsically arising misaligned actions, fostering the development and rigorous assessment of CUA alignment guardrails and detectors (Ning et al., 9 Feb 2026).

1. Motivation and Background

CUAs operationalize high-level user instructions into granular sequences of GUI-based actions, aiming to automate digital tasks in web and OS environments. Despite recent advances, prominent failure modes persist, including misaligned actions due to indirect prompt injections, reasoning errors, or inadequate environment grounding. Such misalignments threaten user safety, compromise data integrity, and erode trust. Prior evaluations typically focus on whole-task or trajectory-based assessments, lacking granularity in capturing fine-grained deviations at the action level. Existing guardrails are tailored to known attack specifications or policy violations, with limited generalizability.

MisActBench was constructed: (a) to enable systematic study of action-level (as opposed to trajectory-level) alignment; (b) to comprehensively span both adversarial and intrinsic causes of misalignment; (c) to provide a large, human-annotated dataset for the training and evaluation of action-level misalignment detectors and correctors.

2. Formalization and Taxonomy

Action Alignment Definition

Let $I$ denote the user instruction. The agent's interaction history up to time $t$ is $T_{<t} = (o_1, a_1, \dots, o_{t-1}, a_{t-1}, o_t)$, with $a_t$ being the action proposed at $t$. MisActBench uses a binary misalignment indicator:

$$M(I, T_{<t}, o_t, a_t) = \begin{cases} 1 & \text{if } a_t \text{ is misaligned w.r.t. } I \ 0 & \text{otherwise} \end{cases}$$

An action is aligned ($M=0$) if and only if all of the following properties are satisfied:

1. Intent Service: $a_t$ advances the original user instruction $I$ (not any injected directive).
2. Safety: $a_t$ does not produce unauthorized or harmful consequences.
3. Task Relevance: $a_t$ can reasonably be interpreted as contributing to $I$.

Violation of any condition renders $a_t$ misaligned ($M=1$). Notably, non-optimal but intent-advancing (e.g., inefficient) actions are considered aligned.

Misalignment Taxonomy

MisActBench classifies misaligned actions into three categories:

Category	Definition	Example
Malicious Instruction Following	Agent executes adversarial, injected instructions instead of the user intent	Deleting .ssh/id_rsa when prompted by a malicious overlay
Harmful Unintended Behavior	Agent induces harm via reasoning or grounding errors	Deleting the source .docx after exporting to PDF
Other Task-Irrelevant Behavior	Agent performs technically harmless but irrelevant actions	Opening media player while editing a document

This taxonomy spans intentional attacks and naturally emerging CUA errors, ensuring comprehensive coverage.

3. Benchmark Construction and Annotation

Data Composition

MisActBench encompasses 558 trajectories, totaling 2,264 human-annotated actions. Of these, 1,264 are annotated as aligned and 1,000 as misaligned, distributed as shown:

Type	Count
Malicious Instruction Following	562
Harmful Unintended Behavior	210
Other Task-Irrelevant Behavior	228

Benchmarked domains include both web and OS workflows, with data comprising multimodal inputs (screenshots and code).

Data Generation Pipeline

Adversarial Trajectories: Aggregated from attack-focused benchmarks such as OS-Harm, DoomArena, RedTeamCUA, and RiOSWorld using prominent LLM-based CUAs (e.g., GPT-4o, Claude Sonnet). Trajectories where the attack successfully induced misalignment were retained.

Synthesized Intrinsic Failures: Starting with benign OSWorld trajectories, the process injects concrete unintended subgoals, grounded via LLMs drawing from 13 risk templates (e.g., file deletion, config modification). Core actions from standalone subgoal executions are merged into benign trajectories at deterministically chosen points, and the resulting trajectory is validated both for plausibility and success of the unintended action.

Annotation Protocol

Annotation proceeds in two stages:

1. Filtering: Eliminate uninformative steps (e.g., wait actions, blank clicks).
2. Labeling: Three annotators independently classify each remaining action (aligned or one of the three misalignment categories).

The process uses a GUI presenting the current user task, summarized trajectory history, action proposals, and pre/post-action screenshots. Inter-annotator agreement is high (Fleiss’ $\kappa = 0.84$). Discrepancies are resolved by majority vote.

4. Evaluation Protocols and Metrics

Offline Detection

Misaligned action detectors are evaluated using standard classification metrics, with ground-truth $M$ and predicted $\hat{M}$:

• Precision $= \frac{TP}{TP+FP}$
• Recall $= \frac{TP}{TP+FN}$
• F1 score $$= 2 \; \frac{\mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

Online Evaluation

Assessment metrics for end-to-end CUA deployment include:

• Attack Success Rate (ASR): Fraction of tasks where the attacker's instruction is successfully executed.
• Utility under Attack (UA): Fraction of tasks where user’s original intent is achieved despite attack attempts.
• Success Rate (SR): Fraction of tasks completed as intended in benign (non-adversarial) scenarios.

These metrics allow simultaneous evaluation of security and usability impacts due to CUA guardrails.

5. Baselines and Comparative Results

Baseline Methods

• Task Shield (Jia et al., 2025): Task decomposition with pairwise action consistency checking.
• InferAct (Fang et al., 2025): Infer likely user task from trajectory and compare to provided $I$.

Performance on MisActBench

Method	Precision	Recall	F1 Score
Task Shield	61.3%	73.6%	66.9%
InferAct	56.0%	70.1%	62.3%
DEACTION	89.9%	76.8%	82.8%

The DEACTION guardrail demonstrates an absolute F1 improvement over the best baseline method by 15.9 points, with similar gains confirmed across alternative LLM backbones.

End-to-End Online Robustness

Defense	ASR	UA	SR (Benign)
No defense	32.0%	48.0%	39.0%
Task Shield	22.0%	58.0%	36.6%
InferAct	22.0%	70.0%	36.5%
DEACTION	2.0%	60.0%	39.6%

DEACTION reduces ASR by over 90% (from 32.0% to 2.0%) while maintaining or slightly improving benign-task SR (Ning et al., 9 Feb 2026).

6. Analysis and Practical Considerations

Offline Performance Gains

• F1 score gain: +15 points.
• Precision elevated from $\sim 60\%$ (baselines) to $\sim 90\%$, counteracting false positives.

Online Robustness

• ASR reduction of approximately 93.8% under red-team conditions.
• UA and SR metrics remain stable or slightly increase, indicating no excessive benign over-blocking.

Latency and Deployment Overhead

• Mean per-step execution time without guardrail: ~21 s; with DEACTION: ~28 s ($+$25\% overhead).
• Guardrail reasoning time averages 7.2 s per step.
• Two-stage deployment: a "fast check" passes $\sim 45\%$ of steps in ~3.2 s; ambiguous steps undergo full systematic analysis (~11 s per step).
• Cost-efficient variant: leveraging a smaller LLM for the fast check reduces guardrail overhead by $\sim 65\%$, with negligible drop in performance.

7. Limitations and Future Directions

Current limitations include susceptibility to disguised injections that evade initial detection checks, degradation in action grounding on novel UI distributions, and possible errors in outcome prediction due to reliance on LLM-based world modeling. The narrowness of task relevance criteria may yield spurious detections in edge cases.

Planned advances target detection robustness (e.g., visual anomaly modules), enhanced multimodal grounding for coordinate-based actions, integration of planning-aware world models for outcome inference, and expansion to new domains (mobile, VR), interface toolkits, and finer-grained misalignment classification. Exploration of multi-agent workflows and richer failure taxonomies is also proposed (Ning et al., 9 Feb 2026).

---

MisActBench establishes the first large-scale, human-annotated, action-level benchmark for misaligned action detection in CUAs, enabling systematic study of both malicious and intrinsic alignment failures. When combined with the DEACTION guardrail, substantial improvements in both offline detection (+15 F1) and real-world robustness (>90% ASR reduction) are realized, providing a foundation for the next generation of intent-aligned, robust computer-use agents.