Inducing Task Models from Computer-Use Traces

Abstract: Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and real-world work is multi-threaded with interleaved goals. Existing methods assume a given task or a single workflow, and produce step-level summaries rather than structured task models. We introduce Task Model Induction (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI's task models improve held-out task accuracy by 30.0% over the strongest baseline.

• The paper introduces Task Model Induction, a three-stage method that grounds screenshots and events, segments activities, and reconciles independently induced objectives and procedures into auditable task trees.
• The method identifies interleaved tasks with 0.974 adjusted Rand index, achieves 74.9% step-description accuracy and 88.5% operator correctness, and substantially outperforms direct generation and workflow summaries.
• The paper shows that induced models improve held-out skill accuracy from 14.29% to 18.57% and executability from 59.35 to 67.65, while revealing sensitivity to grounding errors, repair-heavy activity boundaries, privacy redaction, and limited domain coverage.

Problem and motivation

Naturalistic computer-use traces—passively recorded screenshots paired with mouse and keyboard events—are abundant, but the work they encode is undocumented. The paper "Inducing Task Models from Computer-Use Traces" (2608.20319) addresses the recovery of structured, auditable models of that work. The authors formalize a task model as a tree in which every node pairs an objective (a recursive decomposition of the task into sub-objectives) with a control-flow operator over its children (sequence, for-each, while), drawing on hierarchical task analysis and the Böhm–Jacopini structured programming theorem. Selection is deliberately excluded because a trace records only the enacted path, not unchosen alternatives.

The induction problem is defined over a raw trace $X = \langle x_1, \ldots, x_N \rangle$ where each event carries a screenshot, an operation, and a timestamp. The system must jointly discover the latent task set $\mathcal{T}$—with no tasks, boundaries, or descriptions given—and induce a model $M_t$ per task. This distinguishes the setting from prior workflow induction, which treats a session as one continuous workflow; trace-analysis methods that presuppose a known root task; and LLM summarization, which produces prose rather than structure.

Method

Task Model Induction (TMI) proceeds in three stages, all implemented with gpt-5.4 at temperature 1.0.

Event grounding and activity segmentation. Each raw event is grounded by a vision-LLM using the screenshot pair bracketing it, yielding what was done, to which artifact, in which application. Grounding is constrained to visible evidence: the post-event screenshot clarifies what changed but does not license retroactive intent attribution. Grounded events are segmented twice: backward into semantic actions (a semantic action ends where an artifact reaches its new state), then forward into activities (an activity begins when its local objective is adopted). This directionality follows event segmentation theory.

Latent task induction. Activities are processed chronologically and assigned to the closest existing task or used to open a new one. Each task maintains a profile combining a summary with a small set of referential identifiers (artifacts, named entities). The identifiers are critical: they hold a task together across application switches and naming aliases (e.g., a repository name versus a deployment URL), where summaries alone would drift. A global consolidation pass merges tasks split by objective drift or long interruptions.

Task model construction. For each discovered task, an objective model $O_t$ and procedure model $P_t$ are induced independently under formal validity constraints enforced by a deterministic validator with structured repair feedback. $P_t$ admits for-each only when at least two aligned occurrences of the same pattern exist, differing mainly in the acted-on entity, and while only when repetition continues until an objective-state condition holds; loops must be grounded in recurring evidence rather than asserted. Reconciliation then fuses the two trees: the control-flow operator comes from $P_t$, child objectives from $O_t$, and disagreements are resolved by splitting, merging, or re-parenting until objective scope and control flow agree.

Intrinsic evaluation

Three evaluations use HumanWork (38 recorded human sessions, 42.8 hours, 48.7K events across 15 professional tasks) and SkillsBench agent trajectories.

Robustness to interleaving. Synthetic trajectories are built by cutting $K$ sessions ($K$ from 2 to 15) into $\mathcal{T}$0 segments each ($\mathcal{T}$1 from 2 to 4) and shuffling them, so every task reappears at non-contiguous points. TMI achieves 0.974 overall ARI against ground-truth groupings, degrading only mildly from 0.980 at $\mathcal{T}$2 to 0.968 at $\mathcal{T}$3, with task-count MAE below 1 throughout. Notably, 96.7% of sampled task pairs share an application and 88.9% share a domain, so performance is not driven by surface separability. Error analysis of 1,107 labels attributes residual failures mainly to subgoal promotion (6.8%), of which roughly a third involve genuinely off-task user behavior that is arguably unresolvable from the trace alone.

Task identity stability. Withheld ground-truth labels are compared against induced task sets by two independent judges. Match rates reach 94.74% on human sessions and 93.24% on successful SkillsBench runs under gpt-5.5, with claude-sonnet-5 producing comparable or higher rates (89.47% and 98.65%), indicating the result is not judge-family-specific.

Structural fidelity. Against a workflow-summary baseline and direct one-pass generation, TMI reaches 74.9% step description accuracy and 88.5% operator correctness under gpt-5.5, versus 30.3% and 52.7% for direct generation—the strongest baseline on procedures. Ablations support the central design claim: the objective-only variant drops description accuracy to 63.2%, and joint single-pass induction produces half as many nodes (8.6 versus 19.5 objective nodes per session), so its higher per-node consistency reflects coarser granularity rather than better structure. Reconciliation corrects boundaries of 64.5% of objective nodes via procedure evidence and 21.9% of procedure nodes via objective evidence. Fidelity persists over pervasive non-linear execution—error correction appears in 89% of sessions and exploratory search in 87%—though boundary placement degrades most on such spans (e.g., 59.6% under error correction versus 79.0% on clean spans), since repair supplies no crisp deliverable to anchor it. Human annotation on 20 sessions shows judges fall within annotator ranges, suggesting automatic scores do not inflate quality.

Extrinsic evaluation

On SkillLearnBench, a task model induced from a single successful demonstration per task family feeds Codex's skill creator, and generated skills are executed on held-out instances with gpt-5-mini. Skills derived from TMI's task models improve held-out accuracy from 14.29% to 18.57% over the workflow-summary baseline—a 30% relative gain—and raise executability from 59.35 to 67.65. Two observations qualify this result. First, absolute accuracy remains low across all conditions, including expert-written skills (10.00%), so skill coverage and held-out accuracy do not move together here; the curated condition is reported as a reference point, not an upper bound. Second, all skills are generated and executed by the same model family, isolating the source of learning but leaving cross-model transfer untested.

Limitations

The authors note that naturalistic traces may contain personally identifiable information, and that privacy redaction before induction—and its effect on induction quality—is left open. Additional constraints emerge from the evaluation itself: error propagation analysis shows grounding errors affect about 9% of objective nodes and segmentation errors about 6%, persisting because later stages do not re-segment; misplaced reconciliation boundaries remain the largest residual error source (~12% of nodes); and mis-scoped parents propagate failures vertically to 83% of their children. Boundary grounding on repair-heavy spans is the weakest fidelity dimension. Whether induction quality survives redaction, and how the method scales beyond the 15-task domain coverage evaluated here, remain open questions.

Conclusion

TMI demonstrates that latent tasks and their execution structure can be recovered jointly from unconstrained computer-use traces, pairing hierarchical objective decomposition with observable control flow under formal validity constraints. Its strong intrinsic results—0.974 ARI under dense interleaving and 74.9% step fidelity against 30.3% for the best baseline—together with a 30% relative downstream skill gain, indicate that independently induced and reconciled objective/procedure views capture complementary evidence that neither single-pass nor summary-based approaches recover.