ProAct-75 Benchmark

• The paper introduces ProAct-75, a dataset of 5,383 videos and 91,581 atomic action-step annotations structured by explicit directed acyclic graphs (DAGs).
• It evaluates proactive agents through metrics like trigger detection mF1, Saved Steps, and Parallel Action Rate, highlighting improvements over state-of-the-art systems.
• ProAct-75 is designed for assistance, maintenance, and safety monitoring, enabling research into agents that can plan interventions using serial and parallel procedural structures.

ProAct-75 is a large-scale benchmark developed to support the training and evaluation of structure-aware proactive agents—systems that, in contrast to passive agents, determine when and how to intervene in real-world processes to assist, maintain, or ensure safety. The benchmark provides a comprehensive, multimodal dataset of 75 tasks, each annotated at the atomic action-step level and formalized with explicit directed acyclic graphs (DAGs) representing procedural dependencies and opportunities for concurrent (parallel-threaded) execution. This enables quantitative assessment of agents beyond imitation, focusing on their ability to reason about task structure, initiate timely interventions, and conduct parallel actions (Zhu et al., 3 Feb 2026).

1. Dataset Structure and Composition

ProAct-75 encompasses three proactive-response domains: assistance (human-initiated objectives), maintenance (environment-triggered interventions), and safety monitoring (risk-aversion actions). The dataset covers 75 unique tasks sourced from exocentric videos (Ego-Exo4D, COIN, UCF-Crime) supplemented by 495 newly collected clips to balance coverage across activities. Statistical composition includes 5,383 videos and 91,581 atomic action-step segments, each paired with an explicit task graph per task.

The data is split approximately 3:1 into training ($N_{train} = 4,074$ videos) and test ($N_{test} = 1,309$ videos). For a "best-view" evaluation (one camera view per scene), there are $N_{train} = 1,905$ and $N_{test} = 516$; remaining views constitute an out-of-distribution test set. Each action-step is annotated with a timestamp span $[t_{start}, t_{end})$, a natural language label, and a trigger flag $y_t^{trig} \in \{0,1\}$ denoting intervention salience.

Table: Dataset Statistics

Attribute	Value
Number of tasks	75
Total videos	5,383
Total step annotations	91,581
Domains	Assistance, Maintenance, Safety Monitoring
Split: Train/Test (videos)	4,074 / 1,309
Task graph per task	Yes (AND/OR DAG, multiple execution threads)

2. Task Graph Formalism

Each ProAct-75 task $T$ is formalized as a DAG $T = (V, E)$, where $V$ is a set of nodes and $E$ the set of directed edges encoding temporal dependencies. Nodes partition into executable steps $N_{test} = 1,309$0 and structural non-executable nodes $N_{test} = 1,309$1, with $N_{test} = 1,309$2. Directed edges $N_{test} = 1,309$3 enforce ordering ($N_{test} = 1,309$4 must complete before $N_{test} = 1,309$5), and reachability is recursively defined:

$N_{test} = 1,309$6

Each node $N_{test} = 1,309$7 is assigned a type $N_{test} = 1,309$8, controlling execution dependencies:

• AND-node: executes when all predecessors have executed ($N_{test} = 1,309$9)
• OR-node: executes when any predecessor has executed ($N_{train} = 1,905$0)

Here, $N_{train} = 1,905$1 is the set of executed nodes at time $N_{train} = 1,905$2. Legal next actions at time $N_{train} = 1,905$3 form the set $N_{train} = 1,905$4.

Mid-level start/end structural nodes induce execution threads. Branches between such nodes are mapped to threads via a mapping $N_{train} = 1,905$5, allowing explicit modeling of task parallelism (i.e., actions on distinct threads proceed concurrently when dependencies allow).

3. Annotation Scheme and Agent Outputs

Each atomic step $N_{train} = 1,905$6 receives:

• Frame span $N_{train} = 1,905$7
• Natural language label (e.g., "Tie the bag")
• Trigger flag $N_{train} = 1,905$8

At each agent decision window, outputs required are:

• Trigger prediction $N_{train} = 1,905$9
• Task prediction $N_{test} = 516$0
• Step prediction $N_{test} = 516$1
• Future step sequence $N_{test} = 516$2
• Proactive action $N_{test} = 516$3

This annotation schema supports fine-grained evaluation of not only "what" the agent does, but also "when" and "which thread" actions are prioritized, under procedural constraints of the DAG.

4. Evaluation Metrics

ProAct-75 employs distinct metrics for key challenges in proactive response:

• Trigger Detection: Macro-averaged F1 (mF1) and accuracy, with mF1 averaged over both classes ($N_{test} = 516$4).
• Proactive Action Selection:
  • Saved Steps (SS): For each video $N_{test} = 516$5, with $N_{test} = 516$6 total human steps and $N_{test} = 516$7 human steps remaining post-intervention, $N_{test} = 516$8 and $N_{test} = 516$9. For one-step online inference, $[t_{start}, t_{end})$0.
  • Parallel Action Rate (PA): With $[t_{start}, t_{end})$1 total robot actions and set $[t_{start}, t_{end})$2 of actions advancing a new thread ($[t_{start}, t_{end})$3), $[t_{start}, t_{end})$4.
  • Thread-mixing entropy: For candidate action $[t_{start}, t_{end})$5, the mixing ratio $[t_{start}, t_{end})$6 for each thread $[t_{start}, t_{end})$7 is $[t_{start}, t_{end})$8, with entropy $[t_{start}, t_{end})$9. The aggregate thread-mixing entropy is $y_t^{trig} \in \{0,1\}$0 where $y_t^{trig} \in \{0,1\}$1 normalizes thread activity. Proactive agents are selected to minimize $y_t^{trig} \in \{0,1\}$2.

These metrics align evaluation with the structural and temporal aspects inherent in proactive procedural assistance.

5. ProAct-Helper Framework and Methodology

The ProAct-Helper serves as a reference architecture based on a multimodal LLM (LLM, Qwen2.5-VL-Instruct, 3B/7B parameters), fine-tuned using LoRA and an instruction-tuning regime targeting three objectives:

• $y_t^{trig} \in \{0,1\}$3: Standard autoregressive cross-entropy
• $y_t^{trig} \in \{0,1\}$4: Binary classification for trigger tokens
• $y_t^{trig} \in \{0,1\}$5: Hierarchical Binding Module (HBM) loss

The total loss is $y_t^{trig} \in \{0,1\}$6. The HBM mitigates data imbalance in trigger→task→step prediction via cross-level InfoNCE contrastive binding, increasing discriminability between hierarchical outputs.

Input at each timestep $y_t^{trig} \in \{0,1\}$7 consists of a 5-frame sliding window of keyframes, processed in two prompt stages:

1. Prediction of $y_t^{trig} \in \{0,1\}$8
2. If triggered, prediction of $y_t^{trig} \in \{0,1\}$9

Entropy-driven heuristic search is then applied: candidate actions $T$0 (filtered legal next steps) are ranked by minimizing thread-mixing entropy $T$1, with a lexicographic tie-break using predicted future step positions.

A core property is explicit support for parallel thread execution—the agent may select an action advancing a distinct procedural thread, enabling concurrent progress instead of naïvely mirroring the human’s immediate next step.

6. Experimental Results

On the ProAct-75 test set, ProAct-Helper (7B) achieves superior performance compared to closed-source SoTA (Gemini-2.5-Pro):

• Trigger detection mF1: $T$2 ppt (from $T$3)
• Step detection F1: $T$4 ppt ($T$5)
• Task detection F1: $T$6 ppt ($T$7)
• Saved Steps (SS): $T$8 ($T$9)
• Parallel Action Rate (PA): $T = (V, E)$0 ppt ($T = (V, E)$1)

Ablation studies indicate that incorporating $T = (V, E)$2 improves task mF1 by $T = (V, E)$3 ppt and step mF1 by $T = (V, E)$4 ppt, while adding $T = (V, E)$5 yields $T = (V, E)$6 ppt. Full HBM produces the highest gains.

7. Significance, Applications, and Future Directions

ProAct-75 constitutes the first large-scale, step-level video benchmark pairing AND/OR DAGs (serial and parallel dependencies) with harmonized trigger$T = (V, E)$7task$T = (V, E)$8step annotation, covering diverse proactive domains. This enables rigorous research into agents that reason about intervention timing and choice, grounded in explicit procedural structure.

Applications encompass household assistants (e.g., trash-bag replacement, appliance maintenance), industrial collaboration (assembly support, tool handoff), and safety monitoring (risk mitigation, procedural oversight).

Foreseeable research avenues include:

• Integration of learned graph-feasible decoding within agent generation loops
• Reinforcement-learning or search exploiting explicit DAG structure
• Expansion to open-world tasks with dynamic graph evolution and thread variability
• Cross-domain continual and few-shot adaptation leveraging ProAct-75’s generality

By uniting multimodal perception with structured procedural graphs, the benchmark and baseline delineate a principled pathway toward the design of agents capable of understanding and co-executing complex human workflows (Zhu et al., 3 Feb 2026).