ProAct-Helper Framework for Proactive Assistance

• The paper introduces ProAct-Helper, a novel multimodal framework that fuses LLM perception with task-graph reasoning and entropy-driven planning for proactive robotic assistance.
• It employs a Hierarchical Binding Module and AND/OR task graphs to ensure structured, parallel, and minimally interfering action selection in dynamic, human-centric environments.
• Empirical results indicate substantial improvements, including a +6.21% boost in trigger mF1 and a +15.58% increase in parallel action execution compared to closed-source baselines.

ProAct-Helper is a reference multimodal framework and structural decision baseline for structure-aware proactive assistance in complex, multi-step human environments. Developed in the context of the ProAct-75 benchmark, it fuses a multimodal LLM (MLLM) perception stack with explicit task-graph reasoning and an entropy-driven action planner, enabling grounded, parallel, and minimally interfering interventions for tasks spanning assistance, maintenance, and safety (Zhu et al., 3 Feb 2026).

1. Framework Overview and Benchmark Context

ProAct-Helper was designed to address the limitations of passive or reactive agents that simply follow instructions or imitate human action order. It provides an end-to-end pipeline for five key subtasks: (1) trigger detection, (2) task identification, (3) step localization, (4) future-action prediction, and (5) structure-aware action selection for proactive robotic assistance. Central to this approach is explicit grounding in a task dependency graph (DAG) with AND/OR semantics and explicit representation of parallel execution threads.

Within ProAct-75, a comprehensive vision-based benchmark with 75 everyday human tasks and 91,581 annotated steps, ProAct-Helper serves as both reference implementation and evaluation baseline for structure-aware proactivity, covering domains such as collaborative assistance, routine maintenance, and environmental safety monitoring.

2. Perceptual Stack: Multimodal LLM and Hierarchical Binding Module

State Detection and Perceptual Processing

ProAct-Helper processes incoming video streams using a sliding keyframe window (size 5, stride 3), with changes serialized into textual prompts for the MLLM backbone—Qwen2.5-VL (3B/7B), further adapted via LoRA. The core MLLM pipeline is two-stage:

• Stage 1: Receives keyframes and candidate tasks; outputs JSON including is_trigger (detection of actionable state) and selected task.
• Stage 2: If triggered, receives the same keyframes and the selected task; outputs current step, possible future steps (future_steps), and a vector of scores (urgency, value, priority).

Hierarchical Binding Module (HBM)

To address the long-tail hierarchy inherent in trigger–task–step labels, ProAct-Helper employs an HBM, which extracts the hidden representations for trigger, task, and step spans via mean-pooling of token embeddings. Binding is enforced by paired contrastive InfoNCE losses between each hierarchical parent–child span, supplementing the primary cross-entropy with explicit structure:

$$L(p, c) = -\frac{1}{B}\sum_{i=1}^B \log \frac{\exp(\mathrm{sim}(H_p^{(i)}, H_c^{(i)})/\tau)}{\sum_{j=1}^B \exp(\mathrm{sim}(H_p^{(i)}, H_c^{(j)})/\tau)}$$

where $H_p$ and $H_c$ denote pooled embedding vectors for the parent and child, respectively, and $\tau=0.07$. The composite loss is:

$$L = \text{CE} + \lambda_\text{trig2task} L(\text{trig} \to \text{task}) + \lambda_\text{task2step} L(\text{task} \to \text{step}) + \lambda_\text{trig\_cls} L_\text{binary}(\text{trigger})$$

with recommended weights $\lambda_\text{trig2task}=0.3$, $\lambda_\text{task2step}=0.5$, $\lambda_\text{trig\_cls}=0.5$.

HBM regularization is essential for cross-level consistency, especially in rare subgraphs, as evidenced by substantial gains in task and step mF1 after introducing each binding loss.

3. Task Graph Representation and Legal Action Set

Each ProAct-75 task is represented as a directed acyclic graph $T=(V,E)$ with AND/OR nodes encoding dependencies and parallel threads. Formally:

• Executable step nodes $V_e$ and structural nodes $V_n$
• AND/OR mapping $\phi: V \to \{\text{AND}, \text{OR}\}$ determines precondition satisfaction:
  • AND node $v$: All predecessors in $\mathrm{Prog}_t$ required for execution.
  • OR node $v$: Any predecessor in $\mathrm{Prog}_t$ suffices.

The legal action set at time $t$ is:

$$A_t^\mathrm{legal} = \{ a \in V_e \setminus \mathrm{Prog}_t \mid \text{preconds under }\phi\text{ are met} \}$$

Parallel thread partitions are derived algorithmically such that each mid-level branch is treated as a distinct thread for the purpose of candidate selection and entropy computation.

4. Entropy-Driven Heuristic Search for Action Selection

ProAct-Helper’s planning module interprets MLLM outputs and the current graph state to proactively select actions that maximize thread-level parallelism and minimize human-robot workflow interference. For each candidate $a$:

1. Compute per-thread human/robot counts:
   • $n_k^\text{hum} = |\{x \in H_t : \pi(x) = k\}|$
   • $$n_k^\text{rob} = |\{x \in R_t \cup \{a\} : \pi(x) = k\}|$$
2. Determine mixing ratio and entropy:
   • $$p_k = n_k^\text{hum} / (n_k^\text{hum} + n_k^\text{rob})$$
   • $H_k(p_k) = -p_k \log p_k - (1-p_k) \log(1-p_k)$
3. Weight by frequency and aggregate:
   • $$w_k = (n_k^\text{hum} + n_k^\text{rob}) / \sum_j (n_j^\text{hum} + n_j^\text{rob})$$
   • $$H_\text{mix}(H_t, R_t \cup \{a\}) = \sum_k w_k H_k(p_k)$$

The planner selects

$$a^* = \underset{a \in \text{candidates}}{\operatorname{argmin}}\, H_\text{mix}(H_t, R_t \cup \{a\})$$

preferentially choosing actions on threads the human is not engaged on, guaranteeing parallel throughput and minimizing contention.

5. Decision-Loop and Integration

The end-to-end core loop proceeds as follows:

for t = 0, 1, ...: X_t = extract_keyframe_window(video) is_trig, task = MLLM_stage1(X_t, task_list) if not is_trig: output "Wait"; continue step, futures, scores = MLLM_stage2(X_t, task) Prog_t.update(step) G = task_graph[task] legal = {a in G.V_e \ Prog_t | φ(a) satisfied} candidates = intersect(legal, futures) if not candidates: action = "Wait" else: best_a, best_score = None, +inf for a in candidates: H = compute_H_mix(H_t, R_t ∪ {a}) if (H < best_score) or (H == best_score and pos_in_futures(a) < pos_in_futures(best_a)): best_a, best_score = a, H action = best_a R_t.append(action) output action

This logic ensures all actions are grounded in perceptual context, task-graph structure, and history-sensitive entropy minimization.

6. Empirical Results and Ablation Findings

Extensive evaluation demonstrates the effectiveness of ProAct-Helper within ProAct-75:

Metric	ProAct-Helper (7B)	Best Closed-Source
Trigger mF1	76.56%	70.35%
Saved Steps (SS)	0.361	0.111
Parallel Actions	19.41%	3.83%

ProAct-Helper outperforms closed-source baselines (e.g., Qwen3-VL-Flash, Gemini-2.5-Pro) with +6.21% mF1 in trigger detection, +0.25 average steps saved per decision, and a +15.58% absolute increase in the proportion of thread-parallel actions.

HBM ablation results confirm substantial gains in long-tail rare labels for both task and step classification.

7. Illustrative Scenarios and Significance

Case studies elucidate the planner’s structure-awareness:

• Trash-Bin Replacement: The robot selects "prepare new bag" on a separate branch rather than interfering with the human’s subthread, reducing $H_\text{mix}$ to near zero.
• Sofa Assembly: The human and robot reliably split on parallel branches, e.g., “screwing seat” and “attach leg,” maximizing throughput.

This demonstrates systematic avoidance of myopic imitation and establishes ProAct-Helper as the first multimodal agent to exploit explicit task-graph structure for robust, contextually appropriate proactivity at scale in vision-based collaborative scenarios (Zhu et al., 3 Feb 2026).