---
title: ProAct-Helper Framework for Proactive Assistance
url: https://www.emergentmind.com/topics/proact-helper-framework
type: topic
---

# ProAct-Helper Framework for Proactive Assistance

ProAct-Helper is a reference multimodal framework and structural decision baseline for structure-aware proactive assistance in complex, multi-step human environments. Developed in the context of the ProAct-75 benchmark, it fuses a multimodal large language model (MLLM) perception stack with explicit task-graph reasoning and an entropy-driven action planner, enabling grounded, parallel, and minimally interfering interventions for tasks spanning assistance, maintenance, and safety [2602.03430].

## 1. Framework Overview and Benchmark Context

ProAct-Helper was designed to address the limitations of passive or reactive agents that simply follow instructions or imitate human action order. It provides an end-to-end pipeline for five key subtasks: (1) trigger detection, (2) task identification, (3) step localization, (4) future-action prediction, and (5) structure-aware action selection for proactive robotic assistance. Central to this approach is explicit grounding in a task dependency graph (DAG) with AND/OR semantics and explicit representation of parallel execution threads.

Within ProAct-75, a comprehensive vision-based benchmark with 75 everyday human tasks and 91,581 annotated steps, ProAct-Helper serves as both reference implementation and evaluation baseline for structure-aware proactivity, covering domains such as collaborative assistance, routine maintenance, and environmental safety monitoring.

## 2. Perceptual Stack: Multimodal LLM and Hierarchical Binding Module

### State Detection and Perceptual Processing

ProAct-Helper processes incoming video streams using a sliding keyframe window (size 5, stride 3), with changes serialized into textual prompts for the MLLM backbone—Qwen2.5-VL (3B/7B), further adapted via LoRA. The core MLLM pipeline is two-stage:

- **Stage 1**: Receives keyframes and candidate tasks; outputs JSON including `is_trigger` (detection of actionable state) and selected `task`.
- **Stage 2**: If triggered, receives the same keyframes and the selected task; outputs current step, possible future steps (`future_steps`), and a vector of scores (urgency, value, priority).

### Hierarchical Binding Module (HBM)

To address the long-tail hierarchy inherent in trigger–task–step labels, ProAct-Helper employs an HBM, which extracts the hidden representations for trigger, task, and step spans via mean-pooling of token embeddings. Binding is enforced by paired contrastive InfoNCE losses between each hierarchical parent–child span, supplementing the primary cross-entropy with explicit structure:

$$
L(p, c) = -\frac{1}{B}\sum_{i=1}^B \log \frac{\exp(\mathrm{sim}(H_p^{(i)}, H_c^{(i)})/\tau)}{\sum_{j=1}^B \exp(\mathrm{sim}(H_p^{(i)}, H_c^{(j)})/\tau)}
$$

where $H_p$ and $H_c$ denote pooled embedding vectors for the parent and child, respectively, and $\tau=0.07$. The composite loss is:

$$
L = \text{CE} + \lambda_\text{trig2task} L(\text{trig} \to \text{task}) + \lambda_\text{task2step} L(\text{task} \to \text{step}) + \lambda_\text{trig\_cls} L_\text{binary}(\text{trigger})
$$

with recommended weights $\lambda_\text{trig2task}=0.3$, $\lambda_\text{task2step}=0.5$, $\lambda_\text{trig\_cls}=0.5$.

HBM regularization is essential for cross-level consistency, especially in rare subgraphs, as evidenced by substantial gains in task and step mF1 after introducing each binding loss.

## 3. Task Graph Representation and Legal Action Set

Each ProAct-75 task is represented as a directed acyclic graph $T=(V,E)$ with AND/OR nodes encoding dependencies and parallel threads. Formally:

- **Executable step nodes** $V_e$ and structural nodes $V_n$
- **AND/OR mapping** $\phi: V \to \{\text{AND}, \text{OR}\}$ determines precondition satisfaction:
  - **AND** node $v$: All predecessors in $\mathrm{Prog}_t$ required for execution.
  - **OR** node $v$: Any predecessor in $\mathrm{Prog}_t$ suffices.

The legal action set at time $t$ is:

$$
A_t^\mathrm{legal} = \{ a \in V_e \setminus \mathrm{Prog}_t \mid \text{preconds under }\phi\text{ are met} \}
$$

Parallel thread partitions are derived algorithmically such that each mid-level branch is treated as a distinct thread for the purpose of candidate selection and entropy computation.

## 4. Entropy-Driven Heuristic Search for Action Selection

ProAct-Helper’s planning module interprets MLLM outputs and the current graph state to proactively select actions that maximize thread-level parallelism and minimize human-robot workflow interference. For each candidate $a$:

1. **Compute per-thread human/robot counts**:
   - $n_k^\text{hum} = |\{x \in H_t : \pi(x) = k\}|$
   - $n_k^\text{rob} = |\{x \in R_t \cup \{a\} : \pi(x) = k\}|$

2. **Determine mixing ratio and entropy**:
   - $p_k = n_k^\text{hum} / (n_k^\text{hum} + n_k^\text{rob})$
   - $H_k(p_k) = -p_k \log p_k - (1-p_k) \log(1-p_k)$

3. **Weight by frequency and aggregate**:
   - $w_k = (n_k^\text{hum} + n_k^\text{rob}) / \sum_j (n_j^\text{hum} + n_j^\text{rob})$
   - $H_\text{mix}(H_t, R_t \cup \{a\}) = \sum_k w_k H_k(p_k)$

The planner selects

$$
a^* = \underset{a \in \text{candidates}}{\operatorname{argmin}}\, H_\text{mix}(H_t, R_t \cup \{a\})
$$

preferentially choosing actions on threads the human is not engaged on, guaranteeing parallel throughput and minimizing contention.

## 5. Decision-Loop and Integration

The end-to-end core loop proceeds as follows:

```python
for t = 0, 1, ...:
    X_t = extract_keyframe_window(video)
    is_trig, task = MLLM_stage1(X_t, task_list)
    if not is_trig: output "Wait"; continue
    step, futures, scores = MLLM_stage2(X_t, task)
    Prog_t.update(step)
    G = task_graph[task]
    legal = {a in G.V_e \ Prog_t | φ(a) satisfied}
    candidates = intersect(legal, futures)
    if not candidates: action = "Wait"
    else:
        best_a, best_score = None, +inf
        for a in candidates:
            H = compute_H_mix(H_t, R_t ∪ {a})
            if (H < best_score) or (H == best_score and pos_in_futures(a) < pos_in_futures(best_a)):
                best_a, best_score = a, H
        action = best_a
    R_t.append(action)
    output action
```

This logic ensures all actions are grounded in perceptual context, task-graph structure, and history-sensitive entropy minimization.

## 6. Empirical Results and Ablation Findings

Extensive evaluation demonstrates the effectiveness of ProAct-Helper within ProAct-75:

| Metric            | ProAct-Helper (7B) | Best Closed-Source |
|-------------------|-------------------|-------------------|
| Trigger mF1       | 76.56%            | 70.35%            |
| Saved Steps (SS)  | 0.361             | 0.111             |
| Parallel Actions  | 19.41%            | 3.83%             |

ProAct-Helper outperforms closed-source baselines (e.g., Qwen3-VL-Flash, Gemini-2.5-Pro) with +6.21% mF1 in trigger detection, +0.25 average steps saved per decision, and a +15.58% absolute increase in the proportion of thread-parallel actions.

HBM ablation results confirm substantial gains in long-tail rare labels for both task and step classification.

## 7. Illustrative Scenarios and Significance

Case studies elucidate the planner’s structure-awareness:

- **Trash-Bin Replacement**: The robot selects "prepare new bag" on a separate branch rather than interfering with the human’s subthread, reducing $H_\text{mix}$ to near zero.
- **Sofa Assembly**: The human and robot reliably split on parallel branches, e.g., “screwing seat” and “attach leg,” maximizing throughput.

This demonstrates systematic avoidance of myopic imitation and establishes ProAct-Helper as the first multimodal agent to exploit explicit task-graph structure for robust, contextually appropriate proactivity at scale in vision-based collaborative scenarios [2602.03430].

Source: https://www.emergentmind.com/topics/proact-helper-framework