Egocentric Procedural AI (EgoProceAssist)

• EgoProceAssist is an egocentric AI assistant that integrates video, gaze, and text cues to support multi-step procedural tasks in a first-person view.
• It employs modular architectures and proactive intervention to detect errors, infer user intent, and guide procedural actions.
• Evaluations demonstrate notable improvements in mistake detection, video question answering, and personalized, proactive procedural assistance.

Egocentric Procedural AI Assistant (EgoProceAssist) denotes an egocentric AI assistant tailored to step-by-step support daily procedural tasks in a first-person view. In the formalization introduced for this setting, the assistant is modeled as a multimodal mapping from egocentric video, optional gaze, and textual cues to three outputs: a procedural error signal, a learned procedural plan, and question-answer responses (Li et al., 17 Nov 2025). The concept occupies the intersection of egocentric perception, procedural learning, mistake detection, video question answering, and proactive assistance. It also marks a shift away from assistants that remain largely reactive, responding only to explicit user commands, toward systems that can detect service opportunities, infer user needs, and intervene when appropriate (Wen et al., 16 Oct 2025).

1. Definition and conceptual scope

The canonical formulation of EgoProceAssist is

$$\mathcal{F}:\{V,\,G,\,T\}\longrightarrow(E,\;P,\;A),$$

where $V$ denotes egocentric video frames, $G$ optional gaze coordinates or heatmaps, $T$ textual inputs such as ASR transcripts or procedural scripts, $E$ a binary error indicator per frame or step, $P$ a learned procedural plan with step labels and temporal intervals, and $A$ the set of question-answer pairs produced in response to user queries (Li et al., 17 Nov 2025). This formalization makes explicit that EgoProceAssist is not a single task but a compound system integrating egocentric procedural error detection, egocentric procedural learning, and egocentric procedural question answering.

Within this formulation, procedural tasks are treated as multi-step activities whose success depends on temporal order, state transitions, and often tool- or object-specific affordances. The egocentric setting introduces constraints absent from third-person video understanding: dynamic head motion, partial occlusion by hands, ambiguous object references, long-horizon temporal dependencies, and the need to align assistance with what the wearer actually sees. The survey literature emphasizes that third-person instructional sets do not fully translate to first-person contexts, which is one reason EgoProceAssist is presented as distinct from generic VLM-based assistants (Li et al., 17 Nov 2025).

A further defining property is procedural proactivity. The Alpha-Service framework, proposed under the broader AI4Service paradigm, argues that a truly intelligent and helpful assistant should anticipate user needs and take actions proactively when appropriate (Wen et al., 16 Oct 2025). In the EgoProceAssist setting, that proactive behavior is specialized to procedural state: detecting when a step is incomplete, when a mistake is emerging, when a clarification is needed, or when the next action should be surfaced before a query is spoken.

2. Architectural paradigms

Several architecture families have converged on a modular organization. Alpha-Service is explicitly inspired by the classical von Neumann architecture and comprises five modules: an Input Unit for continuous first-person perception, a Central Processing Unit for orchestration, an Arithmetic Logic Unit for tool calling, a Memory Unit for long-term personalization, and an Output Unit for natural human interaction (Wen et al., 16 Oct 2025). Its Input Unit implements a two-tiered MLLM strategy with a lightweight trigger model, fine-tuned Qwen2.5-VL-3B, operating continuously, and a larger Qwen2.5-VL-7B streaming model invoked only when a service opportunity is detected. In implementation, the framework is realized as six agent processes—Trigger, Streaming, Orchestrator, Tool, Memory, and Output—communicating through JSON messages over simple in-process queues (Wen et al., 16 Oct 2025).

A second paradigm centers on continuous step-aware reasoning. Pro$^2$Assist combines motion-based perception from AR glasses with a step-oriented procedural context extractor and a VLM-based proactive reasoning engine (Xu et al., 5 May 2026). Its perception stack uses head-motion–aware sampling, switching between “normal” and “burst” rates according to head angular velocity, together with optical-flow-based key-moment selection. The reasoning engine conditions on four inputs at each key moment: a structured guideline extracted from a free-form instruction, historical progress, the current image, and hand motion cues. A consistency checker then suppresses single-moment errors and redundant prompts before assistance is displayed on the glasses (Xu et al., 5 May 2026).

A third paradigm decouples interaction from planning. In the planner–watcher–recover design, the joint distribution over decision, utterance, and updated plan is factorized into a duplex interaction model and a planner that is invoked only when an interruption occurs (Kundu et al., 3 Jun 2026). The interaction model operates at 2 fps over plan-anchored clips, dialogue history, and the original goal query, while the planner revises a structured plan containing completed, current, and remaining steps together with visual cues. This factorization is motivated by the observation that long-horizon plan maintenance and low-latency interruption decisions impose different statistical and computational demands (Kundu et al., 3 Jun 2026).

Long-context reasoning architectures add another layer. Egocentric Co-Pilot organizes continuous first-person assistance around a unified event log, Temporal Chain-of-Thought, and Hierarchical Context Compression, with an LLM orchestrating perception, reasoning, and web tools over smart glasses (Yang et al., 1 Mar 2026). EgoLife’s EgoButler system similarly separates clip-level understanding from hierarchical retrieval, storing clip features, captions, and higher-level summaries in a multi-level memory bank for week-scale question answering (Yang et al., 5 Mar 2025). Across these implementations, recurring components are continuous perception, compressed long-horizon memory, structured procedural state, a reasoning or orchestration core, and a low-friction output channel.

3. Procedural perception, grounding, and recovery

EgoProceAssist requires more than generic scene understanding. It must determine what procedural state the user is in, what object or affordance is currently relevant, and whether the observed action is correct, incomplete, or out of plan.

Procedural structure is often represented as a task graph. In differentiable task graph learning, a procedure is modeled as a directed acyclic graph over key-steps, with weighted edges encoding partial ordering and precondition structure (Seminara et al., 2024). The proposed TGML loss directly optimizes edge weights by maximum likelihood over demonstration sequences, and the resulting graphs improve both graph recovery and online mistake detection. On CaptainCook4D, direct optimization achieves 87.8 F$_1$ versus 71.1 for the best prior approach, while online mistake detection gains reach +19.8 points on Assembly101 and +7.5 points on EPIC-Tent (Seminara et al., 2024). In deployment, mistake detection reduces to checking whether predicted preconditions of the current step have already appeared in the observed history.

Affordance grounding is equally central. EgoIntention introduces the first dataset for egocentric visual intention grounding, where the query is an implicit intention rather than an explicit object name (Sun et al., 18 Apr 2025). Its Reason-to-Ground method decomposes grounding into chained intention reasoning and object localization. On EgoIntention, RoG instruction tuning improves MiniGPTv2 from 39.1 to 42.6 overall $[email protected]$ relative to hybrid naive supervised fine-tuning, while also retaining or slightly improving exocentric referring performance (Sun et al., 18 Apr 2025). This is directly relevant to procedural assistance because instructions such as “I need something to stand on” or “What do I use next?” often presuppose affordance reasoning rather than nominal object retrieval.

Question-driven procedural completion supplies another pathway. AssistQ defines Affordance-centric Question-driven Task Completion, in which a model learns from instructional videos to provide step-by-step help in the user’s current view (Wong et al., 2022). Its Q2A model explicitly grounds among the reference instructional video, the video script, the current egocentric user image, and the user’s procedural question. On AssistQ, Q2A reaches $V$0, $V$1, mean rank $V$2, and MRR $V$3, outperforming the listed VQA-style baselines (Wong et al., 2022).

Mistake-aware recovery extends beyond binary anomaly flags. PIE-V formalizes five error types—Deletion, Insertion, Substitution, Transposition, and Wrong Execution—and pairs a psychology-informed error planner with a correction planner, an LLM writer for cascade-consistent rewrites, and an LLM judge for procedural coherence (Loginova et al., 16 Apr 2026). Its nine-metric rubric spans error validity, human plausibility, confusability, taxonomy fit, procedure logic, sequence consistency, state-change coherence, video plausibility, and text–video grounding. This broadens EgoProceAssist from merely detecting deviations to modeling recoveries and the downstream consequences of earlier errors (Loginova et al., 16 Apr 2026).

4. Benchmarks, datasets, and evaluation protocols

The evaluation landscape has expanded from step labels and offline QA to long-duration streaming, personalized proactivity, and out-of-plan recovery. TeleEgo defines a long-duration, streaming, omni-modal benchmark with over 14 hours per participant across five participants, 12 diagnostic subtasks, and 3,291 human-verified QA items spanning memory, understanding, and cross-memory reasoning (Yan et al., 28 Oct 2025). HoloAssist contributes 166 hours of egocentric human interaction data from 350 unique instructor–performer pairs over 20 multi-step tasks, with synchronized RGB, depth, audio, IMU, head pose, gaze, and 3-D hand pose streams, as well as action and conversational annotations (Wang et al., 2023). Ego-EXTRA adds 50 hours of expert–trainee procedural assistance, 123 videos, 10 activities across four scenarios, and approximately 15,000 high-quality visual question-answer sets collected under a Wizard-of-Oz protocol (Ragusa et al., 15 Dec 2025).

Synthetic and semi-synthetic resources play an increasing role. ProAssist aggregates 3,934 videos totaling 479 hours from six public egocentric sources and curates 30,135 train dialogues together with validation and test splits for proactive dialogue generation (Zhang et al., 6 Jun 2025). EgoPro-Bench targets personalized proactive interaction with 12,085 training samples and 2,400 evaluation videos across 12 domains, using simulated user profiles and explicit human-machine interaction timing (Ran et al., 8 May 2026). EgoProactive, paired with Pro$V$4Bench, focuses on proactive procedural assistance under deviations: 700 wearable-egocentric sessions yield 9,935 evaluation instances, including 1,833 scripted Out-of-Plan cases, while Pro$V$5Bench re-annotates Ego4D, EPIC-KITCHENS, Ego-Exo4D, HoloAssist, and HowTo100M into 42,275 evaluation instances and 249,584 training instances (Kundu et al., 3 Jun 2026).

Resource	Primary emphasis	Salient scale/detail
TeleEgo	Streaming omni-modal QA	over 14 hours per participant; 3,291 QA
HoloAssist	Interactive task guidance	166 hours; 350 instructor–performer pairs
Ego-EXTRA	Expert–trainee assistance	50 hours; 123 videos; $V$6 QA
EgoPro-Bench	Personalized proactive interaction	12,085 train; 2,400 eval
EgoProactive / Pro$V$7Bench	OOP recovery and proactive guidance	700 sessions; 42,275 eval / 249,584 train

Evaluation metrics reflect this expansion. TeleEgo introduces Real-Time Accuracy,

$V$8

which counts only answers produced within a 5 s decision window, and Memory Persistence Time, which re-queries previously answered items at 60 s intervals until recall fails (Yan et al., 28 Oct 2025). Pro$V$9Assist adds the Step-aware Timeliness Score for proactive timing within ground-truth intervention windows (Xu et al., 5 May 2026). The planner–watcher–recover benchmark defines G-Mean F1 for balanced interruption timing and Proactive Quality Score, which combines correct silence, correct interruption timing, and judged utterance quality into a session-level end-to-end score (Kundu et al., 3 Jun 2026). These protocols make timing, silence calibration, and recovery quality first-class evaluation targets rather than afterthoughts.

5. Representative methods and empirical findings

Empirical results consistently show that egocentric procedural assistance is not solved by generic vision-language scaling alone. On EgoIntention, RoG outperforms naive and hybrid supervised alternatives on implicit intention grounding, especially under uncommon affordances, indicating that chained reasoning before localization is materially useful in first-person settings (Sun et al., 18 Apr 2025). On CaptainCook4D and PREGO-style mistake detection benchmarks, differentiable task graph learning substantially improves graph inference and online error monitoring, supporting the view that explicit procedural structure remains advantageous even in the MLLM era (Seminara et al., 2024).

For affordance-centric procedural QA, AssistQ’s Q2A model surpasses several VQA-related baselines yet leaves “large room for improvement,” a formulation already present in the original task definition (Wong et al., 2022). For expert-level assistance, the gap is sharper: on Ego-EXTRA, text-only LLMs achieve 8.7%–26.7% accuracy, MLLMs 10.7%–33.1%, and the human upper bound is approximately 89.7%, with the best listed model being LLaVa-OneVision at 33.1% (Ragusa et al., 15 Dec 2025). The dominant failure modes include spatial grounding, temporal step ordering, quantitative reasoning, and coreference under elliptical references such as “it” or “those three parts” (Ragusa et al., 15 Dec 2025).

Step-aware proactivity improves matters but does not eliminate the gap. Pro$G$0Assist reports over 21% improvement in procedural action understanding accuracy relative to the best-performing baselines, together with up to 2.29x proactive timing accuracy, and its user study shows that 90% of 20 participants found the system useful (Xu et al., 5 May 2026). In personalized proactive interaction, ProAct-Stream improves EgoPro-Intent from an SFT baseline of Precision $G$1, Recall $G$2, and $G$3 to $G$4, with Memory Consistency increasing from 3.81 to 4.10 and Response Quality from 3.02 to 3.23 under RL with concise “think” traces (Ran et al., 8 May 2026).

The strongest evidence against zero-shot sufficiency comes from the planner–watcher–recover line. On six proactive-guidance datasets, proprietary and open zero-shot models average G-Mean $G$5 and PQS $G$6, often collapsing to one class (Kundu et al., 3 Jun 2026). After fine-tuning with the decoupled planner–interaction architecture, full PWR reaches G-Mean 0.83 for Qwen 3.6 and 0.76 for Llama 4, with PQS 0.47 and 0.63 respectively (Kundu et al., 3 Jun 2026). On 3,433 Out-of-Plan instances from EgoProactive, PWR-Oracle detects 99.6% of scripted deviations and scores recovery guidance at 4.82/5, whereas the deployable predicted-plan version detects 78.7% of deviations and reaches recovery quality 2.72/5 for Llama 4 (Kundu et al., 3 Jun 2026). These results indicate that plan quality, plan utilization, and interruption calibration are separable bottlenecks.

A related trend appears in exocentric-to-egocentric transfer. Exo2Ego learns mappings between exocentric and egocentric feature spaces through cycle consistency and KL alignment, using 1.1M synchronized ego-exo clip-text pairs for pre-training (Zhang et al., 12 Mar 2025). Its gains on EgoPlan and VLN-QA are reported as +13.4% and +7.0% over baseline, suggesting that exocentric procedural knowledge can be repurposed when the viewpoint gap is explicitly modeled (Zhang et al., 12 Mar 2025).

6. Open problems, misconceptions, and research directions

A persistent misconception is that proactive egocentric assistance is equivalent to event alerting. Recent benchmark work rejects this narrowing. EgoPro-Bench explicitly argues that earlier efforts are largely confined to alert scenarios, neglect personalized context and the precise timing of human-machine interactions (Ran et al., 8 May 2026). TeleEgo, ProAssist, Pro$G$7Assist, and EgoProactive instead frame assistance as a joint problem of remembering, understanding, timing, explanation, silence, and recovery under streaming constraints (Yan et al., 28 Oct 2025).

A second misconception is that personalization necessarily implies online fine-tuning. In Alpha-Service, personalized services are implemented by retrieving symbolic JSON memories and prepending them to the prompt; no gradient-based personalization or online fine-tuning is performed in the proof-of-concept (Wen et al., 16 Oct 2025). This suggests that, at least in current smart-glasses deployments, memory design and retrieval policy may matter more than parameter adaptation.

A third misconception is that sufficiently large zero-shot VLMs can already function as reliable procedural coaches. The cross-dataset proactive results directly contradict this: without task-specific training, even strong proprietary and open models exhibit silence bias or class collapse (Kundu et al., 3 Jun 2026). Survey work correspondingly identifies data scarcity and bias, long-term procedural comprehension, and the tension between annotation dependence and real-time constraints as central unresolved issues (Li et al., 17 Nov 2025).

Several technical fronts remain open. Long-horizon memory and cross-session personalization are still limited; Pro$G$8Assist is designed for single-session tasks, while EgoLife’s hierarchical retrieval addresses week-scale daily-life QA rather than explicit procedural state tracking (Xu et al., 5 May 2026). Ambiguous reference resolution remains difficult, especially in expert-trainee dialogue; Ego-EXTRA highlights failures on spatial deixis, coreference, and quantitative reasoning (Ragusa et al., 15 Dec 2025). Mistake-aware supervision is also insufficiently standardized, which motivates PIE-V’s controlled error injection framework and nine-metric human rubric (Loginova et al., 16 Apr 2026).

Deployment introduces its own trade-offs. Egocentric Co-Pilot reports end-to-end WebRTC latency of 200–350 ms for glass-to-cloud-to-glass interaction, versus roughly 350–450 ms for an on-premise WebSocket baseline, exposing a concrete latency–mobility–resource trade-off (Yang et al., 1 Mar 2026). This suggests that practical EgoProceAssist systems will likely remain hybrid for the near term, combining lightweight on-device triggers, compressed memory, selective cloud offload, and tightly budgeted reasoning.

The main future directions identified across the literature are increasingly convergent: richer first-person data; explicit procedural planning; stronger episodic and semantic memory; unified architectures that fuse error detection, procedure learning, and QA; concise reasoning for low-latency proactivity; and recovery-aware training on predicted, not only gold, plans (Li et al., 17 Nov 2025). Taken together, the field is moving from first-person observation toward first-person procedural agency: systems that can plan, watch, ask, warn, recover, and adapt while remaining aligned with the wearer’s current view and ongoing task.