Adaptive Multiple Normal Action Representation (AMNAR)

• AMNAR is a framework for error detection that maintains adaptive, context-aware representations for all valid next actions in procedural videos.
• It employs a graph-based prediction module coupled with a reconstruction block using dilated convolutions and local attention to model action-branching effectively.
• Experimental results show AMNAR outperforms baselines like EgoPED, significantly improving error detection accuracy on assembly and cooking task benchmarks.

Adaptive Multiple Normal Action Representation (AMNAR) is a framework for error detection in procedural activity videos, addressing the non-determinism in valid next actions following a sequence of executed steps. By maintaining adaptive, context-aware representations for all valid next actions at each step, AMNAR overcomes the limitations of static action prototypes and enhances robustness to environment or label distribution shifts. Its architecture integrates a graph-based valid-action predictor and a learned reconstruction module, yielding state-of-the-art performance on established procedural task benchmarks (Huang et al., 28 Mar 2025).

1. Problem Formulation and Motivation

Procedural tasks, such as assembly and cooking, are characterized by action sequences $s = (y_1, y_2, \ldots, y_T)$ with $y_t \in \mathcal{Y}$ (action classes, possibly including a background class). The valid temporal structure of these actions is embodied in a task graph $G=(V,E)$, a directed acyclic graph encoding permissible transitions. At inference time, given the sequence prefix $s_{<t}=(y_1,\ldots, y_{t-1})$, the challenge is to:

1. Predict the set of all valid next actions $PA_t \subseteq \mathcal{Y}$ according to $G$ and $s_{<t}$.
2. For each $a \in PA_t$, reconstruct a corresponding context-adaptive “normal” feature representation $f_{t,a}^{normal}$.

Error detection is performed by comparing the observed action embedding $f_t^{obs}$ with all $f_{t,a}^{normal}$; a large minimal distance flags an error. The rationale for maintaining multiple adaptive prototypes is to model legitimate action-branching (e.g., either “add sugar” or “add cream” can validly follow “pour coffee”). Static prototype approaches misclassify such permissible variations as errors or overlook true errors, especially under distribution shift.

The set $PA_t$ is operationalized as:

$PA_t = \bigcup_{a \in s^*} A[a] \setminus s^*,$

where $A[a]$ are one-step successors of $a$ in $G$, and $s^*$ is the longest merged subsequence of $s$ conforming to $G$.

2. Architecture: Potential Action Prediction and Representation Reconstruction

AMNAR’s architecture is divided into two causal modules executed at each action time step $t$:

2.1 Potential Action Prediction Block (PAPB)

PAPB, given $G$ and $s_{<t}$, computes $PA_t$ through dynamic programming and adjacency merging:

• Build adjacency list $A$ from $G$.
• For each position $i$ in $s_{<t}$, compute the longest valid non-branching subsequences (arrays $dp[i]$, $subseq[i]$).
• Merge all maximal subsequences and neighbors to form $s^*$.
• Compute $PA_t = (\cup_{a \in s^*} A[a]) \setminus s^*$.

PAPB is algorithmically guaranteed to recover all valid next actions consistent with the observed action prefix and task-graph constraints.

2.2 Representation Reconstruction Block (RRB)

RRB, provided with feature sequence $F_{1:ed_{t-1}} \in \mathbb{R}^{D \times L}$ (up to previous action boundary) and candidate class-embedding vectors $f^{class{-}emb}(a)$, reconstructs for each $a \in PA_t$ a normal feature embedding $f_{t,a}^{normal}$:

• Temporal encoder: five 1D causal dilated Conv layers (kernel size 3, dilations $d_i=3^i$) produce key/value feature maps $K, V$.
• Local cross-attention (window size 32, 2 heads): $f^{class{-}emb}(a)$ as query, attending to $K, V$ yields $f_{t,a}^{normal}$.

The class-embedding for class $y$ is precomputed as:

$$f^{class{-}emb}(y)=\frac{1}{N_y} \sum_{t \in \mathcal{I}_y} f_t^{action},$$

with $\mathcal{I}_y$ the set of normal training samples of label $y$.

3. Mathematical Formulation and Training Objectives

Training employs normal (error-free) sequences:

• Prediction loss (optional):

$$L_{pred} = -\sum_t \sum_{i=1}^{|PA_t|} y_{t,i} \cdot \log p_{t,i}$$

where $p_{t,i}$ is the classifier probability over $PA_t$ and $y_{t,i}$ is the one-hot ground truth.

• Reconstruction loss:

$$L_{rec} = \sum_t \| f_{t,i^*}^{normal} - f_t^{obs} \|_2^2$$

where $i^*$ is the ground-truth next action index.

• Total loss:

$L = L_{pred} + \lambda L_{rec}$

with $\lambda$ a tunable hyperparameter.

• Error detection (inference):

$$d_t = \min_{i=1\ldots |PA_t|} \| f_t^{obs} - f_{t,i}^{normal} \|_2$$

An error is flagged if $d_t > \tau$, where $\tau$ is chosen to achieve a pre-specified true-negative rate on held-out data.

4. Training and Inference Procedures

The end-to-end AMNAR procedure is as follows:

• Training:

1. For each epoch, iterate over normal videos.
2. Use an action segmentation model to extract frame features $F$, boundaries, ground-truth labels $y_t$, and observed embeddings $f_t^{obs}$.
3. For each action $t$: obtain $PA_t$ via PAPB, reconstruct $\{f_{t,i}^{normal}\}$ via RRB, accumulate $L_{rec}$ for the correct candidate, and optionally $L_{pred}$.
4. Backpropagate total loss and update the RRB and, if present, the prediction head.

• Inference (error detection):

1. For a new video, extract features, boundaries, and $f_t^{obs}$.
2. For each $t$: predict $PA_t$ via PAPB, reconstruct all $\{f_{t,i}^{normal}\}$ via RRB, compute $d_t$, and flag errors if $d_t>\tau$.

5. Experimental Evaluation and Benchmarks

AMNAR was evaluated on two large-scale video datasets:

Dataset	Domain	Hours (train/val)	Tasks/labels	Notes
HoloAssist	Assembly	166/20	19 tasks, 1 excl. held	Errors excluded from training
CaptainCook4D	Cooking	94.5	24 recipes, 10K labels	Fine-grained labels and errors

Task graph construction employed transition counts to greedily assemble acyclic DAGs per dataset.

Metrics:

• Frame-wise Error Detection Accuracy (EDA): percentage of correctly labeled (normal/error) frames
• F1 score: per-action error detection

Key results:

• HoloAssist EDA: Random baseline 72.1%, EgoPED 84.7%, AMNAR 88.9%
• Gain on most non-deterministic task (coffee): +6.3 points EDA over EgoPED
• CaptainCook4D: F1 increases from 65.4% (EgoPED) to 71.8% (AMNAR)

Ablation findings:

• Removing RRB (static prototype): –3.1 points EDA
• Only top-1 next-action: –2.4 points EDA
• Non-deterministic steps (coffee): +5.8 points over EgoPED

6. Insights, Limitations, and Future Directions

Modeling multiple adaptive prototypes $f_{t,i}^{normal}$ enables AMNAR to accurately capture legitimate action-branching, minimizing false positives in non-deterministic procedural domains. The temporal encoder’s dilated convolutional structure and local attention allow integration of broad temporal context and fine-grained cues, improving reconstruction fidelity. Ablation studies confirm the necessity of both these elements, and of supporting multiple prototypes over a single static representation.

Limitations include the focus on step-level error detection: sequence-level anomalies such as missing actions or misordering are not directly addressed. AMNAR assumes precise action segmentation and boundary estimation, suggesting that end-to-end integration with segmentation modules is a promising avenue. Automatic threshold calibration for $d_t$ under domain shift or non-stationary environments is unresolved.

7. Significance and Comparative Perspective

AMNAR advances online error detection in procedural videos by explicitly representing all contextually valid action possibilities at each step and learning adaptive, context-sensitive normal features. This approach yields superior performance relative to state-of-the-art methods not accounting for action non-determinism, such as EgoPED, particularly in tasks with substantial branching. The two-stage design—graph-based next-action retrieval plus adaptive feature reconstruction—represents a principled generalization of prior approaches and is adaptable to a variety of procedural learning paradigms (Huang et al., 28 Mar 2025).