--- title: 'ActionDiff: Diffusion Models in Action Analysis' url: https://www.emergentmind.com/topics/actiondiff type: topic --- # ActionDiff: Diffusion Models in Action Analysis Searching arXiv for papers using the term "ActionDiff" to ground the article and disambiguate the topic. ActionDiff is an overloaded term in recent arXiv literature rather than a single canonical model. It has been used for a conditional diffusion model for procedure planning in instructional videos, a diffusion-feature framework for action recognition under severe domain shift, a discrete image-diffusion formulation of temporal action detection, a fine-grained action differencing task over video pairs, and, in a separate categorical line of work, an exposition label for change actions and change-action models [2403.08591] [2509.08908] [2404.01051] [2503.07860] [1902.05465]. Across these usages, the shared motif is not a fixed architecture but the use of diffusion, denoising, or difference structures to model uncertainty, compositionality, or subtle variation in action-related objects. ## 1. Terminological scope and disambiguation The label has been attached to technically distinct objects with different state spaces, objectives, and evaluation regimes. | Usage | Problem setting | Paper | |---|---|---| | ActionDiffusion / ActionDiff | Procedure planning in instructional videos | [2403.08591] | | ActionDiff | Action recognition across unseen domains | [2509.08908] | | ActionDiff / ADI-Diff | Temporal action detection via image diffusion | [2404.01051] | | ActionDiff | Fine-grained differencing between two videos of the same action | [2503.07860] | | ActionDiff | Change actions and change-action models in category theory | [1902.05465] | These names are not interchangeable. One line predicts an action plan from start and goal observations; another uses a frozen Stable Video Diffusion backbone as a feature extractor for classification; another denoises structured “AD images” whose rows are probability distributions over classes or boundaries; another formalizes a benchmark in which a model must describe how two executions of the same action differ; and another develops a semantics of generalized differentiation in cartesian categories. A precise reading therefore requires identifying the paper-specific definition of the underlying action object: discrete action classes, action sequences, temporal proposals, action chunks, or categorical changes. ## 2. ActionDiffusion for procedure planning In the instructional-video setting, ActionDiffusion is a conditional diffusion model for procedure planning that predicts an entire sequence of intermediate actions \(a_1,\dots,a_T\) from a start frame feature \(o_s\) and a goal frame feature \(o_g\) [2403.08591]. The clean variable \(x_0\) concatenates the task-class one-hot \(c\), the one-hot action steps \(a_1,\dots,a_T\), and the visual endpoints \(o_s,o_g\). The model follows a DDPM-style forward process \[ q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I), \] with closed-form \[ x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon, \] and a cosine schedule \[ \bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2). \] Its distinctive mechanism is the action-aware noise mask. Each one-hot action \(a_i\) is embedded as \(a_{e_i}=W_e a_i\in\mathbb R^d\), normalized by \(g(\cdot)\), and accumulated over time: \[ M_a[j]=\sum_{i=1}^j g(a_{e_i}),\quad j=1\dots T. \] The forward process then replaces the isotropic covariance by \(\beta_n(I+M_a)\), so the noised tensor explicitly carries information about action order and identity. The denoiser is a U-Net with timestep embeddings and multi-head self-attention at the bottleneck, which allows cross-step correlations to be recovered during denoising. Training is joint: a task classifier minimizes \(L_{\text{task}}\), the diffusion module minimizes \(L_{\text{diff}}\), and the total loss is \[ L_{\text{total}}=L_{\text{diff}}+\lambda L_{\text{task}},\quad \lambda=1.0. \] The reported hyperparameters are diffusion timesteps \(N=1000\), cosine schedule with \(\tau=0.008\), action embedding dimension \(d=512\), time-embedding dimension \(128\), a U-Net with 4 down/up-sampling levels, channel base \(64\), 8 attention heads at bottleneck, Adam with learning rate \(1\times 10^{-4}\), batch size \(64\), and 50 K training steps. On CrossTask with \(T=3\), ActionDiff reports \(37.86\%\) success rate, \(65.58\%\) mAcc, and \(67.54\%\) mSIoU, compared with PDPP at \(37.20\%\), \(64.67\%\), and \(66.57\%\). On COIN, it reports \(23.58\%\) success rate, \(45.24\%\) mAcc, and \(53.86\%\) mSIoU; on NIV, \(32.96\%\), \(49.01\%\), and \(57.85\%\). The paper states that it outperforms previous state of the art on all metrics on CrossTask and NIV and all metrics except accuracy on COIN. Ablations on COIN show that replacing the accumulated mask with a single-add variant lowers success rate from \(23.58\%\) to \(21.52\%\), and removing self-attention lowers it from \(23.58\%\) to \(23.14\%\). ## 3. ActionDiff for action recognition across untrained domains A different ActionDiff denotes a video-level action-recognition framework built on a frozen Stable Video Diffusion model, specifically SVD-XT, with a lightweight transformer head trained for downstream classification [2509.08908]. Here diffusion is not used to generate action sequences. Instead, it is used as a source of semantically enriched intermediate representations. Given frame \(x^m\), the latent \(z_0^m=E(x^m)\) follows a standard latent diffusion formulation, and the denoiser \(\epsilon_\theta\) is conditioned on a CLIP embedding of the middle frame. The framework extracts features from an intermediate U-Net activation \(h_{t,c}^{m,l}\) at layer \(l=9\) and timestep \(t=20\) out of \(T=30\), chosen to emphasize semantic information over pixel-level detail. The pooled feature is \[ f^m=\frac{1}{h_lw_l}\sum_{i=1}^{h_l}\sum_{j=1}^{w_l} h_{t,c}^{m,l}(i,j). \] These frame features are aggregated by appending a learned class token, projecting to a common dimension, adding 1D positional encodings, and processing the token sequence with a small transformer encoder. Only the action-classification head is trained; the diffusion backbone remains frozen. For single-label tasks the head uses cross-entropy, while multi-label settings use sigmoid with binary cross-entropy or Focal Loss. MixUp is also applied. The evaluation targets domain shift rather than generative fidelity. On Animal Kingdom, ActionDiff reports \(80.79\) mAP on the full dataset, exceeding MAMBA-MSQNet at \(74.60\) mAP, V-JEPA at \(78.64\), and SDv2 at \(78.66\). On unseen-species accuracy it reports \(51.49\%\), compared with MSQNet’s \(42.5\%\) and V-JEPA’s \(51.40\%\). On Charades-Ego, the 1st \(\to\) 1st setting yields \(36.5\) mAP and the 3rd \(\to\) 1st setting \(30.2\) mAP. On cross-context UCF \(\to\) HMDB, it reports \(77.6\%\) accuracy, and on HMDB \(\to\) UCF, \(81.5\%\). The ablations indicate that replacing the transformer with a linear \(+\) MLP head lowers Animal Kingdom performance to \(75.2\) mAP, an MLP head lowers it to \(73.0\) mAP, unconditional features reduce mAP to \(77.4\), and removing Focal Loss or MixUp hurts by approximately \(1\%\). The paper’s layer–timestep grid search further reports that later timesteps perform best in-domain, whereas earlier timesteps yield better out-of-domain generalization. ## 4. ActionDiff as action detection via an image diffusion process A third usage corresponds to ADI-Diff, where action detection is reformulated as the generation of three structured images: an action-class image \(x^a\in[0,1]^{N\times C}\), a starting-point image \(x^s\in[0,1]^{N\times 2}\), and an ending-point image \(x^e\in[0,1]^{N\times 2}\) [2404.01051]. Each row is a discrete probability distribution; in the ground truth, rows are one-hot. The diffusion process is discrete rather than Gaussian. For a row \(z_0\in\{e_1,\dots,e_C\}\), the forward step is \[ z_t=(1-\beta_t)z_{t-1}+\beta_t v_t, \] where \(v_t\sim \text{Multinomial}_K(K,\tfrac1C\mathbf1)/K\). With \[ \alpha_t=1-\beta_t,\qquad \bar\alpha_t=\prod_{\tau=1}^t \alpha_\tau, \] the process converges in expectation to the uniform distribution as \(\bar\alpha_T\to 0\). Reverse denoising is performed by a network \(d_\phi\) conditioned on frozen video features \(f_{ST}\) and a sinusoidal step embedding \(f_t\): \[ \hat z_{t-1}=d_\phi(\hat z_t,f_{ST},f_t). \] Training uses a simplified MSE objective supervising each reverse step against the forward-chain target. The denoiser is a Row-Column Transformer. Column-wise encoding treats each column as a token and applies multi-head self-attention across columns; row-wise encoding applies temporal convolution across rows, then self-attention across time. This design is tailored to the asymmetric structure of the AD images, where rows correspond to frames and columns to classes or boundary indicators. Inference thresholds the first column of \(\hat x_0^s\) and \(\hat x_0^e\) at \(\delta=0.9\), averages frame indices within clusters to obtain boundaries, forms proposals by pairing starts with later ends, averages class scores over each proposal interval, and applies Soft-NMS. The reported setup uses I3D features on THUMOS14, R(2+1)D on ActivityNet-1.3, diffusion steps \(T=50\), block stacks \(L=3\), sample-per-step \(M=10\), and a linearly increasing noise schedule in \((10^{-4},0.02)\). On THUMOS14 the method reports average mAP \(70.8\%\), with mAP \(84.9\) at \(0.3\), \(81.5\) at \(0.4\), \(76.5\) at \(0.5\), \(63.0\) at \(0.6\), and \(48.0\) at \(0.7\), surpassing TriDet at \(69.3\%\) average mAP. On ActivityNet-1.3 it reports average mAP \(38.3\%\), compared with ActionFormer at \(36.6\%\) and DiffTAD at \(36.1\%\). The ablation table shows \(66.0\) average mAP for standard diffusion with a Ho-et-al.-style architecture, \(69.0\) for discrete diffusion with that architecture, \(68.1\) for standard diffusion with the Row-Column block, and \(70.8\) for the full combination. Inference speed is \(0.113\) s per clip on a V100 GPU. ## 5. ActionDiff as video action differencing In the VidDiff line, “ActionDiff” names a task rather than a diffusion architecture: given an action description \(s\) and two untrimmed videos \(v_A\) and \(v_B\) of the same action, the goal is to identify subtle differences between the performances [2503.07860]. In the open-set setting, the model must output up to \(N_{\mathrm{diff}}\) difference statements \(\mathcal D=\{(d_i,p_i)\}\), where \(d_i\) is a natural-language description and \(p_i\in\{A,B\}\) indicates which video exhibits the attribute more strongly. In the closed-set setting, the model is given \(k\) candidate differences and predicts a label vector \(\mathbf p=(p_1,\dots,p_k)\). The paper defines closed-set accuracy as \[ \mathrm{Acc}=\frac1k\sum_{j=1}^k \mathbf 1[p_j=p_j^\star], \] and open-set recall@\(N_{\mathrm{diff}}\) by matching predicted descriptions to ground-truth differences using soft string matching via an LLM. VidDiffBench contains 549 video pairs from five domains—Fitness, Ballsports, Surgery, Music, and Diving—with 4,469 human-written fine-grained differences and 2,075 timestamp annotations. The annotation pipeline first defines a taxonomy of 10–30 skill-relevant, visually testable difference strings for each action, then labels each pair as A/B/C for each taxonomy entry, with \(25\%\) rescored for quality control and \(2\%\) A↔B disagreement, and finally associates each difference with one or more key-points. The reported error analysis identifies two major bottlenecks for large multimodal models: sub-action localization and fine-grained visual comparison. The proposed VidDiff method decomposes the task into three stages. Stage 1 uses GPT-4o-2024-08-06 to propose candidate differences and associated query strings. Stage 2 decomposes the action into ordered sub-actions, embeds frames and retrieval strings with CLIP-ViT-bigG-14, computes similarities \[ s_{t,m}=\max_{k=1..K}\langle \phi(f_t),\psi(r_{m,k})\rangle, \] and solves a Viterbi-style dynamic program \[ \{\hat t_m\}_{m=1}^M=\arg\max_{t_1<\cdots