ActionDiff: Diffusion Models in Action Analysis

• ActionDiff is a multifaceted term referring to various diffusion-based frameworks addressing video procedure planning, action recognition, temporal detection, and fine-grained action differencing.
• In procedure planning, ActionDiff uses conditional diffusion with an action-aware noise mask and U-Net denoising to generate ordered action sequences with measurable performance gains.
• Beyond video tasks, ActionDiff extends to categorical change-action models in abstract mathematics, showcasing its adaptability across empirical and theoretical domains.

Searching arXiv for papers using the term "ActionDiff" to ground the article and disambiguate the topic. ActionDiff is an overloaded term in recent arXiv literature rather than a single canonical model. It has been used for a conditional diffusion model for procedure planning in instructional videos, a diffusion-feature framework for action recognition under severe domain shift, a discrete image-diffusion formulation of temporal action detection, a fine-grained action differencing task over video pairs, and, in a separate categorical line of work, an exposition label for change actions and change-action models (Shi et al., 2024, Guimaraes et al., 10 Sep 2025, Foo et al., 2024, Burgess et al., 10 Mar 2025, Alvarez-Picallo et al., 2019). Across these usages, the shared motif is not a fixed architecture but the use of diffusion, denoising, or difference structures to model uncertainty, compositionality, or subtle variation in action-related objects.

1. Terminological scope and disambiguation

The label has been attached to technically distinct objects with different state spaces, objectives, and evaluation regimes.

Usage	Problem setting	Paper
ActionDiffusion / ActionDiff	Procedure planning in instructional videos	(Shi et al., 2024)
ActionDiff	Action recognition across unseen domains	(Guimaraes et al., 10 Sep 2025)
ActionDiff / ADI-Diff	Temporal action detection via image diffusion	(Foo et al., 2024)
ActionDiff	Fine-grained differencing between two videos of the same action	(Burgess et al., 10 Mar 2025)
ActionDiff	Change actions and change-action models in category theory	(Alvarez-Picallo et al., 2019)

These names are not interchangeable. One line predicts an action plan from start and goal observations; another uses a frozen Stable Video Diffusion backbone as a feature extractor for classification; another denoises structured “AD images” whose rows are probability distributions over classes or boundaries; another formalizes a benchmark in which a model must describe how two executions of the same action differ; and another develops a semantics of generalized differentiation in cartesian categories. A precise reading therefore requires identifying the paper-specific definition of the underlying action object: discrete action classes, action sequences, temporal proposals, action chunks, or categorical changes.

2. ActionDiffusion for procedure planning

In the instructional-video setting, ActionDiffusion is a conditional diffusion model for procedure planning that predicts an entire sequence of intermediate actions $a_1,\dots,a_T$ from a start frame feature $o_s$ and a goal frame feature $o_g$ (Shi et al., 2024). The clean variable $x_0$ concatenates the task-class one-hot $c$, the one-hot action steps $a_1,\dots,a_T$, and the visual endpoints $o_s,o_g$. The model follows a DDPM-style forward process

$$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$

with closed-form

$$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$

and a cosine schedule

$$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$

Its distinctive mechanism is the action-aware noise mask. Each one-hot action $o_s$0 is embedded as $o_s$1, normalized by $o_s$2, and accumulated over time: $o_s$3 The forward process then replaces the isotropic covariance by $o_s$4, so the noised tensor explicitly carries information about action order and identity. The denoiser is a U-Net with timestep embeddings and multi-head self-attention at the bottleneck, which allows cross-step correlations to be recovered during denoising. Training is joint: a task classifier minimizes $o_s$5, the diffusion module minimizes $o_s$6, and the total loss is

$o_s$7

The reported hyperparameters are diffusion timesteps $o_s$8, cosine schedule with $o_s$9, action embedding dimension $o_g$0, time-embedding dimension $o_g$1, a U-Net with 4 down/up-sampling levels, channel base $o_g$2, 8 attention heads at bottleneck, Adam with learning rate $o_g$3, batch size $o_g$4, and 50 K training steps. On CrossTask with $o_g$5, ActionDiff reports $o_g$6 success rate, $o_g$7 mAcc, and $o_g$8 mSIoU, compared with PDPP at $o_g$9, $x_0$0, and $x_0$1. On COIN, it reports $x_0$2 success rate, $x_0$3 mAcc, and $x_0$4 mSIoU; on NIV, $x_0$5, $x_0$6, and $x_0$7. The paper states that it outperforms previous state of the art on all metrics on CrossTask and NIV and all metrics except accuracy on COIN. Ablations on COIN show that replacing the accumulated mask with a single-add variant lowers success rate from $x_0$8 to $x_0$9, and removing self-attention lowers it from $c$0 to $c$1.

3. ActionDiff for action recognition across untrained domains

A different ActionDiff denotes a video-level action-recognition framework built on a frozen Stable Video Diffusion model, specifically SVD-XT, with a lightweight transformer head trained for downstream classification (Guimaraes et al., 10 Sep 2025). Here diffusion is not used to generate action sequences. Instead, it is used as a source of semantically enriched intermediate representations. Given frame $c$2, the latent $c$3 follows a standard latent diffusion formulation, and the denoiser $c$4 is conditioned on a CLIP embedding of the middle frame. The framework extracts features from an intermediate U-Net activation $c$5 at layer $c$6 and timestep $c$7 out of $c$8, chosen to emphasize semantic information over pixel-level detail. The pooled feature is

$c$9

These frame features are aggregated by appending a learned class token, projecting to a common dimension, adding 1D positional encodings, and processing the token sequence with a small transformer encoder. Only the action-classification head is trained; the diffusion backbone remains frozen. For single-label tasks the head uses cross-entropy, while multi-label settings use sigmoid with binary cross-entropy or Focal Loss. MixUp is also applied.

The evaluation targets domain shift rather than generative fidelity. On Animal Kingdom, ActionDiff reports $a_1,\dots,a_T$0 mAP on the full dataset, exceeding MAMBA-MSQNet at $a_1,\dots,a_T$1 mAP, V-JEPA at $a_1,\dots,a_T$2, and SDv2 at $a_1,\dots,a_T$3. On unseen-species accuracy it reports $a_1,\dots,a_T$4, compared with MSQNet’s $a_1,\dots,a_T$5 and V-JEPA’s $a_1,\dots,a_T$6. On Charades-Ego, the 1st $a_1,\dots,a_T$7 1st setting yields $a_1,\dots,a_T$8 mAP and the 3rd $a_1,\dots,a_T$9 1st setting $o_s,o_g$0 mAP. On cross-context UCF $o_s,o_g$1 HMDB, it reports $o_s,o_g$2 accuracy, and on HMDB $o_s,o_g$3 UCF, $o_s,o_g$4. The ablations indicate that replacing the transformer with a linear $o_s,o_g$5 MLP head lowers Animal Kingdom performance to $o_s,o_g$6 mAP, an MLP head lowers it to $o_s,o_g$7 mAP, unconditional features reduce mAP to $o_s,o_g$8, and removing Focal Loss or MixUp hurts by approximately $o_s,o_g$9. The paper’s layer–timestep grid search further reports that later timesteps perform best in-domain, whereas earlier timesteps yield better out-of-domain generalization.

4. ActionDiff as action detection via an image diffusion process

A third usage corresponds to ADI-Diff, where action detection is reformulated as the generation of three structured images: an action-class image $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$0, a starting-point image $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$1, and an ending-point image $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$2 (Foo et al., 2024). Each row is a discrete probability distribution; in the ground truth, rows are one-hot.

The diffusion process is discrete rather than Gaussian. For a row $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$3, the forward step is

$$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$4

where $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$5. With

$$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$6

the process converges in expectation to the uniform distribution as $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$7. Reverse denoising is performed by a network $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$8 conditioned on frozen video features $$q(x_n\mid x_{n-1})=\mathcal N(x_n;\sqrt{1-\beta_n}\,x_{n-1},\beta_n I),$$9 and a sinusoidal step embedding $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$0: $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$1 Training uses a simplified MSE objective supervising each reverse step against the forward-chain target.

The denoiser is a Row-Column Transformer. Column-wise encoding treats each column as a token and applies multi-head self-attention across columns; row-wise encoding applies temporal convolution across rows, then self-attention across time. This design is tailored to the asymmetric structure of the AD images, where rows correspond to frames and columns to classes or boundary indicators. Inference thresholds the first column of $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$2 and $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$3 at $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$4, averages frame indices within clusters to obtain boundaries, forms proposals by pairing starts with later ends, averages class scores over each proposal interval, and applies Soft-NMS.

The reported setup uses I3D features on THUMOS14, R(2+1)D on ActivityNet-1.3, diffusion steps $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$5, block stacks $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$6, sample-per-step $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$7, and a linearly increasing noise schedule in $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$8. On THUMOS14 the method reports average mAP $$x_n=\sqrt{\bar a_n}\,x_0+\sqrt{1-\bar a_n}\,\epsilon,$$9, with mAP $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$0 at $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$1, $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$2 at $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$3, $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$4 at $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$5, $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$6 at $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$7, and $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$8 at $$\bar a_n=f(n)/f(0),\quad f(n)=\cos^2(((n/N+\tau)/(1+\tau))\cdot \pi/2).$$9, surpassing TriDet at $o_s$00 average mAP. On ActivityNet-1.3 it reports average mAP $o_s$01, compared with ActionFormer at $o_s$02 and DiffTAD at $o_s$03. The ablation table shows $o_s$04 average mAP for standard diffusion with a Ho-et-al.-style architecture, $o_s$05 for discrete diffusion with that architecture, $o_s$06 for standard diffusion with the Row-Column block, and $o_s$07 for the full combination. Inference speed is $o_s$08 s per clip on a V100 GPU.

5. ActionDiff as video action differencing

In the VidDiff line, “ActionDiff” names a task rather than a diffusion architecture: given an action description $o_s$09 and two untrimmed videos $o_s$10 and $o_s$11 of the same action, the goal is to identify subtle differences between the performances (Burgess et al., 10 Mar 2025). In the open-set setting, the model must output up to $o_s$12 difference statements $o_s$13, where $o_s$14 is a natural-language description and $o_s$15 indicates which video exhibits the attribute more strongly. In the closed-set setting, the model is given $o_s$16 candidate differences and predicts a label vector $o_s$17. The paper defines closed-set accuracy as

$o_s$18

and open-set recall@$o_s$19 by matching predicted descriptions to ground-truth differences using soft string matching via an LLM.

VidDiffBench contains 549 video pairs from five domains—Fitness, Ballsports, Surgery, Music, and Diving—with 4,469 human-written fine-grained differences and 2,075 timestamp annotations. The annotation pipeline first defines a taxonomy of 10–30 skill-relevant, visually testable difference strings for each action, then labels each pair as A/B/C for each taxonomy entry, with $o_s$20 rescored for quality control and $o_s$21 A↔B disagreement, and finally associates each difference with one or more key-points. The reported error analysis identifies two major bottlenecks for large multimodal models: sub-action localization and fine-grained visual comparison.

The proposed VidDiff method decomposes the task into three stages. Stage 1 uses GPT-4o-2024-08-06 to propose candidate differences and associated query strings. Stage 2 decomposes the action into ordered sub-actions, embeds frames and retrieval strings with CLIP-ViT-bigG-14, computes similarities

$o_s$22

and solves a Viterbi-style dynamic program

$o_s$23

to localize key frames. Stage 3 poses a multiple-choice question to GPT-4o over the localized frames to determine whether video A, video B, or neither exhibits the queried difference more strongly.

On closed-set evaluation, VidDiff reports average accuracy $o_s$24, compared with $o_s$25 for GPT-4o, $o_s$26 for Gemini-1.5-Pro, $o_s$27 for Claude-3.5 Sonnet, $o_s$28 for LLaVA-Video, and $o_s$29 for Qwen2-VL-7B. On open-set recall@$o_s$30, it reports average $o_s$31, compared with $o_s$32 for GPT-4o, $o_s$33 for Gemini, $o_s$34 for Claude-3.5 Sonnet, $o_s$35 for LLaVA-Video, and $o_s$36 for Qwen2-VL. Ablations show that even with ground-truth frames, frame-differencing accuracy falls from $o_s$37 on easy to $o_s$38 on medium and $o_s$39 on hard subsets, and that Viterbi-based localization improves closed-set easy performance from $o_s$40 without Viterbi to $o_s$41.

6. Relation to adjacent diffusion-based action modeling

The broader action-diffusion landscape helps delimit what the various ActionDiff usages do and do not cover. In long-term action anticipation, DiffAnt models future-action embeddings $o_s$42 with a DDPM in latent space, conditions reverse denoising on encoded past video features through cross-attention, and uses DDIM sampling with typically 100 inference steps; it reports strong gains for far-future prediction, including $o_s$43 mean over class accuracy on Breakfast at $o_s$44 and $o_s$45 mAP on EGTEA Gaze+ (Zhong et al., 2023). In temporal action detection, DiffTAD treats proposal boundaries $o_s$46 as denoised temporal proposals in a Transformer decoder and reports $o_s$47–$o_s$48 average mAP on THUMOS with 5–10 denoising steps, while EffiDiffAct adapts diffusion to action-label sequences for temporal action segmentation, introduces a Temporal Dilation Perception encoder and an adaptive skip strategy, and reports $o_s$49 average score on 50Salads at 25 iterations with $o_s$50 s inference time per video (Nag et al., 2023, Wang et al., 2024).

In robotics and continuous control, Self-Guided Action Diffusion modifies each reverse diffusion step by introducing a soft prior toward the previous action chunk, producing guided posterior parameters

$o_s$51

with $o_s$52 per-step complexity under diagonal covariances; on Robomimic benchmarks it reports roughly $o_s$53–$o_s$54 single-sample success versus roughly $o_s$55–$o_s$56 for random sampling and vanilla bidirectional baselines, and on PushT it achieves roughly $o_s$57 success with budget $o_s$58 (Malhotra et al., 17 Aug 2025). PoseDiff uses a conditional diffusion model to map sparse world-model keyframes into dense action segments between frame pairs and stitch them with overlap averaging; on Libero-Object it reports success rates of $o_s$59, $o_s$60, $o_s$61, $o_s$62, and $o_s$63 on Soup, Cheese, Salad, Ketchup, and Tomato respectively (Zhang et al., 29 Sep 2025). DiffAIL inserts a diffusion-based density estimator into adversarial imitation learning by defining $o_s$64, yielding a surrogate reward $o_s$65 based on diffusion loss over state-action pairs (Wang et al., 2023). DivDiff uses a conditional DDPM, DCT-based motion encoding, and a diversified reinforcement sampling function for human motion prediction, reporting on Human3.6M $o_s$66, $o_s$67, and $o_s$68 (Yu et al., 2024).

This suggests that “action diffusion” is best understood as a family of formulations whose principal degree of freedom is the representation being diffused: class labels, proposal boundaries, latent action embeddings, continuous action chunks, state-action pairs, or future motion trajectories.

7. ActionDiff in category theory: change actions and generalized differentiation

Outside machine learning, “ActionDiff” is also used as an exposition label for the theory of change actions and change-action models, a categorical framework for generalized differentiation (Alvarez-Picallo et al., 2019). A change action $o_s$69 consists of an underlying object $o_s$70, a change-space $o_s$71, a commutative monoid $o_s$72, and an action map

$o_s$73

satisfying $o_s$74 and $o_s$75. In a cartesian category $o_s$76, an internal change action is an object

$o_s$77

and a differential map $o_s$78 is a pair $o_s$79 with $o_s$80 satisfying the derivative condition

$o_s$81

the zero rule $o_s$82, and regularity

$o_s$83

Composition is defined by the chain rule

$o_s$84

The framework supports higher-order derivatives by iteration, leading to an $o_s$85-change-action construction that plays a role analogous to the Faà di Bruno construction. The exposition identifies examples from generalized cartesian differential categories, discrete finite-difference calculus on groups, and polynomials over a commutative Kleene algebra. For groups, every function $o_s$86 acquires a discrete derivative

$o_s$87

and for $o_s$88 this recovers the forward-difference operator. The structural results summarized in the exposition include chain and product rules, an equivalence between change actions in $o_s$89 and preorders, a fully faithful 2-functor into $o_s$90, and a final-coalgebra $o_s$91-model. This is a wholly different use of the name from diffusion-based action modeling, but it explains why “ActionDiff” can appear in arXiv contexts that concern differentiation rather than video or control.