Action Matching: Variational Dynamics Learning

• Action Matching is a variational framework that recovers continuous-time dynamics from independent marginal samples without requiring paired trajectories.
• It optimizes an action function whose gradient yields a minimal-kinetic-energy vector field that obeys the continuity equation to match observed distributions.
• The framework extends to stochastic and unbalanced settings, linking optimal transport principles with simulation-free training for applications in biology, physics, and robotics.

Action Matching (AM) most specifically denotes a variational framework for learning continuous-time dynamics from independent samples of temporal marginals, rather than from paired trajectories. In that formulation, AM learns a scalar action $s_t(x)$ whose gradient field $\nabla s_t(x)$ transports particles along an observed path of distributions $q_t$, thereby yielding a simulable process that matches the measured marginals over time (Neklyudov et al., 2022). In adjacent literatures, the same phrase has also been used for category-independent matching of action segments across videos, for matching-based few-shot action recognition, and for action-flow formulations in robotics, so the term is methodologically rich but terminologically non-uniform (Fernando et al., 2016).

1. Canonical formulation: dynamics from unpaired temporal snapshots

In the formulation introduced in "Action Matching: Learning Stochastic Dynamics from Samples" (Neklyudov et al., 2022), the data are samples $x_t^j \sim q_t(x)$ from a time-indexed family of distributions

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$

with no cross-time correspondences between samples. This setting is motivated by domains in which trajectories are unavailable by design, including single-cell biology, quantum systems, physics, chemistry, and generative modeling.

The learned process is an ODE

$\frac{d}{dt}x(t) = v_t(x(t)), \qquad x(t=0)=x,$

with induced density $q_t$. AM restricts the target field to a gradient form

$v_t^*(x) = \nabla s_t^*(x),$

where $s_t^*(x)$ is the action. The corresponding density evolution satisfies the continuity equation

$$\partial_t q_t = -\nabla \cdot \big(q_t \nabla s_t^*(x)\big).$$

The key existence result states that for an absolutely continuous distributional path $\nabla s_t(x)$0, there exists a unique action $\nabla s_t(x)$1, up to an additive constant, whose gradient traces that path through the continuity equation (Neklyudov et al., 2022). This makes AM a method for recovering a canonical dynamics from marginal information alone. The paper further emphasizes that, although the learned simulation-time model is deterministic, it can represent the marginal evolution of a broad class of processes, including processes whose true microscopic dynamics are stochastic.

2. Variational objective and the tractable AM loss

The ideal training target is the intractable action gap

$\nabla s_t(x)$2

Because $\nabla s_t(x)$3 is unknown, AM derives a tractable surrogate. The central decomposition is

$\nabla s_t(x)$4

where $\nabla s_t(x)$5 is a constant independent of $\nabla s_t(x)$6, and

$\nabla s_t(x)$7

This objective is obtained from a constrained kinetic-energy minimization problem: $\nabla s_t(x)$8 The Euler-Lagrange optimality condition yields

$\nabla s_t(x)$9

so the learned field is the minimal-kinetic-energy field compatible with the observed path. The corresponding constant

$q_t$0

is the kinetic energy of the true path (Neklyudov et al., 2022).

A central practical property is that AM is simulation-free during training. The loss is computed directly from samples from $q_t$1, and the paper explicitly states that the method does not require back-propagation through differential equations or optimal transport solvers (Neklyudov et al., 2022). In Monte Carlo form, training samples $q_t$2, $q_t$3, times $q_t$4, and $q_t$5, then evaluates boundary terms, the gradient norm, and the time derivative of $q_t$6.

3. Extensions, stochastic variants, and optimal-transport structure

The same paper extends the framework in several directions (Neklyudov et al., 2022). For stochastic dynamics

$q_t$7

the density follows the Fokker–Planck equation

$q_t$8

and the corresponding Entropic Action Matching objective becomes

$q_t$9

For dynamics with creation and destruction of probability mass, the paper introduces Unbalanced Action Matching, using

$x_t^j \sim q_t(x)$0

with transport and growth tied to the same scalar action: $x_t^j \sim q_t(x)$1 Its tractable loss is

$x_t^j \sim q_t(x)$2

The same framework is further generalized to strictly convex kinetic costs through a convex conjugate $x_t^j \sim q_t(x)$3, producing a generalized objective

$x_t^j \sim q_t(x)$4

A later theoretical note sharpened the connection between AM and quadratic optimal transport. It distinguishes Flow Matching, which learns a vector field for a manually chosen interpolation between $x_t^j \sim q_t(x)$5 and $x_t^j \sim q_t(x)$6, from Action Matching, which learns a vector field for an entire prescribed path $x_t^j \sim q_t(x)$7. Under a restriction to Brenier-type optimal vector fields, the note proves that the AM objective and the dual quadratic OT objective match each other up to an additive constant, so minimizing restricted AM recovers the Brenier map independently of the prescribed intermediate path (Kornilov et al., 31 Oct 2025). This result formalizes the OT intuition already present in the original AM construction.

4. Inference, likelihoods, and empirical use domains

Once trained, AM generates trajectories by integrating the learned ODE

$x_t^j \sim q_t(x)$8

This allows sample propagation forward or backward in time and makes AM a genuine dynamics model rather than a static interpolator (Neklyudov et al., 2022).

When $x_t^j \sim q_t(x)$9 is known, the same formulation supports CNF-style likelihood evaluation: $$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$0 where $$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$1 is the Laplacian of the action. The paper also proves a Wasserstein error bound of the form

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$2

linking small action gap to small marginal mismatch (Neklyudov et al., 2022).

Empirically, the original work reports competitive performance across biology, physics, and generative modeling, and on synthetic cellular differentiation data the entropic variant outperforms MIOFlow in Wasserstein-2 and MMD (Neklyudov et al., 2022). The scope of these experiments is notable because the method is expressly designed for domains in which trajectories are unavailable or physically unobservable.

5. Independent uses of “Action Matching” in vision and robotics

An earlier and independent usage defined unsupervised human action detection by action matching as a task on two long videos: detect all pairs of temporal segments that correspond to the same human action, without category labels or supervised detectors. The method uses rank pooling to encode sliding windows, constructs a Gram matrix

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$3

and extracts temporally consistent diagonal runs satisfying

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$4

followed by pairwise NMS. On MPII Cooking it reports 21.6% precision and 11.7% recall over 946 long video pairs, and on THUMOS it reports 18.4% precision and 25.1% recall over 5094 ground-truth action segment pairs (Fernando et al., 2016).

In few-shot action recognition, later work treats action matching as direct support-query video matching rather than as classifier fitting. Under a common R(2+1)D spatio-temporal backbone, that paper shows that simple non-temporal matching functions become much stronger than earlier literature had suggested, and introduces Chamfer++, a parameter-free matching rule based on symmetric Chamfer aggregation over clip or tuple descriptors. The central claim is that, once temporal information is already encoded in clip features, temporal alignment in the matching stage is often much less necessary (Bertrand et al., 2023).

In spatio-temporal action detection, a 2025 paper operationalizes action matching as person query matching across frames. Its Query Matching Module learns an embedding space in which DETR queries corresponding to the same person are close across frames, thereby constructing action tubes without IoU-based linking. The paper explicitly notes that this is primarily a person association mechanism for action tube generation, rather than direct matching of action labels or action embeddings (Omi et al., 17 Mar 2025).

In robotics, "StreamingVLA" uses action flow matching to replace chunk-wise denoising with a continuous flow over an action/state trajectory. The target field is

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$5

the training loss is a conditional flow-matching objective over $$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$6, and inference converts velocity increments directly into executable actions

$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$7

The paper argues that this removes reliance on action chunking and enables streaming overlap between action generation and execution, reporting 2.4$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$8 latency speedup and 6.5$$q_t \in \mathcal{P}_2(\mathcal{X}), \qquad t \in [0,1],$$9 halting reduction in the full system (Shi et al., 30 Mar 2026). A related robotics paper, "Action-to-Action Flow Matching," is conceptually close in that it learns a transport from previous actions to future actions in latent space, but it does not define a separate method called AM (Jia et al., 7 Feb 2026).

6. Terminological non-uniformity

A persistent source of confusion is that not every paper using “AM” refers to the 2022 dynamics-learning objective, or even to action matching at all. Recent arXiv usage is heterogeneous.

Usage of “AM”	Core meaning	Representative paper
Action Matching	Learn $\frac{d}{dt}x(t) = v_t(x(t)), \qquad x(t=0)=x,$0 from marginal samples	(Neklyudov et al., 2022)
Action matching in video detection	Match action segments across two long videos	(Fernando et al., 2016)
Action flow matching	Learn streaming action/state trajectory fields	(Shi et al., 30 Mar 2026)
Attention Map (AM) Flow	Motion-relevant attention-map differences for video recognition	(Agrawal et al., 2024)
Adjoint Matching (AM)	SOC-based reward fine-tuning for diffusion models	(Shin et al., 12 May 2026)

This heterogeneity matters technically. "AM Flow" uses AM to mean Attention Map, not Action Matching (Agrawal et al., 2024). "Efficient Adjoint Matching" uses AM to mean Adjoint Matching, a stochastic-optimal-control formulation for reward fine-tuning diffusion models, later reformulated with a linear base drift and closed-form adjoint to improve efficiency (Shin et al., 12 May 2026). In robotics, "PAMAE" is layered on top of a standard flow-matching VLA policy and explicitly does not introduce a new flow-matching or Action Matching objective; it changes the action module and routing strategy while keeping the underlying flow-matching loss unchanged (Yang et al., 25 Jun 2026).

The most stable technical usage of Action Matching therefore remains the 2022 variational framework for learning continuous dynamics from marginal snapshots, together with later OT-oriented refinements. Elsewhere, the phrase is best read locally, with close attention to whether it denotes a dynamics-learning objective, a video-segment matching task, a support-query similarity rule, or merely an unrelated abbreviation.