Video Step Grounding (VSG)

• Video Step Grounding (VSG) is the task of aligning video segments with specific procedural steps using online, training-free inference mechanisms.
• Recent methods leverage large multimodal models alongside Bayesian temporal filtering to capture step dependencies and improve recognition accuracy.
• Empirical evaluations on benchmarks like HT-Step, CrossTask, and Ego4D demonstrate VSG’s potential for real-time applications in AR, robotics, and human–machine collaboration.

Video Step Grounding (VSG) is the task of identifying, for each segment of a video and a given list of procedural steps, which step is being performed at that moment. Unlike conventional video grounding, which often focuses on locating a single action or moment corresponding to a query within untrimmed video, VSG generalizes this to handle a sequence of steps and requires robust step detection—ideally in both offline and online (streaming) settings—without relying on costly labeled data or full-video processing. VSG is foundational to applications in real-time human–machine collaboration, AR/XR guidance, industrial robotics, and assistive systems.

1. Problem Definition and Scope

Video Step Grounding (VSG) takes as inputs:

• A set of ordered, natural language step descriptions denoted $A = \{a_1, a_2, \ldots, a_K\}$,
• A video segmented into temporal units $S = \{S_1, S_2, \ldots, S_T\}$,

and aims to assign to each segment $S_t$ the step $a_i$ being executed, or indicate that no given step is currently being performed. The core challenge is accurately detecting and mapping each observed step, often in highly multimodal and ambiguous environments, under limited or even zero training supervision.

Traditional methods approach this task with supervised learning frameworks, often offline, requiring all video data to be seen ahead-of-time. In contrast, VSG as formalized in the most recent literature emphasizes both:

• online inference (streaming, step-wise prediction as video is ingested),
• training-free or zero-shot operation (relying solely on pre-trained vision-LLMs without further task-specific tuning) (Zanella et al., 19 Oct 2025).

Such requirements stem from practical deployment considerations, including cost of annotation, real-time feedback needs, and the desire for broad task generalization.

2. Challenges in Conventional Approaches

Two primary limitations have hampered previous approaches to VSG:

1. Dependence on Labeled Data.
   • Standard models require extensive step-level annotations or narrations, collected for each domain or procedure, which is resource-intensive and limits generalization (Zanella et al., 19 Oct 2025).
2. Offline, Non-Streaming Techniques.
   • Most existing methods require access to the entire video sequence to align steps with actions, impeding their ability to support real-time, incremental predictions or to adapt to long untrimmed videos.

These shortcomings preclude feasible deployment in contexts demanding immediate step localization or flexible adaptation to new procedural domains.

3. Training-Free Online VSG via Large Multimodal Models (LMMs)

A major advance is the demonstration that recent LMMs (large-scale vision–language and multimodal transformers) possess sufficient zero-shot capabilities for high-performing, training-free VSG (Zanella et al., 19 Oct 2025).

Operational paradigm:

• For each incoming segment $S_t$, a multi-choice QA prompt is constructed, including the given step descriptions and an explicit “none” option.
• The LMM is presented with the current segment and prompt ($S_t, \pi_{\text{VSG}}$), and outputs a probability distribution over all candidate steps, i.e.,

$f_{\text{LMM}}(S_t, \pi_{\text{VSG}})[i]$

where $i$ indexes the candidate steps.

Crucially, this operation is online: only segments $S_1,\ldots,S_t$ are visible at timestep $t$, and no full-video context is assumed. This reduces latency and memory requirements, enabling real-time assistance or guidance.

Experimental results show that such prompt-based LMM step assignment, without any task-specific adaptation, matches or outperforms state-of-the-art training-based, offline methods across standard VSG benchmarks (HT-Step, CrossTask, Ego4D Goal-Step).

4. Bayesian Grounding with Large Multimodal Models (BaGLM)

Recognizing that single-segment inference with LMMs overlooks sequential structure and cross-segment context, the BaGLM framework augments LMM predictions using Bayesian temporal filtering (Zanella et al., 19 Oct 2025). This structured approach yields marked performance gains.

Core steps:

1. Transition Modeling:
   • Step dependencies are encoded via a matrix $D$, with entries $D_{ij}$ representing the probability that step $a_j$ is a prerequisite for $a_i$. $D$ is computed via a LLM, capturing task structure.
   • A transition matrix $T$ initializes allowable transitions ($T[i,j] > 0$ only if step $a_j$ can follow $a_i$), typically including diagonal self-transitions.
2. Step Progress Tracking:
   • For each segment, the LMM is prompted to estimate the progress ($\in [0,1]$) of each step.
3. Bayesian Filtering Update:
   • At each timestep $t$, compute “step readiness” ($r_t[i]$) and “step validity” ($v_t[i]$):

   $$r_t[i] = \frac{\sum_{j} D_{ij} \cdot \max_{\tau < t} \text{progress}_\tau[j]}{\sum_j D_{ij}}$$

   $$v_t[i] = \frac{\sum_{j} D_{ji} \cdot (1 - \max_{\tau < t} \text{progress}_\tau[j])}{\sum_j D_{ji}}$$

• Update $T$ to $\tilde{T}_t$ using readiness/validity as multiplicative factors and normalize rows:

  $$\tilde{T}_t[i,j] = \frac{T[i,j] \cdot r_t[j] \cdot v_t[j]}{\sum_k T[i,k] \cdot r_t[k] \cdot v_t[k]}$$

• The predicted belief (prior) is a step-wise propagation:

  $$\text{predict}_t(a_i) = \sum_{j} \tilde{T}_t[j,i] \cdot \text{bel}_{t-1}(a_j)$$

• Fuse with LMM segment likelihoods via Bayes’ rule:

  $$\text{bel}_t(a_i) = \frac{f_{\text{LMM}}(S_t, \pi_{\text{VSG}})[i] \cdot \text{predict}_t(a_i)}{Z}$$

  where $Z$ is a normalizer.

This formulation enables VSG systems to integrate both step-wise dependencies and step progress estimates to improve identification reliability, leading to superior online, real-time performance.

5. Empirical Evaluation

On three VSG datasets—HT-Step, CrossTask, Ego4D Goal-Step—BaGLM with LMM backbones achieves or surpasses the state-of-the-art, including both offline and training-based methods.

• On HT-Step, BaGLM exceeds NaSVA by 4.3% recall@1.
• On CrossTask and Ego4D Goal-Step, the improvement over baselines approaches or exceeds 13–14%, demonstrating that zero-shot, streaming models with Bayesian temporal reasoning are not merely competitive but superior for VSG (Zanella et al., 19 Oct 2025).
• Notably, these results are achieved with no task-specific fine-tuning and in an online setting.

6. Implications and Applications

The training-free, online paradigm for Video Step Grounding has several impactful implications:

• Scalability: By not requiring per-task or per-domain labeled data, VSG can be deployed on new instructional domains with minimal cost.
• Real-Time Guidance: The online formulation supports applications in AR, real-time assembly, surgical assistance, and industrial monitoring.
• Generalization: Leveraging strong pretrained LMMs and temporal priors ensures flexibility and robustness to new, unseen tasks or step formulations.
• Reduced Annotation Overhead: Drastic reduction in human annotation costs enables broad dataset expansion.

This suggests that as LMMs and their temporal reasoning capacities improve, the effectiveness and scope of training-free, online VSG will continue to increase, potentially making such approaches dominant in practical applications.

7. Future Directions

Potential lines of advancement for online, training-free VSG include:

• Enhanced dependency modeling via more expressive LLM prompting or graph neural structures capturing complex procedural logic and loops.
• More accurate progress estimation, possibly using finer temporal segmentation or multimodal sensor fusion (e.g., combining video with audio/narration cues).
• Extension to spatial–temporal VSG (detecting both step temporal boundaries and their associated regions or objects in video frames).
• Benchmarking and deployment in domains requiring robust, reliable step grounding under noisy, ambiguous, or incomplete visual data streams.

The current trajectory of VSG research, as encapsulated by (Zanella et al., 19 Oct 2025), indicates rapid unification of zero-shot, model-free, and streaming techniques leveraged by LMMs and Bayesian temporal reasoning to facilitate scalable, interpretable, and real-time procedural understanding in video.