Multi-Temporal-Scale Video Augmentation

• Multi-temporal-scale video sampling augmentation is a strategy that constructs video views using varied clips, frame rates, and perturbation schedules to mitigate static background bias.
• It leverages methods such as local-global contrast, fixed-rate sampling, and adaptive segment selection to capture both short-term motion and long-range temporal structure.
• The approach enhances tasks like action recognition, temporal grounding, and long-form video processing by improving motion sensitivity and computational efficiency.

Multi-temporal-scale video sampling augmentation refers to a family of video view-construction and data augmentation strategies in which training or inference does not rely on a single fixed temporal view of a video, but instead uses clips, segments, subsequences, or perturbation schedules with different temporal spans, strides, or temporal receptive fields. In the cited literature, the topic appears in self-supervised video representation learning, supervised action recognition, long-form VideoQA, temporal grounding, promptable video object segmentation and tracking, and efficient MLLM inference. These works suggest a common objective: to reduce bias toward static background cues, improve robustness to motion-speed variation, recover long-range temporal structure, and make long-video processing more computationally tractable (Qian et al., 2022, Kim et al., 2022, Li et al., 2021, Gao et al., 2022, Wang et al., 22 Nov 2025).

1. Scope and conceptual variants

The literature does not use a single canonical formulation. Instead, several closely related mechanisms recur. Some methods sample local clips and global videos jointly, some alter the frame sampling rate or construct progressive speed-up views, some vary augmentation magnitude over time, and some perform adaptive selection of segments or subsequences conditioned on the downstream task or query. A further branch applies multi-scale temporal sampling at test time or within domain-specific pipelines such as surgical VOST (Qian et al., 2022, Li et al., 2021, Kim et al., 2022, Gao et al., 2022, Xu et al., 7 Aug 2025, Wang et al., 22 Nov 2025).

Formulation	Temporal mechanism	Representative source
Local-global contrastive sampling	Partition a video into intervals and sample local clips plus a longer global video	(Qian et al., 2022)
Temporal scale and projection	Fixed-rate and progressive speed-up sampling	(Li et al., 2021)
Temporally dynamic augmentation magnitude	Fourier-basis magnitude schedule across frames	(Kim et al., 2022)
Floating temporal perturbation	Frame-varying box locations or mixing ratios	(Gorpincenko et al., 2022)
Adaptive segment selection	Question-conditioned top-$k$ segment and region selection	(Gao et al., 2022)
Test-time temporal sampling	Multiple short subsequences packed in one forward pass	(Wang et al., 22 Nov 2025)
Multi-scale stride sampling for VOST	Training sequences generated with different sampling strides	(Xu et al., 7 Aug 2025)

A central distinction is between sampling the video itself and sampling the augmentation policy over time. The former changes which frames or subclips are presented to the model; the latter keeps the clip but changes augmentation parameters frame by frame. The literature also separates explicit multi-temporal-scale augmentation from methods that are merely temporally consistent. S3Aug, for example, applies category sampling identically to all frames and explicitly does not perform augmentation at multiple temporal scales (Sugiura et al., 2023).

2. Core formulations and temporal parameterizations

One common formulation begins with a video of length $T$, partitions it into $K$ temporal intervals, and samples a local clip $v_k$ centered within $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$. In the controllable augmentation framework, these local clips are paired with a global video $v$, and the encoder extracts $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$ and $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$ with $T_v > T_c$ (Qian et al., 2022). This construction explicitly introduces multiple temporal receptive fields into the same learning problem.

A second formulation changes the sampling rate. In V3S, temporal scale is written as

$$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$

while temporal projection uses a multi-stage progressive speed-up,

$T$0

The first targets motion magnitude through fixed-interval sampling; the second targets motion direction and pattern changes through piecewise sampling at different rates within the same sample (Li et al., 2021).

A third formulation keeps the clip length fixed but makes augmentation magnitude a temporal signal. DynaAugment models a magnitude sequence $T$1 as

$T$2

where the sequence is generated as a random weighted sum of sinusoidal basis functions. This gives smooth yet diverse frame-wise perturbation schedules rather than a single scalar magnitude for the whole video (Kim et al., 2022).

A fourth formulation is stride-based subsampling. TSMS-SAM2 defines

$T$3

with $T$4, and uses temporal scales such as $T$5 during training to simulate different apparent motion speeds (Xu et al., 7 Aug 2025). At inference time, T3S samples multiple short subsequences and performs additional token subsampling, reducing self-attention cost from $T$6 to $T$7, where $T$8 (Wang et al., 22 Nov 2025).

These formulations show that “multi-temporal-scale” can mean at least four different things: different clip spans, different frame rates, different frame-wise perturbation schedules, or different subsequence budgets. This suggests that the topic is better understood as a design axis than as one fixed algorithm.

3. Role in self-supervised video representation learning

In self-supervised video representation learning, multi-temporal-scale augmentation was introduced largely to correct limitations of clip-only contrastive learning. The controllable augmentation framework argues that positive pairs sampled from the same video tend to have limited temporal receptive field, usually share similar background, and differ in motions; this biases learning toward static background and weak global temporal structure. Its remedy is to jointly use local clips and global videos, establish a soft correspondence

$T$9

apply soft spatio-temporal region contrast, minimize mutual information to avoid low-level redundancy shortcuts, and impose local-global temporal order dependency through ordered clip sequences $K$0 (Qian et al., 2022). The paper reports superiority on three video benchmarks in action recognition and video retrieval and attributes the gains to more accurate temporal dynamic modeling (Qian et al., 2022).

V3S represents a different self-supervised use of multi-temporal-scale augmentation. It combines sampling across width, height, and time, with temporal scale $K$1 and temporal projection $K$2 as pretext transformations. On UCF101 with an R(2+1)D backbone, the ablation reported $K$3 for $K$4 with $K$5, $K$6 for $K$7, $K$8 for $K$9, and $v_k$0 when all four transformations $v_k$1 were combined. The same paper reported that V3S reached $v_k$2 on UCF101 with R(2+1)D versus $v_k$3 for PacePred, and $v_k$4 with S3D-G versus $v_k$5 for SpeedNet (Li et al., 2021).

CATE addresses a different issue: standard contrastive pipelines implicitly assume invariance to view selection mechanisms such as temporal shifts, even when downstream tasks violate that invariance. It therefore explicitly encodes augmentation parameters such as temporal shift and arrow of time and feeds those encodings into a Transformer projection head: $v_k$6 On Something-Something-v1, the reported top-1 accuracy rose from $v_k$7 for a SimCLR++ baseline with temporal augmentation only to $v_k$8 with time encoding and $v_k$9 with crop and time encoding. On a time-shift classification proxy task, absolute time-shift encoding reached $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$0, while the baseline achieved $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$1 (Sun et al., 2021).

FreqAug is adjacent rather than identical to multi-temporal-scale sampling, but it addresses the same static-bias problem by stochastically removing spatial or temporal low-frequency components in the frequency domain. The paper reports consistent improvements when transferring representations to five video action recognition and two temporal action localization tasks (Kim et al., 2022). A plausible implication is that multi-temporal-scale sampling and frequency-selective perturbation are complementary routes to the same objective: forcing the representation to depend less on static information.

4. Temporally dynamic augmentation in supervised recognition

In supervised video recognition, a central critique is that many pipelines simply extend image augmentation by applying the same operation to all frames. DynaAugment argues that this misses real-world temporal variation, while naive per-frame randomization breaks temporal consistency and produces unnatural videos. Its alternative is to change the magnitude of augmentation operations over time through Fourier Sampling, producing smooth and diverse temporal variation. The paper reports gains on large-scale, small-scale, fine-grained, transfer, and corrupted-video settings. The summary gives the following examples: Kinetics-400 with SlowFast-R50-8x8 improved from $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$2 to $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$3, Something-Something-v2 with TSM-R50-16 from $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$4 to $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$5, UCF-101 from $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$6 to $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$7, HMDB-51 from $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$8 to $\left[\frac{(k-1)T}{K}, \frac{kT}{K}\right]$9, Diving-48 from $v$0 to $v$1, and THUMOS'14 mAP@0.5 from $v$2 to $v$3. On corrupted Kinetics with $v$4, DynaAugment showed the smallest drop (Kim et al., 2022).

“Extending Temporal Data Augmentation for Video Action Recognition” pursues a related agenda but through explicit temporal manipulations such as VideoReverse, FrameFadeIn, VideoCutMix, MagAugment, and “floating” variants of CutOut, CutMix, and MixUp. FrameFadeIn is defined as

$v$5

with $v$6 varying across the video; floating methods interpolate box locations or mixing ratios across frames. The reported results include $v$7 top-1 on UCF-101 and $v$8 on HMDB-51 for MagAugment, compared with $v$9 and $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$0 for spatial RandAugment. Among mixing-based methods, FloatFrameCutMixUp achieved $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$1 on UCF-101 versus $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$2 for FrameCutMixUp and $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$3 on HMDB-51 versus $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$4 for CutMixUp (Gorpincenko et al., 2022).

Not all temporally consistent augmentation belongs to this category. S3Aug performs label-based scene-category replacement across all frames using the same permutation $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$5, optionally with semantic sampling and temporal feature shift in the generator, but the method explicitly does not sample or process the video at different temporal resolutions or select different frame rates (Sugiura et al., 2023). This is an important boundary condition: temporal consistency alone is not equivalent to multi-temporal-scale augmentation.

5. Adaptive and inference-time temporal selection for long videos

Long-form video understanding introduces a different pressure: dense temporal sampling is computationally prohibitive, but sparse fixed sampling is inadequate for multi-event and multi-granularity reasoning. MIST addresses this by decomposing dense spatial-temporal self-attention into iterative segment and region selection modules conditioned on the question. Segment selection is written as

$$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$6

and region selection applies an analogous top-$$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$7 selector at the patch level. By stacking ISTA layers, MIST can shift focus across segments and regions over multiple reasoning steps, which the paper links to better handling of multi-event questions and improved efficiency and interpretability on AGQA, NExT-QA, STAR, and Env-QA (Gao et al., 2022).

T3S moves multi-temporal-scale sampling to inference for MLLM video understanding. It samples multiple short and diverse subsequences $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$8, randomly subsamples a fraction $$f(v_k) \in \mathbb{R}^{C \times T_c \times H \times W}$$9 of tokens per trial, packs all trials in a single forward pass with block-diagonal attention, and aggregates logits. The reported complexity reduction is from $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$0 to $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$1. On LongVideoBench, Qwen2.5-VL-7B improved from $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$2 to $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$3, LLaVA-Video-7B from $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$4 to $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$5, and Oryx-1.5-7B from $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$6 to $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$7. First-token delay was reduced by up to $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$8 (Wang et al., 22 Nov 2025).

In temporal grounding, TempSamp-R1 treats the temporal search space itself as the optimization problem. The paper contrasts pure on-policy sampling in GRPO with a hybrid of on-policy exploration and off-policy supervision from ground-truth timestamps, combined with a non-linear soft advantage computation and hybrid CoT/non-CoT training. It reports $$f'(v) \in \mathbb{R}^{C \times T_v \times H \times W}$$9 [email protected] on Charades-STA $T_v > T_c$0, $T_v > T_c$1 [email protected] on ActivityNet Captions $T_v > T_c$2, and $T_v > T_c$3 mAP on QVHighlights $T_v > T_c$4 (Li et al., 22 Sep 2025). Although this is not augmentation in the classical pixel-space sense, it extends the same principle to temporally precise sampling over large search spaces.

6. Domain-specific extensions, adjacent methods, and limitations

In domain-specific settings, multi-temporal-scale sampling augmentation is often motivated by motion regimes that differ sharply from consumer-video benchmarks. TSMS-SAM2 applies stride-based augmentation to promptable VOST in surgical videos and finds that two temporal scales, $T_v > T_c$5, are optimal. In its ablation, adding temporal sampling alone increased J&F from $T_v > T_c$6 to $T_v > T_c$7 and Dice from $T_v > T_c$8 to $T_v > T_c$9; combining temporal sampling with memory splitting and pruning produced $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$0 J&F and $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$1 Dice on EndoVis2018. The stride set $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$2 outperformed $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$3 and $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$4, indicating that more aggressive temporal gaps can reduce frame association quality (Xu et al., 7 Aug 2025).

EventAug offers an event-based analogue through Multi-scale Temporal Integration (MSTI), which integrates event streams at multiple temporal scales such as base, finer, and coarser windows. On DVS128 Gesture with a CSNN backbone, the reported ablation gave $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$5 for base scale only, $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$6 for long-term scale only, and $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$7 for MSTI; the full EventAug reached $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$8, a $$V^s = [v_r, v_{r+s-1}, v_{r+2(s-1)}, \ldots, v_{r+l(s-1)}],$$9 accuracy gain (Tian et al., 2024). Although event data are not conventional RGB video, the same multi-scale temporal principle is used to diversify motion speed and temporal pattern.

Several adjacent techniques clarify the limits of the concept. Channel sampling strategies for 2D networks reorder RGB or grayscale channels from neighboring frames to capture short-term frame-to-frame changes without increasing computational cost; the reported gains reached up to $T$00 over the standard video input on some settings, but the method targets short-term temporal information rather than explicit multi-scale temporal sampling (Kim et al., 2022). FreqAug, similarly, is spatio-temporal augmentation in the frequency domain rather than frame selection (Kim et al., 2022). S3Aug is temporally coherent and context-diversifying, but not explicitly multi-scale (Sugiura et al., 2023).

Two limitations recur across the literature. First, temporal consistency matters: DynaAugment explicitly argues that naive per-frame random augmentation destroys temporal consistency and can harm learning (Kim et al., 2022). Second, more scales are not automatically better: TSMS-SAM2 found that strides beyond $T$01 hurt performance, and S3Aug reports that excessive augmentation probability $T$02 hurts performance because too much content is replaced (Xu et al., 7 Aug 2025, Sugiura et al., 2023). These results counter a common misconception that temporal diversity can be increased monotonically without destabilizing the underlying action or object identity.

Overall, the literature presents multi-temporal-scale video sampling augmentation not as a single recipe but as a recurring strategy for exposing models to heterogeneous temporal evidence. Across contrastive pretraining, supervised recognition, long-video reasoning, temporal grounding, VOST, and event-based learning, the recurring design choice is to construct views that differ in temporal span, temporal rate, or temporal perturbation schedule, while preserving enough semantic continuity for the task at hand (Qian et al., 2022, Kim et al., 2022, Li et al., 2021, Gao et al., 2022, Xu et al., 7 Aug 2025).