Directional Adaptive Semantic Perturbation (DASP)

• The paper demonstrates that DASP enhances text–video retrieval by perturbing normalized CLIP text embeddings along a cross-modal variance direction to achieve robust, discriminative alignment.
• DASP employs a directional, adaptive perturbation integrated with dual-branch contrastive loss and frame-level gated fusion, balancing stochastic and deterministic behaviors using a learnable scalar.
• Empirical results on MSR-VTT and other datasets show that DASP outperforms isotropic stochastic methods with significant gains in retrieval metrics while maintaining near-baseline inference speed.

Directional Adaptive Semantic Perturbation (DASP) is a perturbation mechanism for text-to-video retrieval that operates on the global text embedding within the GAID framework and is intended to make text–video alignment more robust and discriminative while preserving single-pass inference efficiency (Yang et al., 3 Aug 2025). In its published usage, DASP denotes a directional, adaptively scaled, structure-aware perturbation of normalized text embeddings along a cross-modal variance direction estimated from fused audio-visual features. The method is paired with frame-level gated audio-visual fusion and is trained through dual-branch contrastive objectives that use both a perturbed text embedding and a directional support embedding (Yang et al., 3 Aug 2025).

1. Definition and formal formulation

In GAID, the base text representation is a normalized global text embedding $\mathbf{t} \in \mathbb{R}^d$, obtained from the CLIP text encoder and projected to a shared space with $d=512$. DASP uses a vector $\text{std} \in \mathbb{R}^d$, described as a cross-modal variance estimate derived from fused video features and used as the perturbation direction, together with a learnable scalar $\alpha$ controlling perturbation strength and training-time stochasticity (Yang et al., 3 Aug 2025).

The method is defined relative to a recalled naïve stochastic text perturbation (STP),

$$\mathbf{t}_{\text{pert}} = \mathbf{t} + \sigma \cdot \text{std}, \quad \sigma \sim \mathcal{N}(0,1),$$

which requires multiple samples at both training and inference. DASP replaces this with a directional perturbation whose training-time form is

$$\mathbf{t}_{\text{pert}} = \mathbf{t} + (\alpha \cdot \sigma + 1 - \alpha)\cdot \text{std}, \quad \sigma \sim \mathcal{N}(0,1),$$

and whose inference-time form is

$$\mathbf{t}_{\text{pert}} = \mathbf{t} + \alpha \cdot \text{std}.$$

The interpolation term $(\alpha \sigma + 1 - \alpha)$ governs the relation between stochastic and deterministic behavior. When $\alpha=1$, the method reduces to classical STP, $\mathbf{t} + \sigma \cdot \text{std}$; when $d=512$0, it becomes a purely deterministic shift $d=512$1 (Yang et al., 3 Aug 2025). The paper characterizes this as perturbation along the same direction $d=512$2, rather than unconstrained noise injection.

A second representation, the support embedding $d=512$3, is positioned along the same cross-modal variance direction at the perturbation boundary and is treated as a directional support embedding near the decision margin (Yang et al., 3 Aug 2025). The paper does not provide a closed-form formula for $d=512$4, but it does define its role in the optimization.

2. Motivation, intuition, and geometric interpretation

The method is motivated by two stated problems in text-to-video retrieval. First, the CLIP-based text embedding can be unstable when visual or audio evidence is noisy or incomplete, or when multiple videos are visually similar but differ in subtle semantics such as speech or sound. Second, prior stochastic perturbation methods such as STP are described as isotropic and inference-expensive: they apply noise in arbitrary directions on the hypersphere and require multiple inference passes for sampling (Yang et al., 3 Aug 2025).

The paper defines the three terms in the name with technical specificity. DASP is directional because perturbations are aligned to a specific direction vector $d=512$5 derived from cross-modal variance of fused video features. It is adaptive because $d=512$6 depends on the current video–text pair’s fused audio-visual representation and because the scalar $d=512$7 adjusts perturbation strength and stochasticity between training and inference. It is semantic because the perturbation direction is not arbitrary noise but is based on cross-modal variance, so movement occurs along dimensions that reflect real variation in how videos express the same language concept (Yang et al., 3 Aug 2025).

The geometric argument is made explicit through a spherical-cap analysis. In $d=512$8-dimensional space, the probability mass of a cone with half-angle $d=512$9 is written as

$\text{std} \in \mathbb{R}^d$0

For $\text{std} \in \mathbb{R}^d$1 and $\text{std} \in \mathbb{R}^d$2, the paper gives

$\text{std} \in \mathbb{R}^d$3

This formalizes the claim that even a $\text{std} \in \mathbb{R}^d$4 cone occupies a vanishingly small portion of directions on the hypersphere, and is used to contrast DASP’s concentrated directional perturbations with STP’s full-hypersphere stochasticity (Yang et al., 3 Aug 2025).

The paper further states that DASP “generates a biased directional cone, where perturbations concentrate along the semantic axis and are shifted toward the positive direction, suppressing negative perturbations.” This suggests that the method is intended not merely to regularize by noise, but to regularize by semantically aligned displacement within a narrow, structured subspace.

3. Placement within GAID and interaction with multimodal fusion

DASP is one of two central components of GAID; the other is Frame-level Gated Fusion (FGF). The GAID pipeline first encodes text with CLIP-Text, video frames with CLIP-ViT, and audio with Whisper or Wav2vec2.0, then fuses frame-level audio and visual features under textual guidance before applying DASP on the text side (Yang et al., 3 Aug 2025).

The base text embedding is produced as

$\text{std} \in \mathbb{R}^d$5

with all features projected to a 512-dimensional space. On the video side, the frame-level gate is defined as

$\text{std} \in \mathbb{R}^d$6

and the fused frame representation is

$\text{std} \in \mathbb{R}^d$7

These fused features are aggregated into a global video embedding $\text{std} \in \mathbb{R}^d$8, and a lightweight cross-attention from text to the fused video features further refines $\text{std} \in \mathbb{R}^d$9 and/or $\alpha$0 (Yang et al., 3 Aug 2025).

Within this pipeline, DASP sits after multimodal fusion and text–video cross-attention. The sequence described in the paper is: raw inputs, modality-specific encoders, FGF to obtain fused frame features and global video embedding, text–video cross-attention, computation of the cross-modal variance direction $\alpha$1 from fused representation, and then perturbation of the text embedding into $\alpha$2 and $\alpha$3 (Yang et al., 3 Aug 2025).

This division of labor is conceptually important. FGF is described as reducing the modality gap by producing a stronger multimodal video representation with frame-wise audio–visual information guided by text, while DASP stabilizes the alignment between text and those enriched video representations by modeling semantic variability in text space along directions supported by video. A plausible implication is that the perturbation mechanism is only as informative as the fused representation from which $\alpha$4 is derived.

4. Training objective and inference behavior

DASP is optimized with a dual-branch contrastive loss. The perturbation branch uses the perturbed text embedding and defines

$\alpha$5

where each term is an InfoNCE contrastive loss. The support branch uses the directional support embedding and defines

$\alpha$6

The total loss is

$\alpha$7

The scalar $\alpha$8 controls the trade-off between robustness and boundary shaping (Yang et al., 3 Aug 2025).

The paper emphasizes that the perturbed branch enforces successful retrieval even when the text embedding is moved along the semantic direction, while the support branch simulates a “worst-case positive” near the decision margin and tightens the boundary accordingly. Non-perturbed text remains present in the system for computing cross-attention with video, computing gates in FGF, and defining the perturbation center (Yang et al., 3 Aug 2025).

A central design goal is avoidance of multi-pass inference. Prior stochastic perturbation methods realize robustness by sampling many variants of text at inference and aggregating similarity, making inference time approximately proportional to the number of samples. DASP instead uses the deterministic inference-time form $\alpha$9, so only one forward pass is required per query (Yang et al., 3 Aug 2025). The paper presents this as an expectation-like deterministic counterpart of training-time stochastic perturbation.

5. Empirical behavior and efficiency

The direct perturbation ablation on MSR-VTT with ViT-B/32 compares no perturbation, STP, and DASP (Yang et al., 3 Aug 2025).

Method	Retrieval metrics	Time cost (1k retrieval)
No Stochastic	R@1 53.8, R@5 80.4, R@10 88.2, MnR 8.25	6.1s
STP (naïve)	R@1 54.0, R@5 80.6, R@10 89.2, MnR 7.4	98.2s
DASP	R@1 55.0, R@5 83.0, R@10 89.9, MnR 7.7	6.5s

These numbers support two claims made in the paper. First, DASP improves retrieval relative to both no perturbation and naïve STP, especially on R@1 and R@5. Second, it retains near-baseline inference cost: 6.5s versus 6.1s, compared with 98.2s for STP with 20 samples (Yang et al., 3 Aug 2025).

The broader GAID results span MSR-VTT, DiDeMo, VATEX, and LSMDC. For example, on MSR-VTT with ViT-B/32, GAID reports R@1 55.0, R@5 83.0, R@10 89.9, and MnR 7.7, compared with T-MASS at R@1 50.2, R@5 75.3, R@10 85.1, and MnR 11.9. On VATEX, GAID reports R@1 67.7, R@5 92.9, R@10 96.3, and MnR 2.5, compared with T-MASS at R@1 63.0, R@5 92.3, R@10 96.4, and MnR 3.2 (Yang et al., 3 Aug 2025). Because GAID changes both fusion and perturbation, the paper notes that the gains are due to both; however, the perturbation ablation demonstrates that directional semantic perturbation contributes a tangible portion of the improvement.

The cosine-similarity distribution analysis is also cited as evidence: DASP yields higher mean similarity for true text–video pairs and lower variance than STP, which the paper interprets as more stable alignment of positive pairs (Yang et al., 3 Aug 2025).

6. Related methods, conceptual antecedents, and nomenclature

DASP in GAID is positioned primarily against stochastic text perturbation methods such as T-MASS. The distinction is that T-MASS models text embeddings as stochastic Gaussian variables and uses sampling-based perturbations at training and inference, whereas DASP uses cross-modal variance as a semantically informed direction, perturbs within a directional cone, and replaces multi-sampling at inference with a deterministic single-pass update (Yang et al., 3 Aug 2025).

A useful antecedent outside retrieval is DASA, “Difficulty-Aware Semantic Augmentation,” for speaker verification (Wang et al., 2023). DASA perturbs speaker embeddings in representation space along semantic directions obtained from speaker-wise covariance matrices and derives a closed-form upper bound on the expected loss under infinite semantic augmentations. Its perturbations are class-conditional, covariance-driven, and difficulty-aware through DAAM-Softmax (Wang et al., 2023). The relationship is conceptual rather than terminological: DASA provides a covariance-based example of directional semantic perturbation in embedding space, while GAID’s DASP uses a cross-modal variance direction estimated from fused video features.

The acronym itself is not unique across recent arXiv literature. In “Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation,” DASP stands for “Decoupling Adaptation for Stability and Plasticity,” a framework for multi-modal test-time adaptation of audio–video classifiers (He et al., 28 Feb 2026). That usage is unrelated to Directional Adaptive Semantic Perturbation. For that reason, the meaning of “DASP” is context-dependent and must be resolved by the surrounding task and paper title.

Within the retrieval setting, Directional Adaptive Semantic Perturbation refers specifically to a cross-modally guided perturbation of text embeddings that couples semantic regularization, directional control, and deterministic inference efficiency (Yang et al., 3 Aug 2025).