SDST: Sparse-Dense Side-Tuning for VTG

• Sparse-Dense Side-Tuner (SDST) is a dual-stream side-tuning architecture that separates dense frame refinement for highlight detection from sparse moment retrieval.
• It introduces a novel reference-based deformable self-attention mechanism that leverages left, center, and right video references for improved contextual modeling.
• SDST achieves competitive performance with significantly fewer parameters by freezing a large pretrained backbone and employing an efficient dual-stream design.

to=arxiv_search 天天中彩票篮球json code 北京赛车计划 {"query":"(Pujol-Perich et al., 10 Jul 2025) Sparse-Dense Side-Tuner for efficient Video Temporal Grounding","max_results":5,"sort_by":"relevance"} to=search_arxiv 大发快三走势图 ՞ւjson code {"query":"(Pujol-Perich et al., 10 Jul 2025) Sparse-Dense Side-Tuner for efficient Video Temporal Grounding","max_results":5} Sparse-Dense Side-Tuner (SDST) is a dual-stream side-tuning architecture for efficient Video Temporal Grounding (VTG) that separates frame-level multimodal refinement from moment-level proposal refinement while keeping a large pretrained backbone frozen (Pujol-Perich et al., 10 Jul 2025). It is formulated for the joint setting of Moment Retrieval (MR) and Highlight Detection (HD), and is presented as the first anchor-free side-tuning architecture for VTG (Pujol-Perich et al., 10 Jul 2025). In conceptual terms, SDST inherits the frozen-backbone-plus-side-network paradigm introduced by side-tuning, but replaces a single dense corrective branch with a task-structured decomposition: a dense branch aligned with HD and a sparse branch aligned with MR (Zhang et al., 2019, Pujol-Perich et al., 10 Jul 2025).

1. Position within the side-tuning literature

The general side-tuning formulation adapts a pretrained base model $B:\mathbb{X}\to\mathbb{Y}$ by learning a side model $S:\mathbb{X}\to\mathbb{Y}$ and fusing the two through

$R(x)=B(x)\oplus S(x),$

with the main fusion rule instantiated as

$R(x)=\alpha B(x)+(1-\alpha)S(x),$

where $\alpha$ is a learnable blending coefficient (Zhang et al., 2019). In that formulation, the base model is frozen, while the side network, the mixing coefficient, and the task-specific decoder are trainable. The original side-tuning paper is explicit that its contribution is a dense additive side-network formulation and that it does not introduce any explicit sparsity mechanism, mask learning, sparse routing, or sparse parameterization (Zhang et al., 2019).

Later side-tuning variants extended the same frozen-backbone logic to other domains. Quantized Side Tuning (QST), for example, applies the pattern to decoder-only LLM finetuning by using a 4-bit frozen backbone, a reduced-width side network, and gated additive fusion, while updating only the side-tuning machinery (Zhang et al., 2024). QST remains parameter-efficient rather than explicitly sparse: its efficiency comes from quantization, reduced width, low-rank or bottleneck downsamplers, and avoidance of backpropagation through the backbone, not from pruning or sparse masks (Zhang et al., 2024).

SDST should therefore be read as a specialized evolution of side-tuning rather than a generic synonym for any parameter-efficient adaptation method. Its novelty is not merely the presence of a side branch, but the explicit sparse-dense decomposition of VTG reasoning: dense frame refinement for HD and sparse recurrent decoder queries for MR (Pujol-Perich et al., 10 Jul 2025).

2. Problem formulation and motivation in video temporal grounding

VTG aims to localize video content relevant to a natural-language query. For a video-query pair

$$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$

the model predicts frame saliency scores for HD,

$\mathbf{Y}^s\in\mathbb{R}^{T},$

and temporal moments for MR,

$\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$

Here $T$ is the number of frames, $L$ the number of text tokens, and $S:\mathbb{X}\to\mathbb{Y}$0 the number of target moments (Pujol-Perich et al., 10 Jul 2025).

The motivation for SDST is tied to the limitations of prevailing VTG adaptation strategies. Most VTG methods use frozen pretrained backbones and consume only final-layer features, which the paper identifies as limiting under downstream domain shift, especially when transferring from image-language pretraining to video-language grounding (Pujol-Perich et al., 10 Jul 2025). Full fine-tuning of large video-language backbones is described as often computationally impractical, while standard PEFT methods such as prompting and adapters may reduce trainable parameters but still require backpropagation through the full backbone and therefore remain memory-heavy (Pujol-Perich et al., 10 Jul 2025).

The paper also argues that prior side-tuning for VTG is structurally mismatched to MR. The closest prior side-tuning method, R$S:\mathbb{X}\to\mathbb{Y}$1-Tuning, recursively refines multimodal frame-level CLIP features and adapts them to MR through a dense anchor set; SDST critiques this design on the grounds that MR is inherently sparse, whereas dense anchor-based formulations impose a dense proposal structure (Pujol-Perich et al., 10 Jul 2025). This motivates an anchor-free, DETR-style sparse decoder for MR alongside a dense frame-level stream for HD.

A second motivation concerns deformable attention. The paper diagnoses a limitation in standard deformable cross-attention used in anchor-free VTG decoders: offsets tend to collapse near initialization values, and the mechanism struggles to attend beyond currently estimated moment boundaries, especially for long actions (Pujol-Perich et al., 10 Jul 2025). This diagnosis directly motivates the introduction of Reference-based Deformable Self-Attention (RDSA).

3. Architectural organization of SDST

SDST is organized as a dual-stream side-tuning architecture with a dense branch $S:\mathbb{X}\to\mathbb{Y}$2 for frame-level multimodal and temporal refinement and a sparse branch $S:\mathbb{X}\to\mathbb{Y}$3 for moment proposal refinement (Pujol-Perich et al., 10 Jul 2025). The backbone is a frozen InternVideo2-1B model that outputs $S:\mathbb{X}\to\mathbb{Y}$4 intermediate visual and textual representations,

$S:\mathbb{X}\to\mathbb{Y}$5

These are extracted from the last $S:\mathbb{X}\to\mathbb{Y}$6 layers of InternVideo2; in the best configuration, $S:\mathbb{X}\to\mathbb{Y}$7 with layers $S:\mathbb{X}\to\mathbb{Y}$8 out of 40 total (Pujol-Perich et al., 10 Jul 2025).

The side-tuning state consists of a dense embedding

$S:\mathbb{X}\to\mathbb{Y}$9

initialized to zero, and $R(x)=B(x)\oplus S(x),$0 learnable recurrent decoder queries comprising references

$R(x)=B(x)\oplus S(x),$1

representing center-width moments, and latent query embeddings

$R(x)=B(x)\oplus S(x),$2

Refinement then proceeds recurrently as

$R(x)=B(x)\oplus S(x),$3

Within the dense branch, visual and textual backbone features are projected into a shared hidden space using two MLPs,

$R(x)=B(x)\oplus S(x),$4

A learned scalar $R(x)=B(x)\oplus S(x),$5, initialized at zero, interpolates between the current dense state and the current visual representation,

$R(x)=B(x)\oplus S(x),$6

Textual information and temporal context are then injected by cross-attention, self-attention, and a point-wise feedforward network: $R(x)=B(x)\oplus S(x),$7 The output $R(x)=B(x)\oplus S(x),$8 is used both for HD and as conditioning for the sparse branch (Pujol-Perich et al., 10 Jul 2025).

The sparse branch behaves as a recurrent DETR-style decoder over $R(x)=B(x)\oplus S(x),$9 moment queries. It first updates sparse queries through text interaction,

$R(x)=\alpha B(x)+(1-\alpha)S(x),$0

and then conditions them on the dense video representation via

$R(x)=\alpha B(x)+(1-\alpha)S(x),$1

This is the central sparse-dense interaction: dense frame features support sparse moment refinement (Pujol-Perich et al., 10 Jul 2025).

The frozen-trainable split is strict. Frozen components are the InternVideo2-1B backbone and the reused AdaptivePool module. Trainable components are the visual and text projection MLPs, dense-branch $R(x)=\alpha B(x)+(1-\alpha)S(x),$2, sparse-branch $R(x)=\alpha B(x)+(1-\alpha)S(x),$3, prediction heads for classification, regression, actionness, and HD saliency, and the alignment-loss projection or pooling operations. All SDST modules are shared over the $R(x)=\alpha B(x)+(1-\alpha)S(x),$4 refinement levels, which the ablations identify as both more parameter-efficient and more accurate than unshared alternatives (Pujol-Perich et al., 10 Jul 2025).

4. Reference-based Deformable Self-Attention

RDSA is introduced to replace standard deformable cross-attention as the video-conditioning mechanism inside the sparse branch (Pujol-Perich et al., 10 Jul 2025). In the standard deformable formulation, query and key projections are

$R(x)=\alpha B(x)+(1-\alpha)S(x),$5

and offsets and attention weights are predicted from the queries alone,

$R(x)=\alpha B(x)+(1-\alpha)S(x),$6

Aggregation then samples keys relative to the current center $R(x)=\alpha B(x)+(1-\alpha)S(x),$7 and width $R(x)=\alpha B(x)+(1-\alpha)S(x),$8: $R(x)=\alpha B(x)+(1-\alpha)S(x),$9 The paper argues that this is poorly suited to VTG cross-attention because the offsets are predicted from query embeddings that lack real contextual access to the key-value space (Pujol-Perich et al., 10 Jul 2025).

RDSA addresses this by making offset prediction reference-aware and video-derived. Given current references $\alpha$0 and dense embeddings $\alpha$1, the model first enriches dense embeddings with a CNN, samples left, center, and right reference embeddings, concatenates them, and forms an alternative query: $\alpha$2 with

$\alpha$3

Offsets and attention scores are then predicted from this reference-based query,

$\alpha$4

and deformable aggregation reuses the same sampling equation with $\alpha$5 and $\alpha$6 (Pujol-Perich et al., 10 Jul 2025).

The mechanism is described as improving context modeling in three ways: key-space awareness, boundary-sensitive reasoning via left and right reference embeddings, and local neighborhood context via a 2-layer CNN (Pujol-Perich et al., 10 Jul 2025). Empirically, on QVHighlights validation, RDSA improves over standard deformable cross-attention from $\alpha$7 to $\alpha$8 on $\alpha$9, from $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$0 to $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$1 on Avg mAP, and from $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$2 to $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$3 on HD mAP; on long actions, it improves mAP from $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$4 to $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$5 (Pujol-Perich et al., 10 Jul 2025). The best sampling strategy is left-center-right with CNN, yielding Avg mAP $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$6, HD mAP $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$7, and HIT@1 $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$8 (Pujol-Perich et al., 10 Jul 2025).

5. Training objectives, inference, and implementation profile

HD is computed from the final dense embedding $$(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},$$9 and a pooled text embedding $\mathbf{Y}^s\in\mathbb{R}^{T},$0 using frame-wise cosine similarity,

$\mathbf{Y}^s\in\mathbb{R}^{T},$1

Training uses an InfoNCE / SampledNCE ranking-style loss,

$\mathbf{Y}^s\in\mathbb{R}^{T},$2

with the supplementary text describing the objective as ranking positive frames higher (Pujol-Perich et al., 10 Jul 2025).

MR uses DETR-style bipartite matching via the Hungarian algorithm. The classification term is focal loss,

$\mathbf{Y}^s\in\mathbb{R}^{T},$3

the boundary term is

$\mathbf{Y}^s\in\mathbb{R}^{T},$4

the localization-overlap term is

$\mathbf{Y}^s\in\mathbb{R}^{T},$5

and the actionness term is

$\mathbf{Y}^s\in\mathbb{R}^{T},$6

The overall MR loss is

$\mathbf{Y}^s\in\mathbb{R}^{T},$7

with all terms except $\mathbf{Y}^s\in\mathbb{R}^{T},$8 optimized across refinement levels to accelerate convergence. Alignment is further regularized by two SampledNCE / InfoNCE-based contrastive terms summarized as $\mathbf{Y}^s\in\mathbb{R}^{T},$9, and the final objective is

$\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$0

Default loss weights are mostly simple: $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$1, $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$2, $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$3, $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$4, with saliency- and alignment-related coefficients around $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$5 (Pujol-Perich et al., 10 Jul 2025).

At inference time, each query outputs predicted boundaries, class probability $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$6, and actionness $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$7. These are combined into

$\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$8

and final moments are selected with soft-NMS. HD scores are directly the frame-wise cosine similarities $\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.$9 (Pujol-Perich et al., 10 Jul 2025).

The implementation profile is tightly specified. The best configuration uses hidden dimension $T$0, $T$1 refinement levels, sinusoidal positional encoding, dropout $T$2, DropPath $T$3, $T$4 attention heads, FFN expansion ratio $T$5, ReLU, LayerNorm with PostNorm, and $T$6 deformable sampling points. The sparse-branch RDSA context module is a 2-layer CNN with hidden dimension $T$7, and concatenated sampled tokens are projected to a $T$8-dimensional latent before offset and attention prediction. The number of queries is $T$9 on QVHighlights and Charades-STA and $L$0 on TACoS. The classification head is a 1-layer MLP; the regression and actionness heads are 3-layer MLPs; actionness operates on RoI pooled features with RoI size $L$1 (Pujol-Perich et al., 10 Jul 2025).

6. Quantitative profile, efficiency, and empirical interpretation

SDST is reported to achieve highly competitive or SOTA VTG results while remaining strongly parameter-efficient (Pujol-Perich et al., 10 Jul 2025). On QVHighlights, the main parameter comparison is SG-DETR at $L$2M, Flash-VTG at $L$3M, R$L$4-Tuning at $L$5M, and SDST at $L$6M. Relative to SG-DETR, SDST uses about $L$7 of the parameters, corresponding to about $L$8 fewer parameters, which matches the paper’s “up to 73% reduction” claim (Pujol-Perich et al., 10 Jul 2025).

Dataset	SDST result	Comparative note
QVHighlights val	Avg mAP 55.60, HD mAP 44.00, HIT@1 72.00, Params 4.1M	Nearly matches SG-DETR Avg mAP 55.64 with 15.0M params
QVHighlights test	Avg mAP 53.31, HD mAP 43.40, HIT@1 69.13, Params 4.1M	Very close to SG-DETR while clearly outperforming R$L$9-Tuning
Charades-STA	[email protected] 72.0, [email protected] 52.6, mIoU 61.2	$S:\mathbb{X}\to\mathbb{Y}$00 [email protected] and $S:\mathbb{X}\to\mathbb{Y}$01 mIoU over prior SOTA
TACoS	[email protected] 44.5, [email protected] 32.3, mIoU 42.2	$S:\mathbb{X}\to\mathbb{Y}$02 [email protected] and $S:\mathbb{X}\to\mathbb{Y}$03 mIoU over SG-DETR, with slight drop in [email protected]

On QVHighlights validation, SDST records $S:\mathbb{X}\to\mathbb{Y}$04, $S:\mathbb{X}\to\mathbb{Y}$05, $S:\mathbb{X}\to\mathbb{Y}$06, $S:\mathbb{X}\to\mathbb{Y}$07, Avg mAP $S:\mathbb{X}\to\mathbb{Y}$08, HD mAP $S:\mathbb{X}\to\mathbb{Y}$09, and HIT@1 $S:\mathbb{X}\to\mathbb{Y}$10 (Pujol-Perich et al., 10 Jul 2025). In the fair comparison using InternVideo2 features only, it is statistically on par with SG-DETR while surpassing R$S:\mathbb{X}\to\mathbb{Y}$11-Tuning by $S:\mathbb{X}\to\mathbb{Y}$12 Avg mAP on MR and $S:\mathbb{X}\to\mathbb{Y}$13 HD mAP (Pujol-Perich et al., 10 Jul 2025). On Charades-STA it reaches $S:\mathbb{X}\to\mathbb{Y}$14 at $S:\mathbb{X}\to\mathbb{Y}$15, $S:\mathbb{X}\to\mathbb{Y}$16 at $S:\mathbb{X}\to\mathbb{Y}$17, and $S:\mathbb{X}\to\mathbb{Y}$18 mIoU; on TACoS it reaches $S:\mathbb{X}\to\mathbb{Y}$19 at $S:\mathbb{X}\to\mathbb{Y}$20, $S:\mathbb{X}\to\mathbb{Y}$21 at $S:\mathbb{X}\to\mathbb{Y}$22, and $S:\mathbb{X}\to\mathbb{Y}$23 mIoU (Pujol-Perich et al., 10 Jul 2025).

Efficiency is favorable but not uniformly dominant. On QVHighlights with batch size $S:\mathbb{X}\to\mathbb{Y}$24, SDST uses $S:\mathbb{X}\to\mathbb{Y}$25M parameters, $S:\mathbb{X}\to\mathbb{Y}$26 GB memory, and $S:\mathbb{X}\to\mathbb{Y}$27 iterations per second, compared with $S:\mathbb{X}\to\mathbb{Y}$28 GB and $S:\mathbb{X}\to\mathbb{Y}$29 it/s for R$S:\mathbb{X}\to\mathbb{Y}$30-Tuning and $S:\mathbb{X}\to\mathbb{Y}$31 GB and $S:\mathbb{X}\to\mathbb{Y}$32 it/s for Flash-VTG (Pujol-Perich et al., 10 Jul 2025). The paper therefore does not present SDST as the absolute lightest in memory or runtime; rather, its central efficiency claim is strong parameter efficiency relative to high-performing InternVideo2-based baselines (Pujol-Perich et al., 10 Jul 2025).

Ablations clarify where the gains originate. Parameter sharing across refinement levels reduces the model from $S:\mathbb{X}\to\mathbb{Y}$33M to $S:\mathbb{X}\to\mathbb{Y}$34M parameters while improving Avg mAP from $S:\mathbb{X}\to\mathbb{Y}$35 to $S:\mathbb{X}\to\mathbb{Y}$36 (Pujol-Perich et al., 10 Jul 2025). Removing RDSA causes a severe collapse in sparse-branch localization, dropping $S:\mathbb{X}\to\mathbb{Y}$37 to $S:\mathbb{X}\to\mathbb{Y}$38 and Avg mAP to $S:\mathbb{X}\to\mathbb{Y}$39, which indicates that the sparse branch depends critically on dense-branch video conditioning (Pujol-Perich et al., 10 Jul 2025). Conditioning the sparse branch with progressively richer dense features also improves performance, with the best result obtained from the post-$S:\mathbb{X}\to\mathbb{Y}$40 dense representation rather than raw visual features or earlier dense states (Pujol-Perich et al., 10 Jul 2025). Reusing InternVideo2’s frozen AdaptivePool across intermediate layers is likewise decisive: on QVHighlights validation it outperforms CLS pooling by $S:\mathbb{X}\to\mathbb{Y}$41 Avg mAP, $S:\mathbb{X}\to\mathbb{Y}$42 HD mAP, and $S:\mathbb{X}\to\mathbb{Y}$43 HIT@1 (Pujol-Perich et al., 10 Jul 2025).

7. Scope, terminology, and common points of confusion

The term “Sparse-Dense” in SDST refers to the decomposition of VTG computation into a sparse branch for moment proposal refinement and a dense branch for frame-level multimodal and temporal refinement (Pujol-Perich et al., 10 Jul 2025). It does not denote the kind of explicit sparsity machinery absent from the original side-tuning literature, where the side network is dense and additively fused with a frozen base (Zhang et al., 2019). Nor is it identical to parameter-efficient dense side-branch methods such as QST, whose efficiency derives from quantization, reduced width, and low-rank or pooling downsamplers rather than from a sparse proposal structure (Zhang et al., 2024).

A second common confusion is to treat SDST as a generic PEFT wrapper around a frozen video backbone. The paper’s results indicate that its performance depends on a specific conjunction of design choices: anchor-free sparse decoding for MR, dense frame refinement for HD, RDSA for reference-grounded video conditioning, recurrent use of intermediate InternVideo2 layers, and reuse of the frozen AdaptivePool module (Pujol-Perich et al., 10 Jul 2025). In that sense, SDST is not merely side-tuning applied to VTG; it is a VTG-specific side-tuning architecture whose sparse-dense split follows the structural asymmetry between MR and HD.

A plausible implication is that SDST provides a particularly informative comparison point for future VTG adaptation methods: it retains the frozen-backbone modularity associated with side-tuning, but shows that task-structured side branches can substantially outperform frame-centric dense side-tuning baselines without requiring full fine-tuning (Zhang et al., 2019, Pujol-Perich et al., 10 Jul 2025).