---
title: 'SDST: Sparse-Dense Side-Tuning for VTG'
url: https://www.emergentmind.com/topics/sparse-dense-side-tuner-sdst
type: topic
---

# SDST: Sparse-Dense Side-Tuning for VTG

to=arxiv_search  天天中彩票篮球json code  北京赛车计划  {"query":"arXiv:2507.07744 Sparse-Dense Side-Tuner for efficient Video Temporal Grounding","max_results":5,"sort_by":"relevance"}
to=search_arxiv  大发快三走势图 ՞ւjson code  {"query":"2507.07744 Sparse-Dense Side-Tuner for efficient Video Temporal Grounding","max_results":5}
Sparse-Dense Side-Tuner (SDST) is a dual-stream side-tuning architecture for efficient Video Temporal Grounding (VTG) that separates frame-level multimodal refinement from moment-level proposal refinement while keeping a large pretrained backbone frozen [2507.07744]. It is formulated for the joint setting of Moment Retrieval (MR) and Highlight Detection (HD), and is presented as the first anchor-free side-tuning architecture for VTG [2507.07744]. In conceptual terms, SDST inherits the frozen-backbone-plus-side-network paradigm introduced by side-tuning, but replaces a single dense corrective branch with a task-structured decomposition: a dense branch aligned with HD and a sparse branch aligned with MR [1912.13503], [2507.07744].

## 1. Position within the side-tuning literature

The general side-tuning formulation adapts a pretrained base model \(B:\mathbb{X}\to\mathbb{Y}\) by learning a side model \(S:\mathbb{X}\to\mathbb{Y}\) and fusing the two through
\[
R(x)=B(x)\oplus S(x),
\]
with the main fusion rule instantiated as
\[
R(x)=\alpha B(x)+(1-\alpha)S(x),
\]
where \(\alpha\) is a learnable blending coefficient [1912.13503]. In that formulation, the base model is frozen, while the side network, the mixing coefficient, and the task-specific decoder are trainable. The original side-tuning paper is explicit that its contribution is a dense additive side-network formulation and that it does not introduce any explicit sparsity mechanism, mask learning, sparse routing, or sparse parameterization [1912.13503].

Later side-tuning variants extended the same frozen-backbone logic to other domains. Quantized Side Tuning (QST), for example, applies the pattern to decoder-only LLM finetuning by using a 4-bit frozen backbone, a reduced-width side network, and gated additive fusion, while updating only the side-tuning machinery [2401.07159]. QST remains parameter-efficient rather than explicitly sparse: its efficiency comes from quantization, reduced width, low-rank or bottleneck downsamplers, and avoidance of backpropagation through the backbone, not from pruning or sparse masks [2401.07159].

SDST should therefore be read as a specialized evolution of side-tuning rather than a generic synonym for any parameter-efficient adaptation method. Its novelty is not merely the presence of a side branch, but the explicit sparse-dense decomposition of VTG reasoning: dense frame refinement for HD and sparse recurrent decoder queries for MR [2507.07744].

## 2. Problem formulation and motivation in video temporal grounding

VTG aims to localize video content relevant to a natural-language query. For a video-query pair
\[
(\mathbf{X}^v,\mathbf{X}^t), \qquad \mathbf{X}^v\in\mathbb{R}^{T\times H\times W\times 3}, \quad \mathbf{X}^t\in\mathbb{R}^{L\times F_e},
\]
the model predicts frame saliency scores for HD,
\[
\mathbf{Y}^s\in\mathbb{R}^{T},
\]
and temporal moments for MR,
\[
\mathbf{Y}^m\in\mathbb{R}^{M\times 2}.
\]
Here \(T\) is the number of frames, \(L\) the number of text tokens, and \(M\) the number of target moments [2507.07744].

The motivation for SDST is tied to the limitations of prevailing VTG adaptation strategies. Most VTG methods use frozen pretrained backbones and consume only final-layer features, which the paper identifies as limiting under downstream domain shift, especially when transferring from image-language pretraining to video-language grounding [2507.07744]. Full fine-tuning of large video-language backbones is described as often computationally impractical, while standard PEFT methods such as prompting and adapters may reduce trainable parameters but still require backpropagation through the full backbone and therefore remain memory-heavy [2507.07744].

The paper also argues that prior side-tuning for VTG is structurally mismatched to MR. The closest prior side-tuning method, R\(^2\)-Tuning, recursively refines multimodal frame-level CLIP features and adapts them to MR through a dense anchor set; SDST critiques this design on the grounds that MR is inherently sparse, whereas dense anchor-based formulations impose a dense proposal structure [2507.07744]. This motivates an anchor-free, DETR-style sparse decoder for MR alongside a dense frame-level stream for HD.

A second motivation concerns deformable attention. The paper diagnoses a limitation in standard deformable cross-attention used in anchor-free VTG decoders: offsets tend to collapse near initialization values, and the mechanism struggles to attend beyond currently estimated moment boundaries, especially for long actions [2507.07744]. This diagnosis directly motivates the introduction of Reference-based Deformable Self-Attention (RDSA).

## 3. Architectural organization of SDST

SDST is organized as a dual-stream side-tuning architecture with a dense branch \(\mathcal{D}\) for frame-level multimodal and temporal refinement and a sparse branch \(\mathcal{S}\) for moment proposal refinement [2507.07744]. The backbone is a frozen InternVideo2-1B model that outputs \(K\) intermediate visual and textual representations,
\[
\Tilde{V}\in\mathbb{R}^{K\times T\times F_v}, \qquad \Tilde{T}\in\mathbb{R}^{K\times L\times F_t}.
\]
These are extracted from the last \(K\) layers of InternVideo2; in the best configuration, \(K=4\) with layers \([37,38,39,40]\) out of 40 total [2507.07744].

The side-tuning state consists of a dense embedding
\[
\mathbf{D}^0\in\mathbb{R}^{T\times F},
\]
initialized to zero, and \(M\) learnable recurrent decoder queries comprising references
\[
\mathbf{R}^0\in\mathbb{R}^{M\times 2},
\]
representing center-width moments, and latent query embeddings
\[
\mathbf{H}^0\in\mathbb{R}^{M\times F}.
\]
Refinement then proceeds recurrently as
\[
\mathbf{D}^{\ell+1},\mathbf{R}^{\ell+1},\mathbf{H}^{\ell+1}
=
SDST(\mathbf{D}^{\ell},\mathbf{R}^{\ell},\mathbf{H}^{\ell},\Tilde{\mathbf{V}}^{\ell},\Tilde{\mathbf{T}}^{\ell}).
\]

Within the dense branch, visual and textual backbone features are projected into a shared hidden space using two MLPs,
\[
\mathbf{V}^{\ell}=\mathcal{F}_v(\Tilde{\mathbf{V}}^{\ell}), \qquad
\mathbf{T}^{\ell}=\mathcal{F}_t(\Tilde{\mathbf{T}}^{\ell}).
\]
A learned scalar \(\beta^\ell\in[0,1]\), initialized at zero, interpolates between the current dense state and the current visual representation,
\[
\mathbf{D}^{\ell}:=\beta^\ell \mathbf{D}^{\ell}+(1-\beta^\ell)\mathbf{V}^{\ell}.
\]
Textual information and temporal context are then injected by cross-attention, self-attention, and a point-wise feedforward network:
\[
\mathbf{D}^{\ell+1}=PFFN\!\left(SA\!\left(CA(\mathbf{D}^{\ell},\mathbf{T}^{\ell},\mathbf{T}^{\ell})\right)\right).
\]
The output \(\mathbf{D}^{\ell+1}\) is used both for HD and as conditioning for the sparse branch [2507.07744].

The sparse branch behaves as a recurrent DETR-style decoder over \(M\) moment queries. It first updates sparse queries through text interaction,
\[
\mathbf{H}^{\ell}=SA\!\left(CA(\mathbf{H}^{\ell},\mathbf{T}^{\ell},\mathbf{T}^{\ell})\right),
\]
and then conditions them on the dense video representation via
\[
\mathbf{H}^{\ell}=PFFN\!\left(RDSA(\mathbf{R}^{\ell},\mathbf{H}^{\ell},\mathbf{D}^{\ell+1})\right).
\]
This is the central sparse-dense interaction: dense frame features support sparse moment refinement [2507.07744].

The frozen-trainable split is strict. Frozen components are the InternVideo2-1B backbone and the reused AdaptivePool module. Trainable components are the visual and text projection MLPs, dense-branch \(CA/SA/PFFN\), sparse-branch \(CA/SA/RDSA/PFFN\), prediction heads for classification, regression, actionness, and HD saliency, and the alignment-loss projection or pooling operations. All SDST modules are shared over the \(K\) refinement levels, which the ablations identify as both more parameter-efficient and more accurate than unshared alternatives [2507.07744].

## 4. Reference-based Deformable Self-Attention

RDSA is introduced to replace standard deformable cross-attention as the video-conditioning mechanism inside the sparse branch [2507.07744]. In the standard deformable formulation, query and key projections are
\[
\mathbf{Q}=\mathbf{X}_{\mathcal{Q}}\mathbf{W}_{\mathcal{Q}}^{def}, \qquad
\mathbf{K}=\mathbf{X}_{\mathcal{K}}\mathbf{W}_{\mathcal{K}}^{def},
\]
and offsets and attention weights are predicted from the queries alone,
\[
\Delta=\mathcal{G}_{\Delta}(\mathbf{Q})\in\mathbb{R}^{M\times P}, \qquad
\mathbf{A}=\mathcal{G}_{A}(\mathbf{Q})\in\mathbb{R}^{M\times P}.
\]
Aggregation then samples keys relative to the current center \(c\) and width \(w\):
\[
\mathbf{S}
=
\sum_{p=1}^{P}\left(\mathbf{A}_{:,p}\;\mathbf{K}[c+w\odot\Delta_{:,p}]\right)\in\mathbb{R}^{M\times F}.
\]
The paper argues that this is poorly suited to VTG cross-attention because the offsets are predicted from query embeddings that lack real contextual access to the key-value space [2507.07744].

RDSA addresses this by making offset prediction reference-aware and video-derived. Given current references \(\mathbf{R}^{\ell}\in\mathbb{R}^{M\times 2}\) and dense embeddings \(\mathbf{D}^{\ell}\in\mathbb{R}^{T\times F}\), the model first enriches dense embeddings with a CNN, samples left, center, and right reference embeddings, concatenates them, and forms an alternative query:
\[
\hat{\mathbf{X}}_{\mathcal{Q}} = CNN(\mathbf{D}^{\ell})[l,c,r], \qquad
\hat{\mathbf{Q}}=\hat{\mathbf{X}}_{\mathcal{Q}}\mathbf{W}_{\mathcal{Q}}^{def},
\]
with
\[
\hat{\mathbf{X}}_{\mathcal{Q}}\in\mathbb{R}^{M\times 3F}.
\]
Offsets and attention scores are then predicted from this reference-based query,
\[
\hat{\Delta}=\mathcal{G}_{\Delta}(\hat{\mathbf{Q}}), \qquad
\hat{\mathbf{A}}=\mathcal{G}_{A}(\hat{\mathbf{Q}}),
\]
and deformable aggregation reuses the same sampling equation with \(\hat{\Delta}\) and \(\hat{\mathbf{A}}\) [2507.07744].

The mechanism is described as improving context modeling in three ways: key-space awareness, boundary-sensitive reasoning via left and right reference embeddings, and local neighborhood context via a 2-layer CNN [2507.07744]. Empirically, on QVHighlights validation, RDSA improves over standard deformable cross-attention from \(58.19\) to \(60.90\) on \(R1@0.7\), from \(54.27\) to \(55.60\) on Avg mAP, and from \(43.26\) to \(44.00\) on HD mAP; on long actions, it improves mAP from \(56.92\) to \(59.76\) [2507.07744]. The best sampling strategy is left-center-right with CNN, yielding Avg mAP \(55.60\), HD mAP \(44.00\), and HIT@1 \(72.00\) [2507.07744].

## 5. Training objectives, inference, and implementation profile

HD is computed from the final dense embedding \(\mathbf{D}^K\) and a pooled text embedding \(T^{pool}\) using frame-wise cosine similarity,
\[
\hat{Y}^s = cos\_sim(D^K,T^{pool})
=
\frac{\sum_{j=1}^{F} D^K_j T^{pool}_j}{\|D^K\|\,\|T^{pool}\|}.
\]
Training uses an InfoNCE / SampledNCE ranking-style loss,
\[
\mathcal{L}_{HD}=\lambda_{0}\mathcal{L}_{InfoNCE},
\]
with the supplementary text describing the objective as ranking positive frames higher [2507.07744].

MR uses DETR-style bipartite matching via the Hungarian algorithm. The classification term is focal loss,
\[
\mathcal{L}_{cls}
=
-\frac{1}{M}\sum_{m=1}^{M}\alpha(1-\hat{p}_m)^\gamma \log(\hat{p}_m),
\]
the boundary term is
\[
\mathcal{L}_{L1}
=
\frac{1}{M^*}\sum_{i=1}^{M^*}|\hat{Y}^m_i-Y^m_i|,
\]
the localization-overlap term is
\[
\mathcal{L}_{IoU}
=
1-\frac{\sum_{i=1}^{M^*}\text{IoU}(\hat{Y}^m_i,Y^m_i)}{M^*},
\]
and the actionness term is
\[
\mathcal{L}_{act}
=
\frac{1}{M}\sum_{i=1}^{M}
\left|
\hat{Y}^a_i-\max_{j}^{M^*}\big(\text{IoU}(R^\ell_i,Y^m_j)\big)
\right|.
\]
The overall MR loss is
\[
\mathcal{L}_{MR}
=
\lambda_{1}\mathcal{L}_{act}
+
\sum_{\ell=1}^{K}
\lambda_{2}\mathcal{L}_{cls}^{\ell}
+
\lambda_{3}\mathcal{L}_{l1}^{\ell}
+
\lambda_{4}\mathcal{L}_{IoU}^{\ell},
\]
with all terms except \(\mathcal{L}_{act}\) optimized across refinement levels to accelerate convergence. Alignment is further regularized by two SampledNCE / InfoNCE-based contrastive terms summarized as \(\mathcal{L}_{align}\), and the final objective is
\[
\mathcal{L}
=
\lambda_{5}\mathcal{L}_{HD}
+
\lambda_{6}\mathcal{L}_{MR}
+
\lambda_{7}\mathcal{L}_{align}.
\]
Default loss weights are mostly simple: \(\lambda_{L1}=1\), \(\lambda_{IoU}=1\), \(\lambda_{act}=1\), \(\lambda_{cls}=1\), with saliency- and alignment-related coefficients around \(0.1\) [2507.07744].

At inference time, each query outputs predicted boundaries, class probability \(\hat p\), and actionness \(\hat Y^a\). These are combined into
\[
\hat{C}=\sqrt{\hat{p}\cdot\hat{Y}^a},
\]
and final moments are selected with soft-NMS. HD scores are directly the frame-wise cosine similarities \(\hat{Y}^s_t=cos\_sim(D_t^K,T^{pool})\) [2507.07744].

The implementation profile is tightly specified. The best configuration uses hidden dimension \(256\), \(K=4\) refinement levels, sinusoidal positional encoding, dropout \(0.5\), DropPath \(0.25\), \(8\) attention heads, FFN expansion ratio \(4\), ReLU, LayerNorm with PostNorm, and \(4\) deformable sampling points. The sparse-branch RDSA context module is a 2-layer CNN with hidden dimension \(256\), and concatenated sampled tokens are projected to a \(64\)-dimensional latent before offset and attention prediction. The number of queries is \(30\) on QVHighlights and Charades-STA and \(5\) on TACoS. The classification head is a 1-layer MLP; the regression and actionness heads are 3-layer MLPs; actionness operates on RoI pooled features with RoI size \(16\) [2507.07744].

## 6. Quantitative profile, efficiency, and empirical interpretation

SDST is reported to achieve highly competitive or SOTA VTG results while remaining strongly parameter-efficient [2507.07744]. On QVHighlights, the main parameter comparison is SG-DETR at \(15.0\)M, Flash-VTG at \(10.9\)M, R\(^2\)-Tuning at \(2.7\)M, and SDST at \(4.1\)M. Relative to SG-DETR, SDST uses about \(27.3\%\) of the parameters, corresponding to about \(72.7\%\) fewer parameters, which matches the paper’s “up to 73% reduction” claim [2507.07744].

| Dataset | SDST result | Comparative note |
|---|---:|---|
| QVHighlights val | Avg mAP 55.60, HD mAP 44.00, HIT@1 72.00, Params 4.1M | Nearly matches SG-DETR Avg mAP 55.64 with 15.0M params |
| QVHighlights test | Avg mAP 53.31, HD mAP 43.40, HIT@1 69.13, Params 4.1M | Very close to SG-DETR while clearly outperforming R\(^2\)-Tuning |
| Charades-STA | R@0.5 72.0, R@0.7 52.6, mIoU 61.2 | \(+2.71\) R@0.7 and \(+2.06\) mIoU over prior SOTA |
| TACoS | R@0.5 44.5, R@0.7 32.3, mIoU 42.2 | \(+2.39\) R@0.7 and \(+1.27\) mIoU over SG-DETR, with slight drop in R@0.5 |

On QVHighlights validation, SDST records \(R1@0.5=73.68\), \(R1@0.7=60.90\), \(mAP@0.5=73.52\), \(mAP@0.75=57.42\), Avg mAP \(55.60\), HD mAP \(44.00\), and HIT@1 \(72.00\) [2507.07744]. In the fair comparison using InternVideo2 features only, it is statistically on par with SG-DETR while surpassing R\(^2\)-Tuning by \(+3.82\) Avg mAP on MR and \(+2.21\) HD mAP [2507.07744]. On Charades-STA it reaches \(72.0\) at \(R@0.5\), \(52.6\) at \(R@0.7\), and \(61.2\) mIoU; on TACoS it reaches \(44.5\) at \(R@0.5\), \(32.3\) at \(R@0.7\), and \(42.2\) mIoU [2507.07744].

Efficiency is favorable but not uniformly dominant. On QVHighlights with batch size \(32\), SDST uses \(4.1\)M parameters, \(3.4\) GB memory, and \(4.16\) iterations per second, compared with \(2.4\) GB and \(5.55\) it/s for R\(^2\)-Tuning and \(2.3\) GB and \(5.2\) it/s for Flash-VTG [2507.07744]. The paper therefore does not present SDST as the absolute lightest in memory or runtime; rather, its central efficiency claim is strong parameter efficiency relative to high-performing InternVideo2-based baselines [2507.07744].

Ablations clarify where the gains originate. Parameter sharing across refinement levels reduces the model from \(12.43\)M to \(4.10\)M parameters while improving Avg mAP from \(54.10\) to \(55.60\) [2507.07744]. Removing RDSA causes a severe collapse in sparse-branch localization, dropping \(R1@0.7\) to \(30.71\) and Avg mAP to \(29.09\), which indicates that the sparse branch depends critically on dense-branch video conditioning [2507.07744]. Conditioning the sparse branch with progressively richer dense features also improves performance, with the best result obtained from the post-\(CA+SA+FFN\) dense representation rather than raw visual features or earlier dense states [2507.07744]. Reusing InternVideo2’s frozen AdaptivePool across intermediate layers is likewise decisive: on QVHighlights validation it outperforms CLS pooling by \(+5.07\) Avg mAP, \(+2.99\) HD mAP, and \(+7.74\) HIT@1 [2507.07744].

## 7. Scope, terminology, and common points of confusion

The term “Sparse-Dense” in SDST refers to the decomposition of VTG computation into a sparse branch for moment proposal refinement and a dense branch for frame-level multimodal and temporal refinement [2507.07744]. It does not denote the kind of explicit sparsity machinery absent from the original side-tuning literature, where the side network is dense and additively fused with a frozen base [1912.13503]. Nor is it identical to parameter-efficient dense side-branch methods such as QST, whose efficiency derives from quantization, reduced width, and low-rank or pooling downsamplers rather than from a sparse proposal structure [2401.07159].

A second common confusion is to treat SDST as a generic PEFT wrapper around a frozen video backbone. The paper’s results indicate that its performance depends on a specific conjunction of design choices: anchor-free sparse decoding for MR, dense frame refinement for HD, RDSA for reference-grounded video conditioning, recurrent use of intermediate InternVideo2 layers, and reuse of the frozen AdaptivePool module [2507.07744]. In that sense, SDST is not merely side-tuning applied to VTG; it is a VTG-specific side-tuning architecture whose sparse-dense split follows the structural asymmetry between MR and HD.

A plausible implication is that SDST provides a particularly informative comparison point for future VTG adaptation methods: it retains the frozen-backbone modularity associated with side-tuning, but shows that task-structured side branches can substantially outperform frame-centric dense side-tuning baselines without requiring full fine-tuning [1912.13503], [2507.07744].

Source: https://www.emergentmind.com/topics/sparse-dense-side-tuner-sdst