RepCaM: Re-parameterization for Content-Aware Video SR

• The paper demonstrates that RepCaM significantly reduces per-chunk parameter growth by re-parameterizing multi-branch structures into a single shared backbone at inference.
• RepCaM employs a multi-branch convolution design with cascaded 1×1 filters to capture diverse content features during training while maintaining an exactly re-parameterizable structure.
• Experimental results show consistent performance gains and reduced bandwidth usage across various backbones, affirming its practical benefits for neural video delivery.

Re-parameterization Content-aware Modulation (RepCaM) is a content-aware super-resolution mechanism for neural video delivery in which chunk-level adaptation is learned during training by an over-parameterized convolutional structure and then exactly collapsed into the original backbone at inference time. In RepCaM++, RepCaM is the core module used to modulate all video chunks with one shared super-resolution model, thereby avoiding the parameter accumulation that arises when a separate model, or chunk-specific partial parameters, must be delivered for each chunk (Zhang et al., 17 Sep 2025). Within the broader lineage of neural video delivery, RepCaM can be understood as a structural re-parameterization response to earlier content-aware schemes such as Content-aware Feature Modulation (CaFM), which reduced but did not eliminate per-chunk parameter growth (Liu et al., 2021).

1. Problem setting and historical position

Neural Video Delivery (NVD) divides a video into $N$ chunks; the server trains content-aware super-resolution models and streams low-resolution chunks together with model information to the client. Earlier content-aware approaches followed two dominant patterns. One streams a separate model per chunk, with delivery cost $N\times(S+L)$. The other shares a backbone but attaches chunk-specific partial parameters whose size still grows linearly with the number of chunks. As video duration increases, or when multiple scenes and dynamics are present, these partial-parameter designs accumulate linearly with length, increasing transmission cost and often degrading performance as parameter-sharing assumptions break (Zhang et al., 17 Sep 2025).

CaFM is the principal antecedent in this line. It applies affine, channel-wise modulation after convolution, using chunk-specific parameters $\alpha_\ell^{(i)}$ and $\beta_\ell^{(i)}$, and reports that each video chunk requires less than $1\%$ of the original parameters to be streamed, with the 1×1 CaFM payload for EDSR reported as approximately $0.58$–$0.66\%$ depending on scale (Liu et al., 2021). That design already established the practical value of overfitting chunk-specific statistics while sharing a backbone, but it still requires a separate CaFM blob per chunk and therefore retains linear growth with video length.

RepCaM addresses this specific bottleneck. Rather than preserving chunk-specific parameter sets at inference, it trains a single multi-branch parallel–cascade convolutional structure jointly across all chunks and then uses inference-time structural re-parameterization to fold the expanded training structure back into the original backbone. The resulting delivery cost becomes $S+N\times L$, or $S+N\times(L+T)$ when combined with Transparent Visual Prompt (TVP) (Zhang et al., 17 Sep 2025). A plausible implication is that RepCaM shifts the content-awareness burden from delivered chunk-specific weights to training-time representational capacity.

2. Formal definition and exact re-parameterization

RepCaM augments a standard super-resolution backbone by replacing core convolution layers in residual modules with a horizontally oriented multi-branch RepCaM convolution. Each branch consists of a $3\times3$ convolution preceded by a varying-length cascade of $N\times(S+L)$0 convolutions. If $N\times(S+L)$1 is a frame and $N\times(S+L)$2 an intermediate feature, the training-time RepCaM convolution is defined as

$N\times(S+L)$3

where $N\times(S+L)$4 is the number of parallel branches, $N\times(S+L)$5 is a $N\times(S+L)$6 Conv, and $N\times(S+L)$7 is a sequence of $N\times(S+L)$8 consecutive $N\times(S+L)$9 Convs (Zhang et al., 17 Sep 2025). In the main instantiation, RepCaM uses a tri-branch design.

The purpose of the varying $\alpha_\ell^{(i)}$0 cascades is to modulate features at different granularities while maintaining an exactly re-parameterizable linear structure. The paper states that training-time over-parameterization improves content modulation capacity and helps capture empirically linear relationships between chunk-specific feature maps. This is significant because exact inference-time fusion requires that the train-time operations remain algebraically reducible to the baseline convolution graph (Zhang et al., 17 Sep 2025).

The re-parameterization has two stages. First, each branch’s cascade is condensed into a single equivalent $\alpha_\ell^{(i)}$1 convolution. Denoting a $\alpha_\ell^{(i)}$2 convolution by a fourth-order tensor $\alpha_\ell^{(i)}$3 and a bias by $\alpha_\ell^{(i)}$4, the paper writes the output of a $\alpha_\ell^{(i)}$5–$\alpha_\ell^{(i)}$6 cascade as

$\alpha_\ell^{(i)}$7

and then states that, by convolution linearity, an equivalent single $\alpha_\ell^{(i)}$8 Conv exists,

$\alpha_\ell^{(i)}$9

Second, after condensing each branch, the parallel branches are fused by concatenating kernels along the output-channel axis. For two branches with kernels $\beta_\ell^{(i)}$0 and $\beta_\ell^{(i)}$1 and biases $\beta_\ell^{(i)}$2 and $\beta_\ell^{(i)}$3, the concatenation yields $\beta_\ell^{(i)}$4 and $\beta_\ell^{(i)}$5 such that the whole multi-branch block coalesces to a single $\beta_\ell^{(i)}$6 Conv under identical branch configurations (Zhang et al., 17 Sep 2025).

A frequent misconception is to treat RepCaM as a FiLM-style feature modulation layer. In RepCaM++, this is explicitly not the case: modulation is achieved structurally via branch-specific $\beta_\ell^{(i)}$7 cascades, and the method does not employ feature-wise scaling/shift $\beta_\ell^{(i)}$8, normalization, or gating beyond standard convolution and bias operations (Zhang et al., 17 Sep 2025).

3. Backbone integration and Transparent Visual Prompt

RepCaM is integrated into widely used image super-resolution backbones, including EDSR, ESPCN, VDSR, and SRCNN. The main experimental backbone is EDSR with 16 residual blocks and no BatchNorm. In that setting, the “traditional Conv layer within the residual module” is replaced by the tri-branch RepCaM convolution, while residual connections of the backbone are preserved (Zhang et al., 17 Sep 2025).

The method is explicitly per-frame rather than temporally aligned. Temporal alignment and aggregation components such as EDVR or BasicVSR are not required; RepCaM operates on individual frames but is trained jointly across all chunks for content-aware modulation. Chunk-wise modulation is realized implicitly by the cascaded $\beta_\ell^{(i)}$9 filters learning linear relationships across chunk features, and there are no per-chunk parameter sets at inference (Zhang et al., 17 Sep 2025). This distinguishes RepCaM from video restoration frameworks that rely on motion compensation or recurrent temporal propagation.

RepCaM++ adds Transparent Visual Prompt (TVP) to improve fine-detail recovery. TVP is a minimal, zero-initialized image-level prompt, reported as less than $1\%$0 of parameters, that is added element-wise to the input frame within a centered spatial window. Its parameterization uses a prompt tensor $1\%$1, an input frame $1\%$2, and a prompted input $1\%$3 such that the prompt is activated only inside the centered region $1\%$4 (Zhang et al., 17 Sep 2025). TVP is additive rather than concatenative or multiplicative, and the paper states that there is no gating.

TVP is trained jointly with RepCaM under an $1\%$5 reconstruction objective; the paper denotes this simply as “We utilize L1 loss” and gives the corresponding gradients with respect to $1\%$6 (Zhang et al., 17 Sep 2025). Zero initialization is used to avoid blocking pixels and to align with the overfitting objective, and a single TVP is reused across multiple frames to reduce redundancy. This suggests that TVP is intended as a transparent local bias rather than a separate representational branch.

4. Training, delivery, and optimization

Server-side training divides the video into chunks, for example nine chunks, and trains a single RepCaM-enhanced model end-to-end on all chunks. Video Patch Sampling (VPS) can be applied to prioritize informative patches. The reported training details are HR/LR patches of $1\%$7, scale factors $1\%$8, $1\%$9, and $0.58$0, Adam optimizer with $0.58$1, $0.58$2, and $0.58$3, learning rate $0.58$4 with decay at 200 epochs, evaluation on 1/10 sampled frames, and training on an RTX 3090 GPU (Zhang et al., 17 Sep 2025).

The client-side deployment path is simpler. The server transmits low-resolution chunks plus the single re-parameterized super-resolution model, and, in RepCaM++, the tiny TVP. Re-parameterization collapses the training-time branches back into the baseline backbone, so there is no parameter accumulation with chunk length at delivery (Zhang et al., 17 Sep 2025).

Scheme	Delivery accounting	Interpretation
Separate per-chunk models	$0.58$5	Full model delivered for every chunk
RepCaM	$0.58$6	One shared model, no extra branch parameters at inference
RepCaM++	$0.58$7	RepCaM plus TVP

The optimization objective remains deliberately narrow. RepCaM++ uses only $0.58$8 reconstruction; no perceptual, adversarial, or temporal losses are added (Zhang et al., 17 Sep 2025). Relative to earlier CaFM-based training, this preserves the same broad patch-based content-aware regime—CaFM also used $0.58$9 HR patches, bicubic downsampling, Adam with $0.66\%$0, and $0.66\%$1 loss—but RepCaM eliminates the need to preserve chunk-specific affine parameters at inference (Liu et al., 2021).

5. Experimental performance and efficiency

The main evaluation uses VSD4K with six scenes—vlog, game, interview, city, sports, and dance—and durations of 15s, 30s, 45s, 1min, 2min, and 5min. High-resolution data are at 1080p, and low-resolution inputs are generated by bicubic downsampling. Metrics are PSNR, SSIM, and LPIPS (Zhang et al., 17 Sep 2025).

Across backbones including ESPCN, SRCNN, VDSR, and EDSR, RepCaM++ consistently outperforms NAS, CaFM, and EMT. Specific examples reported for 15s videos include ESPCN game-15s PSNR of $0.66\%$2 at $0.66\%$3, $0.66\%$4 at $0.66\%$5, and $0.66\%$6 at $0.66\%$7; SRCNN inter-15s PSNR of $0.66\%$8, $0.66\%$9, and $S+N\times L$0; and EDSR vlog-15s PSNR of $S+N\times L$1, $S+N\times L$2, and $S+N\times L$3 for the same scales (Zhang et al., 17 Sep 2025). For 45s videos, the reported EDSR game-45s scores are $S+N\times L$4 at $S+N\times L$5, $S+N\times L$6 at $S+N\times L$7, and $S+N\times L$8 at $S+N\times L$9, while vlog-45s reaches $S+N\times(L+T)$0, $S+N\times(L+T)$1, and $S+N\times(L+T)$2.

Perceptual and structural quality metrics follow the same trend. In Table 7, EDSR city-45s reaches LPIPS $S+N\times(L+T)$3 of $S+N\times(L+T)$4 and SSIM $S+N\times(L+T)$5 of $S+N\times(L+T)$6, both reported as best among baselines, while ESPCN sports-45s reaches LPIPS $S+N\times(L+T)$7 of $S+N\times(L+T)$8 (Zhang et al., 17 Sep 2025). Figure 1 further reports that RepCaM achieves higher PSNR for a given delivery size than NAS, CaFM, EMT, SRVC, and DVC.

Bandwidth and consistency measurements are central to the method’s claim. On VSD4K-45s, RepCaM++ achieves LR+Model size of 3.89 MB, decomposed as $S+N\times(L+T)$9 MB, compared with NAS at 6.05 MB and EMT at 3.96 MB, while improving PSNR (Zhang et al., 17 Sep 2025). Table 6 reports average frame smoothness across six scenes, measured as MSE$3\times3$0, of 3.320 for RepCaM++ versus 3.542 for EMT, where lower is better.

Long-duration results indicate that the method’s advantage is retained as chunk counts increase. For EDSR game-1min, the paper reports PSNR of $3\times3$1 at $3\times3$2, $3\times3$3 at $3\times3$4, and $3\times3$5 at $3\times3$6; for game-5min the values are $3\times3$7, $3\times3$8, and $3\times3$9, with RepCaM++ maintaining its lead as duration increases (Zhang et al., 17 Sep 2025). At matched storage budgets, it also surpasses H.264 and H.265 across all scenes and scales; one reported example is game-45s $N\times(S+L)$00 PSNR of $N\times(S+L)$01 versus H.265 at $N\times(S+L)$02.

Efficiency measurements are equally explicit. FLOPS on LR $N\times(S+L)$03 are 9.18G, the same as the EDSR baseline. Training GPU memory is 6421 MB, and training time per iteration at batch 256 is 0.0046 s, higher than EMT’s 0.0027 s because of the multi-branch over-parameterization. In deployment, however, the branches are removed, and real-world testing reports 29 FPS for 270p→1080p on Huawei P50 Pro, satisfying the stated mobile latency criteria of less than 500 ms and more than 20 FPS (Zhang et al., 17 Sep 2025).

6. Relation to prior methods, misconceptions, and limitations

RepCaM’s nearest predecessor in neural video delivery is CaFM. CaFM modulates each chunk using affine, channel-wise parameters $N\times(S+L)$04 and $N\times(S+L)$05 applied after convolution, and it can be exactly folded into chunk-specific convolution weights and biases for a fixed chunk (Liu et al., 2021). The decisive difference is that CaFM still requires chunk-specific parameters at delivery, whereas RepCaM learns a shared multi-branch modulation structure across all chunks and then removes the extra branches entirely at inference (Zhang et al., 17 Sep 2025).

In the broader structural re-parameterization literature, ASR formalizes a related but distinct problem: conventional channel attention uses input-dependent multiplicative gates $N\times(S+L)$06, and because $N\times(S+L)$07 depends on the inference input, there is generally no fixed parameter transformation that folds the modulation into static weights for all inputs (Zhong et al., 2023). RepCaM sidesteps this obstacle by avoiding multiplicative attention altogether. This suggests that RepCaM is better viewed as structurally re-parameterized content modulation rather than re-parameterized attention.

Several misconceptions are explicitly ruled out by the RepCaM++ formulation. RepCaM is not FiLM-style $N\times(S+L)$08 modulation; it does not introduce normalization or gating beyond standard convolution and bias; and it is not a temporal alignment architecture such as EDVR or BasicVSR (Zhang et al., 17 Sep 2025). Its content awareness is learned jointly across chunks but executed at inference by a plain, re-parameterized convolutional backbone.

The limitations are also clearly delineated. Training requires more GPU memory and time because of multi-branch over-parameterization, although this overhead is confined to the server side. The method is primarily per-frame super-resolution and does not explicitly model temporal alignment or motion compensation, which may limit temporal consistency under extreme motion, even though the unified model and TVP improve average smoothness. Generalization to unseen scenes or codecs can weaken because TVP is zero-initialized and overfits chunk-specific patterns; gains may diminish under strongly out-of-distribution content without retraining or adaptive prompts. For very long videos with frequent scene cuts, prompt placement and size and branch count become more sensitive, and excessive prompts can introduce redundancy (Zhang et al., 17 Sep 2025).

The ablation studies make those trade-offs concrete. Increasing the branch count from 1 to 3 improves performance substantially—for example, game-45s $N\times(S+L)$09 PSNR rises from 42.55 to 43.61 dB—whereas 4 or 5 branches yield smaller incremental gains. TVP ablations report the best performance with 9 prompts of size $N\times(S+L)$10; too many prompts, such as 27, or an oversized prompt of $N\times(S+L)$11 can introduce redundancy or degrade focus on high-frequency details. VPS improves convergence by reaching higher PSNR at fewer epochs (Zhang et al., 17 Sep 2025). Together, these findings define RepCaM as a training-heavy but inference-neutral mechanism for content-aware video super-resolution, with its principal contribution lying in exact inference-time elimination of the adaptation structure rather than in dynamic online modulation.