TVRN: Invertible Video Rescaling Framework

• TVRN is a compression-aware temporal video rescaling framework that downscales high-frame-rate videos for efficient transmission and later reconstructs them via an invertible mapping.
• It integrates components like MIMO-TWT, MIMO-VRN, and an HF reconstruction module to isolate and preserve essential high-frequency details in a reversible manner.
• The method employs a surrogate codec and compression-aware ranking to handle non-differentiable codecs and adapt to varying bitrate conditions, yielding superior RD performance.

TVRN is most explicitly the name of an end-to-end framework for compression-aware temporal video rescaling in which a high-frame-rate video is downscaled to a low-frame-rate sequence for efficient transmission and then upscaled after lossy compression by an invertible mapping that separates a transmitted low-frequency component from a reconstructed high-frequency component (Feng et al., 15 May 2026). In the supplied literature, the same label also appears in other contexts—TVR-Ranking, an overview that uses TVART/TVRN interchangeably, and tensor-based analysis of time-varying multilayer networks—so the term is context dependent rather than uniquely standardized (Liang et al., 2024, Harris et al., 2019, Billio et al., 2017).

1. Problem formulation in compression-aware temporal video rescaling

In its primary explicit usage, TVRN addresses a sender–receiver pipeline in which a high-frame-rate (HFR) video is temporally downscaled to a low-frame-rate (LFR) representation, compressed by a lossy codec, and later restored to HFR. The paper formalizes the input HFR sequence as $X=\{x_t\}_{t=1}^T$, the downscaler output as $Y$, the codec as $\mathcal{C}(\cdot)$ or $\mathcal{H}(\cdot)$, and the upscaler as $\mathcal{U}(\cdot)$. The central invertible mapping is written as

$$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$

where $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$ is the compressed LFR signal and $\hat{\mathbf{z}}$ is the reconstructed high-frequency component (Feng et al., 15 May 2026).

The formulation is motivated by the observation that existing frame-rate rescaling methods typically connect downscaling and upscaling only through training objectives, without fully exploiting their reciprocal structure. TVRN therefore treats the two operations as an explicitly invertible pair. The optimization objective is fidelity-guided and couples HFR reconstruction to an LFR target:

$$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$

where $\mathbf{x}_e$ denotes the even-indexed frames and $Y$0 controls bitrate and training stability. This construction makes the transmitted LFR sequence visually constrained while preserving the information needed for accurate HFR reconstruction.

A central difficulty is that industrial codecs are non-differentiable. TVRN addresses this not by removing the codec from the loop, but by preserving the real codec in the forward path and introducing a differentiable surrogate for backward propagation. That design choice defines the method’s compression-aware character and distinguishes it from frame interpolation pipelines that assume clean or differentiably degraded inputs.

2. Invertible architecture and temporal decomposition

TVRN is built from three coupled components: MIMO-TWT, MIMO-VRN, and an HF reconstruction module. MIMO-TWT, the Multi-Input Multi-Output Temporal Wavelet Transform, implements a prediction-first lifting scheme over a clip $Y$1. Frames are split into even and odd subsequences, $Y$2 and $Y$3, and decomposed as

$Y$4

Here $Y$5 is a lightweight VFI network, specifically UPR-Net, and $Y$6 is a stack of Dense3D-T blocks. The inverse MIMO-TWIT restores the original clip by reversing the lifting steps:

$Y$7

The paper emphasizes that this removes the need to pass explicit motion cues between forward and inverse mappings and isolates temporal high-frequency content in a structurally reversible way (Feng et al., 15 May 2026).

The second component, MIMO-VRN, is a stack of invertible affine coupling blocks operating jointly on the temporal low- and high-frequency branches. For block $Y$8, the forward update is

$Y$9

$\mathcal{C}(\cdot)$0

and the inverse is

$\mathcal{C}(\cdot)$1

$\mathcal{C}(\cdot)$2

All transforms $\mathcal{C}(\cdot)$3, $\mathcal{C}(\cdot)$4, $\mathcal{C}(\cdot)$5, and $\mathcal{C}(\cdot)$6 are Dense3D-T networks, and the exponential scales are normalized via sigmoid for stability.

The HF reconstruction module reconstructs $\mathcal{C}(\cdot)$7 at the temporal locations discarded by downsampling. From two LFR reference frames $\mathcal{C}(\cdot)$8 and $\mathcal{C}(\cdot)$9, a lightweight bi-directional flow estimator produces $\mathcal{H}(\cdot)$0 and $\mathcal{H}(\cdot)$1, which are scaled to target time $\mathcal{H}(\cdot)$2 as

$\mathcal{H}(\cdot)$3

An initial estimate is formed by the difference of forward-warped references,

$\mathcal{H}(\cdot)$4

and then refined with a context-aware U-Net fed by multi-level aligned contextual features.

Invertibility is not a secondary property but a defining structural constraint. MIMO-TWT/TWIT are reversible because they follow a lifting scheme, while MIMO-VRN is invertible because affine coupling yields a closed-form inverse and a triangular Jacobian with tractable log-determinant. TVRN does not optimize likelihood, but the bijective construction ensures that all information routed into $\mathcal{H}(\cdot)$5 is preserved up to the untransmitted high-frequency component that must later be reconstructed.

3. Surrogate codec, asymmetric restoration, and compression-aware ranking

Because $\mathcal{H}(\cdot)$6 is non-differentiable, TVRN introduces a surrogate network $\mathcal{H}(\cdot)$7 and optimizes

$\mathcal{H}(\cdot)$8

The surrogate is built from compression-aware invertible, or Q-Invertible, blocks and a feature collapse unit. At time $\mathcal{H}(\cdot)$9, it uses codec motion vectors from the bitstream to initialize a codec-inspired temporal decomposition:

$\mathcal{U}(\cdot)$0

$\mathcal{U}(\cdot)$1

Three Q-Invertible coupling layers then separate and condition subbands, while QP-aware channel attention adapts the computation to bitrate through a two-layer MLP that produces a scale vector $\mathcal{U}(\cdot)$2 and applies it as $\mathcal{U}(\cdot)$3 (Feng et al., 15 May 2026).

Forward propagation uses the actual codec output rather than the surrogate output. The paper states that forward values are reassigned from the real codec so that the network sees true distortions, while gradients still flow through $\mathcal{U}(\cdot)$4. Training alternates between updating $\mathcal{U}(\cdot)$5, the HF module, and the enhancement modules using $\mathcal{U}(\cdot)$6, and updating the surrogate using $\mathcal{U}(\cdot)$7.

A further extension makes the system asymmetric. Two identical STDR-based VQE modules are inserted, one before upscaling and one after upscaling. The purpose is to improve robustness under varying compression levels: the first branch helps at low bitrates, whereas the second is intended for cases in which upscaling decouples signals sufficiently that artifact suppression becomes easier afterward. These branches are guided by a Siamese compression encoder $\mathcal{U}(\cdot)$8 trained with a pairwise learning-to-rank objective,

$\mathcal{U}(\cdot)$9

with margin $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$0 and $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$1 determined by the QP ordering. The learned compression-aware features are fused into a spatial weight map

$$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$2

followed by

$$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$3

The reported comparison indicates that compression-aware features outperform naïve QP conditioning for a single VQE model across multiple QPs.

4. Training protocol, datasets, and evaluation methodology

The training data for TVRN are drawn from Vimeo90K-Septuplet, comprising 65K 7-frame clips, with evaluation on UCF-101, Vimeo90K test, and SNU-FILM Medium. The temporal group size is 7, and the downscale ratio is $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$4, meaning that two even frames are kept per three discarded. Training clips are also compressed with H.265 at $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$5 for ranker pretraining and fine-tuning, while QP is uniformly sampled from 17 to 27 during training. Training uses x265 with zero-latency tune, and evaluation additionally includes HEVC via x265, VVC via VVenC, and AV1 via SVT-AV1 (Feng et al., 15 May 2026).

Optimization proceeds in two stages. First, the compression encoder and ranker are pretrained with $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$6 for 100k iterations at learning rate $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$7. Second, $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$8, the HF module, and VQE are alternately trained with $$(\mathbf{y}, \mathbf{z}) = \mathcal{F}_\sharp(\mathbf{x}), \qquad \widehat{\mathbf{x}} = \mathcal{F}_\sharp^{-1}(\widetilde{\mathbf{y}}, \hat{\mathbf{z}}),$$9, while the surrogate is trained with $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$0 for 50k iterations. The optimizer is Adam with $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$1 and $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$2; the learning rate starts at $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$3 and is halved every 10k iterations. The reported batch size is 8 on $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$4 GTX 1080Ti, and runtime is approximately 90 hours.

Evaluation includes PSNR, SSIM, LPIPS, and VMAF, together with temporal consistency measures tOF and $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$5. The temporal optical-flow discrepancy is defined as

$\widetilde{\mathbf{y}}=\mathcal{C}(Y)$6

with optical flow estimated by RAFT. Rate–distortion behavior is assessed with RD curves and Bjøntegaard BD-rate relative to strong VFI baselines, while “Reference” denotes direct HFR encoding. This evaluation protocol places TVRN within the practical regime of industrial video compression rather than isolated frame interpolation benchmarks.

5. Quantitative performance, ablations, and limitations

At fixed bitrate, approximately $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$7 bpp, the reported HFR reconstruction scores show that TVRN-S attains PSNR/SSIM of $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$8 on UCF-101, $\widetilde{\mathbf{y}}=\mathcal{C}(Y)$9 on Vimeo90K, and $\hat{\mathbf{z}}$0 on SNU-FILM, with cost $\hat{\mathbf{z}}$1 s and $\hat{\mathbf{z}}$2 TFLOPs. The full TVRN reports $\hat{\mathbf{z}}$3, $\hat{\mathbf{z}}$4, and $\hat{\mathbf{z}}$5 on the same datasets, with cost $\hat{\mathbf{z}}$6 s, $\hat{\mathbf{z}}$7 TFLOPs, and $\hat{\mathbf{z}}$8. Against GIMM-VFI, identified as the best VFI baseline at $\hat{\mathbf{z}}$9 s and $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$0 TFLOPs, TVRN improves PSNR by $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$1 dB on UCF-101, $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$2 dB on Vimeo90K, and $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$3 dB on SNU-FILM, while the lightweight variant is more compute-efficient (Feng et al., 15 May 2026).

The BD-rate results are reported against GIMM-VFI under three industrial codecs. Under HEVC, the gains are listed as UCF-101 $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$4 (PSNR/SSIM), Vimeo90K $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$5, and SNU-FILM $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$6. Under VVC, the reported values are UCF-101 $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$7, Vimeo90K $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$8, and SNU-FILM $$\mathcal{L}_{\text{basic}} = \ell_1(\mathbf{x}, \widehat{\mathbf{x}}) + \lambda \cdot \ell_1(\mathbf{y}, \mathbf{x}_e),$$9. Under AV1, the corresponding values are UCF-101 $\mathbf{x}_e$0, Vimeo90K $\mathbf{x}_e$1, and SNU-FILM $\mathbf{x}_e$2. The paper characterizes TVRN as the first learnable temporal rescaling method to surpass top VFI under modern lossy codecs in RD performance.

Ablation studies isolate several design contributions. For surrogate gradient strategies on Vimeo90K, straight-through estimation gives BDBR $\mathbf{x}_e$3, a DenseBlock surrogate gives $\mathbf{x}_e$4, TINN without Q-Invertible blocks gives $\mathbf{x}_e$5, and the full TINN surrogate gives $\mathbf{x}_e$6. For MIMO-TWT placement on SNU-FILM, the MIMO-VRN baseline yields $\mathbf{x}_e$7 BDBR, adding MIMO-TWT after the main network reduces this to $\mathbf{x}_e$8, and placing MIMO-TWT before the network yields the best reported setting at $\mathbf{x}_e$9. HF reconstruction further improves performance, with a U-Net version at $Y$00 and a context-based version at $Y$01. For asymmetric restoration, a separate per-QP VQE gives $Y$02, a single VQE with QP gives $Y$03, and a single VQE with compression-aware features gives the best result, $Y$04.

The computational profile is also reported explicitly. For upscaling at resolution $Y$05 and a $Y$06 frame setting on RTX 3090, TVRN-S requires $Y$07 TFLOPs and $Y$08 s, whereas TVRN requires $Y$09 TFLOPs and $Y$10 s. End-to-end sender–receiver latency at 1080p on SNU-FILM with HEVC is $Y$11 s for TVRN-S and $Y$12 s for TVRN. Peak GPU memory is $Y$13 GB for TVRN-S and $Y$14 GB for TVRN, which the paper describes as feasible on 4 GB laptop GPUs.

The stated limitations are specific. Extreme long-range temporal motion can exceed the effective receptive field of flow-based HF reconstruction, producing flow misalignment in sparse sampling. Sensitivity to codec distribution shifts is mitigated by compression-aware features but not eliminated. The paper identifies dynamic grouping strategies adaptive to network conditions, further reduction of upscaling complexity for edge devices, and unified spatio-temporal rescaling as future directions.

6. Other documented uses of the label “TVRN”

The supplied literature uses “TVRN” in more than one way. The following usages are explicitly documented.

Usage	Description	Source
TVRN	“Invertible Neural Networks for Compression-Aware Temporal Video Rescaling”	(Feng et al., 15 May 2026)
TVRN	TVR-Ranking dataset for Ranked Video Moment Retrieval with imprecise queries	(Liang et al., 2024)
TVART/TVRN	Low-rank tensor model for time-varying autoregression	(Harris et al., 2019)
TVRN	Time-varying multilayer networks discussed through Bayesian Dynamic Tensor Regression	(Billio et al., 2017)

In the TVR-Ranking material, TVRN denotes a dataset and benchmark for Ranked Video Moment Retrieval. The task retrieves and ranks temporal moments drawn from a corpus of videos in response to natural-language queries, explicitly allowing imprecise queries and multiple relevant moments with graded relevance. The dataset contains 3,281 imprecise queries, of which 500 are for validation and 2,781 for test, and 94,442 consensus graded relevance labels over query–moment pairs. Its proposed evaluation metric is $Y$15, combining temporal localization accuracy with graded ranking quality (Liang et al., 2024).

In the overview of “Time-varying Autoregression with Low Rank Tensors,” the manuscript-level exposition states that it will use TVART/TVRN interchangeably. There, the method learns windowed linear autoregressive models for multivariate time series by coupling per-window system matrices through a low-rank third-order tensor with canonical polyadic decomposition. The parameter count is reduced from $Y$16 to $Y$17, and alternating minimization is used to estimate factor matrices under Tikhonov regularization and either total-variation or spline penalties on temporal factors (Harris et al., 2019).

In the Bayesian dynamic tensor-regression exposition, TVRN is used in connection with time-varying multilayer networks represented as tensor-valued responses $Y$18. The framework defines tensor autoregressive processes, uses PARAFAC low-rank parameterization for coefficient tensors, imposes global–local shrinkage priors on tensor marginals, and derives vectorized impulse response functions for shock propagation across nodes, layers, and time. In the empirical application, the response tensor has size $Y$19 and encodes international trade and bilateral outstanding capital across 10 countries over 2003–2016 (Billio et al., 2017).

Taken together, these usages show that “TVRN” is not a single stable term across arXiv-adjacent research communities. In current video-systems literature it most clearly denotes the invertible, codec-aware temporal rescaling framework of 2026, whereas in retrieval and tensor-modeling contexts it functions as a dataset label, an interchangeable shorthand, or a network-oriented notation.