Temporally-Guided Total Variation (TGTV)

• TGTV is a structure-aware regularization method that uses a temporally adjacent high-res guide image to generate adaptive weights for preserving spatial structure.
• It replaces uniform TV with directional penalization across horizontal, vertical, and diagonal gradients, ensuring smoothness in homogeneous areas and edge retention near structural details.
• Integrated within the TSSTF framework, TGTV improves noise robustness and yields higher PSNR and MSSIM compared to traditional TV regularization in spatiotemporal satellite image fusion.

Temporally-Guided Total Variation (TGTV) is a regularization function introduced for satellite-image spatiotemporal fusion within the Temporally-Similar Structure-Aware ST fusion framework (TSSTF). Its stated purpose is to promote spatial piecewise smoothness while preserving structural details by using a reference high spatial resolution image acquired on a nearby date as guidance. The construction rests on the explicit assumption that, when the reference and target dates are temporally close, the corresponding high-resolution images are expected to have similar spatial structure; TGTV therefore replaces blind, spatially uniform smoothing with directional, spatially adaptive penalization derived from a temporally adjacent guide image (Isono et al., 15 Aug 2025).

1. Problem setting and conceptual role

In spatiotemporal fusion, the target is a high-spatial-resolution image at a target date, estimated from an HR image at a nearby reference date, a low-spatial-resolution image at the reference date, and an LR image at the target date. The practical setting emphasized for TGTV is not idealized fusion, but fusion under degradation by Gaussian/random noise, sparse outliers, and missing values. Within that setting, the paper identifies a specific limitation of prior robust ST fusion methods, especially ROSTF: although they improve noise robustness by incorporating explicit observation models and regularization, they still suffer from loss of original spatial structure.

The central criticism is directed at standard total variation regularization. TV promotes piecewise smoothness and is effective for denoising, but it treats local differences in a spatially uniform way and does not explicitly use the intrinsic spatial structure of the underlying scene. The consequences stated for this setting are that meaningful fine structures can be mistaken for noise, edges may be over-penalized, the reconstructed HR image becomes oversmoothed, and unnatural artifacts can appear. TGTV is introduced precisely to address the question of whether noise-robust ST fusion can be achieved while preserving the original spatial structure.

Conceptually, TGTV differs from standard TV by evaluating neighborhood differences with adaptive weights derived from a temporally adjacent HR guide image. Standard TV penalizes all local gradients equally; TGTV penalizes local gradients selectively, depending on whether the corresponding location and direction appear smooth or edge-like in the guide. This makes it structure-aware, edge-aware, and temporally guided. The paper does not explicitly position TGTV against the broader literature on weighted TV or guided TV by name, but functionally it is described as a weighted vectorial TV whose weights are generated from a temporally adjacent HR image after noise attenuation.

2. Mathematical definition

The construction begins with four directional difference operators $$\mathbf{D}_1,\mathbf{D}_2,\mathbf{D}_3,\mathbf{D}_4 \in \mathbb{R}^{WHB \times WHB}$$:

$$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$

$$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$

The stacked operator is

$$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$

The use of four anisotropic directions is a notable design feature: TGTV is not restricted to horizontal and vertical gradients, but incorporates horizontal and diagonal-type neighborhoods.

Because the observed reference HR image $\mathbf{h}_r$ may be noisy, the guide is not formed directly from the raw observation. Instead, the paper defines a noise-attenuated guide image $h' \in \mathbb{R}^{N_h}$ as

$$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$

where $[\mathbf{h}_r]_b$ is the $b$-th spectral band of the reference HR image and $\mathrm{Med}(\cdot)$ is a median filter. The stated purpose is to suppress noise, especially sparse outliers and missing values, average across bands to obtain a grayscale guide, and preserve dominant spatial structure while reducing random noise.

Directional weights are then defined for each pixel location $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$0 and direction $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$1 by

$$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$2

with $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$3 controlling sensitivity to spatial differences. If the guide image is locally smooth in direction $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$4, then $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$5 is small and $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$6 is close to $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$7; if the guide has a strong edge in direction $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$8, then the weight becomes small. Large weights therefore encourage smoothing, whereas small weights preserve edges.

An additional selection step introduces a parameter $$[\mathbf{D}_1 \mathbf{x}]_{i,j,b} = x_{i+1,j,b} - x_{i,j,b}, \qquad [\mathbf{D}_2 \mathbf{x}]_{i,j,b} = x_{i+1,j-1,b} - x_{i,j,b},$$9. For each pixel $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$0, the $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$1 smallest values among $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$2 are set to zero. The paper states that this “effectively suppresses the influence of directions with minimal structural similarity.” This makes TGTV more selective than plain weighted TV because the most edge-like directions at a pixel may receive no penalty at all.

For each direction $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$3, the scalar weights are assembled into a diagonal matrix $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$4, replicated across the $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$5 spectral bands to form $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$6, and then stacked into the full weight matrix $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$7. The exact TGTV definition for any HR image $$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$8 is

$$[\mathbf{D}_3 \mathbf{x}]_{i,j,b} = x_{i,j-1,b} - x_{i,j,b}, \qquad [\mathbf{D}_4 \mathbf{x}]_{i,j,b} = x_{i-1,j-1,b} - x_{i,j,b}.$$9

The mixed $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$0 norm is vectorial TV across all spectral bands and all four directions, grouped at each spatial location. The paper states that this promotes smoothness where the guide image is homogeneous and preserves edges and structural discontinuities where the guide has strong spatial differences (Isono et al., 15 Aug 2025).

3. Embedding in TSSTF

TGTV is not presented as a standalone denoiser but as the principal regularizer in the full TSSTF model. The observation model includes noisy HR and LR observations with Gaussian noise and sparse noise, and the noiseless HR/LR relation is modeled as

$$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$1

where $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$2 is a blur operator, $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$3 is a downsampling operator, and $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$4 is a modeling error.

The companion mechanism to TGTV is the Temporally-Guided Edge Constraint (TGEC),

$$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$5

This uses the same weight matrix $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$6 as TGTV. In smooth regions, where the weights are large, edge agreement is enforced strongly; in edge regions, where the weights are small, edge-intensity differences are tolerated. The paper reports that $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$7 works best.

The full optimization problem minimizes two TGTV terms,

$$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$8

subject to TGEC, per-band mean-brightness consistency between $$\mathbf{D} := ({\mathbf{D}_1}^{\top}~{\mathbf{D}_2}^{\top}~{\mathbf{D}_3}^{\top}~{\mathbf{D}_4}^{\top})^{\top} \in \mathbb{R}^{4WHB \times WHB}.$$9 and $\mathbf{h}_r$0, $\mathbf{h}_r$1 fidelity constraints for the HR and LR observations, and $\mathbf{h}_r$2 constraints on sparse-noise variables $\mathbf{h}_r$3, $\mathbf{h}_r$4, and $\mathbf{h}_r$5. In this formulation, TGTV is the only regularization in the objective, while the remaining modeling assumptions are imposed as hard constraints.

The paper assigns distinct roles to the two TGTV terms. The term on $\mathbf{h}_r$6 denoises the reference HR image while preserving its structure. The term on $\mathbf{h}_r$7 regularizes the target HR estimate toward piecewise smoothness with the same structure-aware weighting. TGEC links the two HR images by constraining their weighted edge maps to be similar, while the data-fidelity constraints maintain consistency with observed HR/LR images and the blur/downsample observation model. Sparse-noise $\mathbf{h}_r$8 constraints explicitly absorb outliers and missing values. For fixed $\mathbf{h}_r$9, the paper states that the model is convex and convergence to the optimal solution is guaranteed.

4. Optimization and proximal structure

The constrained problem is rewritten with indicator functions and auxiliary variables:

$h' \in \mathbb{R}^{N_h}$0

together with variables for the observation-model constraints. This reformulation places the problem in a generic primal-dual splitting template.

The solver is a preconditioned primal-dual splitting method (P-PDS) with operator-norm-based variable-wise diagonal preconditioning (OVDP). TGTV is algorithmically convenient because it appears through convex, proximable mixed $h' \in \mathbb{R}^{N_h}$1 terms composed with the linear operator $h' \in \mathbb{R}^{N_h}$2. The proximal operator required for the mixed $h' \in \mathbb{R}^{N_h}$3 norm is

$h' \in \mathbb{R}^{N_h}$4

In the TGTV setting, the paper states that the same group soft-thresholding principle is applied to the split variable associated with $h' \in \mathbb{R}^{N_h}$5, and that Moreau’s identity is used to compute proximal maps of conjugates efficiently.

The primal updates for the HR images include gradient-adjoint contributions from the TGTV-related dual variables:

$h' \in \mathbb{R}^{N_h}$6

$h' \in \mathbb{R}^{N_h}$7

The dual updates for the TGTV terms advance $h' \in \mathbb{R}^{N_h}$8 and $h' \in \mathbb{R}^{N_h}$9 with $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$0 and $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$1, followed by the TGTV-specific shrinkage steps based on the mixed $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$2 proximal maps.

OVDP provides automatic step sizes. The paper gives

$$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$3

$$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$4

where $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$5. This removes the need for hand-tuning of primal-dual step sizes. For the practical solver, $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$6 is updated adaptively during optimization. The paper reports that $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$7 stabilizes after roughly 1000 iterations, update errors decrease almost monotonically, and PSNR flattens after about 2000 iterations (Isono et al., 15 Aug 2025).

5. Practical behavior and parameterization

TGTV preserves structure by modulating the local gradient penalty according to the guide image. In homogeneous regions of the guide, weights are large and the regularizer strongly suppresses variations. Near edges and structural discontinuities, weights are small and the regularizer relaxes smoothing. The stated consequence is preservation of edge sharpness, fine spatial structure, and boundaries between land-cover types, while still promoting piecewise smoothness elsewhere. Relative to standard TV, the regularization is therefore reference-aware rather than spatially uniform.

The paper provides recommended values for the direct TGTV hyperparameters. For the sensitivity parameter in

$$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$8

the recommended value is $$h' := \frac{1}{B}\sum_{b=1}^{B} \mathrm{Med}([\mathbf{h}_r]_b),$$9. Sensitivity analysis tested $[\mathbf{h}_r]_b$0 and found performance consistently peaked at $[\mathbf{h}_r]_b$1. For the directional-selection parameter, the recommended value is $[\mathbf{h}_r]_b$2; the paper evaluated $[\mathbf{h}_r]_b$3 and found PSNR consistently peaked at $[\mathbf{h}_r]_b$4, suggesting that selecting only two dominant directions is most effective. For the balance between reference and target TGTV terms, the recommended value is $[\mathbf{h}_r]_b$5, which regularizes the denoised reference HR image and the target HR image equally.

Two implementation details are emphasized. First, guide-image construction is explicitly two-stage: median-filter each spectral band, then average the filtered bands to obtain the grayscale guide image $[\mathbf{h}_r]_b$6. This is intended to suppress sparse noise and outliers before weight construction. Second, the same spatial weight map per direction is replicated across all bands, so TGTV is guided by a single grayscale structural image but applied jointly to all spectral channels. This reduces complexity and enforces band-consistent structure.

Experimentally, the paper reports that TSSTF performs comparably to state-of-the-art methods under noise-free conditions and outperforms them under noisy conditions. Across simulated and real data, TSSTF achieves the highest PSNR in all noisy simulated cases across all sites and usually the best MSSIM as well, while in real data it consistently gives best or near-best results. Compared with ROSTF, the paper states that ROSTF still suffers from oversmoothing and artifacts, especially around fine details and edges, because uniform TV cannot distinguish noise from intrinsic structure; TSSTF instead preserves edge sharpness and reduces artifacts. The paper does not provide a clean ablation table isolating “TGTV only” from “TGTV+TGEC,” so stronger attribution to TGTV alone would be inference rather than direct experimental proof (Isono et al., 15 Aug 2025).

6. Assumptions, limitations, and relation to broader spatiotemporal TV

The explicit assumption behind TGTV is that temporally adjacent HR images have similar spatial structure. The paper states that TGTV does not assume identical radiometry or equal edge magnitudes over time; rather, the model is guided by structural similarity, and TGEC allows spectral variations while enforcing consistency in edge locations. This suggests that the method is particularly appropriate when the time gap is short and changes are mainly spectral or intensity changes rather than structural reconfiguration.

The paper also states or strongly implies several limitations. TGTV depends on temporal structural similarity; if that assumption breaks, the guide may be misleading. It depends on the quality of the reference HR image; although the guide image is noise-attenuated via median filtering, severely corrupted or structurally inconsistent reference imagery may yield inaccurate weights. The practical solver does not strictly solve the original optimization problem because $[\mathbf{h}_r]_b$7 in TGEC is updated adaptively at each iteration. A plausible implication is that abrupt structural changes, registration errors, or a strong mismatch between reference geometry and target geometry could bias the target reconstruction toward outdated structure. A further plausible implication is that collapsing all bands into one grayscale guide may miss band-specific structures, although the paper does not discuss this as a stated problem.

As related background, the paper “High-Accuracy Total Variation for Compressed Video Sensing” develops a 3D spatiotemporal TV regularizer for compressed video sensing in which temporal information enters through a direct temporal derivative term computed by high-order-accuracy FIR differentiation filters rather than the usual $[\mathbf{h}_r]_b$8 difference filter (Hosseini et al., 2013). That formulation is highly relevant to TGTV as background on temporal regularization, but it is not temporal guidance in the strong TGTV sense: it does not use motion compensation, optical flow, frame warping, reference-frame guidance, or temporally adaptive weights. Its temporal component is a generic temporal smoothness or temporal-gradient sparsity prior, whereas TGTV uses adaptive weights derived from a nearby-date reference HR image to decide where smoothing should be strong and where edges should be preserved. This contrast places TGTV within a narrower class of structure-aware, temporally guided regularizers rather than the broader family of generic spatiotemporal TV models (Isono et al., 15 Aug 2025).