Sparse to Dense Lifting (S2D) Overview

Updated 4 July 2026

Sparse to Dense Lifting (S2D) is a versatile methodological framework that converts sparse signals into dense outputs for improved reconstruction, segmentation, and control.
It integrates various implementations, from monocular SLAM and motion segmentation to mesh deformation and reinforcement learning, by leveraging reliable sparse cues.
S2D employs learning techniques, confidence weighting, and geometric propagation to create dense intermediate representations that serve as robust priors for downstream tasks.

Searching arXiv for the specified S2D-related papers to ground the article in the cited literature. arxiv_search(query="all:\"Sparse to Dense Lifting\" OR all:\"Sparse2Dense\" OR id:(Tang et al., 2019) OR id:(Kardoost et al., 2020) OR id:(Ji et al., 11 Mar 2026) OR id:(Sick et al., 16 Dec 2025)", max_results=10) arxiv_search(query="(Tang et al., 2019, Kardoost et al., 2020, Ji et al., 11 Mar 2026, Sick et al., 16 Dec 2025)", max_results=10) Sparse to Dense Lifting, commonly abbreviated S2D and in some papers written as Sparse2Dense, denotes a family of formulations in which sparse observations, sparse supervision, or sparse rewards are transformed into denser geometric, semantic, or training signals. In the arXiv literature, the label has been used for a monocular SLAM system that turns a sparse, geometrically optimized odometry estimate into a dense 3D reconstruction (Tang et al., 2019), a self-supervised motion segmentation method that learns dense masks from sparse trajectory labels (Kardoost et al., 2020), an unsupervised video instance segmentation pipeline that converts sparse keymasks into dense pseudo-labels (Sick et al., 16 Dec 2025), a facial mesh decoder that maps sparse landmark displacements to dense per-vertex deformation (Otberdout et al., 2021), a toddler-inspired sparse-to-dense reward transition in goal-oriented reinforcement learning (Park et al., 2024), and a minimal-input 3D Gaussian Splatting pipeline that lifts sparse geometry into dense novel-view guidance (Ji et al., 11 Mar 2026). This suggests that S2D is best understood as a recurrent methodological pattern rather than a single canonical algorithm.

1. Terminological scope and recurring formulation

The acronym has no field-independent fixed meaning. Its uses are unified by a common structural move: retain sparse anchors because they are informative, reliable, or easy to obtain, then learn or infer a denser object that is more useful for reconstruction, segmentation, control, or downstream prediction.

Formulation	Sparse source	Dense target or outcome
Sparse2Dense monocular SLAM	sparse optimized depth cloud from DSO	dense keyframe depth and global 3D model
Self-supervised motion segmentation	sparse trajectory labels	dense pixelwise segmentation
Unsupervised video instance segmentation	sparse keymasks	dense pseudo-label set
S2D-Dec for facial animation	68 landmark displacements	dense per-vertex displacement
Toddler-inspired RL transition	sparse reward	potential-based dense reward
Minimal-input 3DGS S2D	sparse point cloud and sparse views	dense novel-view guidance

A frequent misconception is to treat S2D as synonymous with dense depth completion. The literature is broader. In some works, the dense object is a 3D map or mesh; in others it is a segmentation mask, a pseudo-label set, or a reward curriculum. A second misconception is to assume that “dense” always means fully observed ground truth. Several S2D systems instead construct a dense intermediate that is confidence-weighted, teacher-generated, or geometrically regularized rather than directly measured.

2. Sparse2Dense in monocular SLAM and dense 3D reconstruction

The 2019 "Sparse2Dense: From direct sparse odometry to dense 3D reconstruction" system is a monocular SLAM pipeline built from four stages: learning-based prior generation, sparse direct visual odometry, sparse-to-dense reconstruction, and dense global fusion. When a new keyframe is created, a CNN predicts a depth or disparity map and a surface normal map from a single RGB image. The system is built on DSO, which performs direct photometric alignment and optimizes camera poses, sparse inverse depths of selected points, and affine brightness parameters. The CNN depth is used as a prior to initialize points and to reduce monocular scale ambiguity. The sparse optimized depth cloud from DSO is then projected into the current keyframe, and the predicted normals are used to propagate depth from sparse points to neighboring pixels that are likely to lie on the same local surface. Dense keyframe maps are further refined and fused with a surfel-based mapper based on ElasticFusion, producing a consistent global 3D model and supporting loop closure (Tang et al., 2019).

Its central geometric operation is a normal-guided local planar lifting. For neighboring pixels satisfying normal similarity and spatial proximity, the reconstructed depth is

$Z_{re}(u_i)= \frac{ \sum_{j\in \mathcal C_i} n_j^T n_i \, z_{ij} }{ \sum_{j\in \mathcal C_i} n_j^T n_i },$

with

$\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$

where the paper sets the normal similarity threshold to $0.95$ and the spatial window size to $5$ pixels. Only nearby pixels with similar normals contribute, and weighting by $n_j^T n_i$ emphasizes coplanar or nearly coplanar neighbors. The same work trains depth and normals in a tightly coupled manner through depth-to-normal and normal-to-depth consistency, using a combined objective with $\alpha=1.0$ , $\beta=0.1$ , and $\gamma=0.05$ . The paper states that depth-to-normal acts more as a training regularizer than as a runtime post-processing step. Empirically, tracking is evaluated with ATE and dense mapping with PCD, and the system outperforms original DSO, DSO with FCRN depth prior, CNN-SLAM, and other classical baselines in most sequences. On KITTI, the indoor-trained model does not generalize perfectly, but fine-tuning the CNN greatly improves translational accuracy (Tang et al., 2019).

3. Densifying sparse supervision in motion and video segmentation

In self-supervised motion segmentation, S2D is a two-stage pipeline. Sparse motion labels are first generated from point trajectories extracted from optical flow and clustered either with the classical translational motion model from Keuper et al. or with a Siamese-GRU trajectory affinity model. These trajectories are segmented by a minimum cost multicut solved with the Kernighan-Lin heuristic solver. A U-Net-style encoder-decoder is then trained on single frames, using only the sparse trajectory labels as supervision, and predicts a dense segmentation function $u\colon \Omega \to \{1,\ldots,K\}$ . The loss is restricted to sparse labeled pixels only. The method is self-supervised because the labels are automatically produced from the same video sequence, and sequence-specific because the dense model is trained per sequence. On DAVIS16 validation, the combination Siamese-GRU + U-Net densification reaches 66.2 IoU, compared with 55.3 IoU for Keuper et al. + Ochs densification and 57.7 IoU for Siamese-GRU + Ochs densification. On FBMS59, using sparse labels from Keuper et al. and FlowNet2, the method reaches 89.59 Precision, 68.29 Recall, and 77.56 F-measure (Kardoost et al., 2020).

The 2025 video instance segmentation method "S2D: Sparse-To-Dense Keymask Distillation for Unsupervised Video Instance Segmentation" uses a different densification logic. It starts from noisy single-frame unsupervised instance masks from CutS3D, identifies a sparse set of temporally reliable keymasks, and uses those anchors to train VideoMask2Former for implicit mask propagation. Temporal coherence is estimated by a deep motion prior from point tracking. For each mask $m_i^t$ , point trajectories define a visibility ratio

$\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 0

with visibility threshold $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 1, and visibility vectors are clustered by DBSCAN. Shared-visibility groups are further split by proxy propagate-and-match using the Point-Mask Jaccard Index

$\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 2

with $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 3. Training then combines sparse anchor supervision with Temporal DropLoss, which drops the mask loss on unannotated frames, and Sparse-To-Dense Distillation, in which a teacher VideoMask2Former is updated by EMA of the student weights. The method uses a dual-stage training process and reports 26.2 AP on YouTube-VIS 2019 versus 24.5 AP for VideoCutLER, 20.1 AP on YouTube-VIS 2021 versus 18.0 AP, and 50.6 J&F on DAVIS-all versus 44.9 J&F. When scaled to 13K real videos, it also outperforms a model trained on 1.3 million synthetic videos from VideoCutLER on YouTube-VIS 2019, YouTube-VIS 2021, YouTube-VIS 2022, OVIS, and Cityscapes (Sick et al., 16 Dec 2025).

Taken together, these two segmentation lines illustrate two distinct sparse-to-dense regimes. One learns dense masks from sparse labeled pixels in a sequence-specific manner; the other filters noisy framewise masks into sparse reliable anchors and then densifies supervision through teacher-student propagation. The shared principle is that sparse supervision is treated as higher quality than naively dense but noisy alternatives.

4. Sparse-to-dense lifting in mesh deformation and sparse-view 3D Gaussian Splatting

In dynamic 3D facial expression generation, S2D appears as S2D-Dec, a Sparse-to-Dense mesh Decoder. The overall framework separates expression dynamics over time from mesh deformation. A conditional GAN, Motion3DGAN, generates the motion of a sparse set of $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 4 3D landmarks over time using the Square Root Velocity Function (SRVF) representation, and S2D-Dec maps the resulting sparse landmark displacements to dense mesh displacements:

$\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 5

The paper uses $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 6 landmarks, so the input displacement has size $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 7. The output is a dense per-vertex displacement field $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 8, which is added to the neutral mesh. Architecturally, the decoder uses a fully connected layer of size 2688, followed by five spiral convolution layers with 64, 32, 32, 16, and 3 filters, each followed by up-sampling by a factor of 4. Training combines dense displacement reconstruction with a weighted mesh reconstruction loss, where each vertex weight is the inverse distance to the nearest landmark and the final loss is $\mathcal C_i = \{(x_j,y_j,z_j)\mid n_j^T n_i > \psi,\ |u_i-u_j|<\sigma,\ |v_i-v_j|<\sigma\},$ 9 with $0.95$0 and $0.95$1. On CoMA and D3DFACS, reported mean per-vertex errors are 0.52 ± 0.59 mm and 0.28 ± 0.31 mm for the expression-independent split, and 0.55 ± 0.62 mm and 0.27 ± 0.30 mm for the identity-independent split. In the ablation, using only $0.95$2 gives 1.27 ± 1.88 mm, adding unweighted mesh $0.95$3 gives 0.92 ± 1.33 mm, and adding weighted $0.95$4 gives 0.50 ± 0.56 mm (Otberdout et al., 2021).

The 2026 "S2D: Sparse to Dense Lifting for 3D Reconstruction with Minimal Inputs" extends the idea to sparse-view 3D Gaussian Splatting. It inserts an explicit lifting stage between sparse-view reconstruction and dense novel-view optimization. Input views are passed through a VFM, in practice $0.95$5, to obtain a point cloud; novel cameras are rendered from the point cloud; an intermediate 3DGS scene is optimized up to $0.95$6; and a one-step latent diffusion artifact fixer corrects the corrupted novel renders using a point-cloud rendering and a nearby reference image. Optimization from $0.95$7 to $0.95$8 uses both input views and fixed novel views, stabilized by random sample drop and weighted gradient. The paper sets the target sampling ratio $0.95$9 and the minimum novel-view pixel weight $5$0. The fixer is trained on about 2.1M image pairs from DL3DV-960 with a loss

$5$1

On static scenes, S2D reports 21.41 PSNR, 0.77 SSIM, 0.27 LPIPS, and 54.0 FID on 3DOVS @ 1 image, compared with 14.10, 0.47, 0.56, and 98.2 for DIFIX; 27.62 PSNR, 0.86 SSIM, 0.24 LPIPS, and 34.7 FID on RE10K @ 2 images; 20.97 PSNR, 0.70 SSIM, 0.35 LPIPS, and 58.1 FID on MIP360 @ 6 images; and 23.2 PSNR, 0.81 SSIM, 0.26 LPIPS, and 41.2 FID on DL3DV @ 6 images. On Waymo driving scenes, the interpolation setting reaches 31.44 PSNR and 0.07 LPIPS, and lane-shift FID is 46.1 for side 2m, 53.9 for side 3m, and 41.3 for up 1.5m (Ji et al., 11 Mar 2026).

5. Sparse-to-dense transition as reward curriculum in goal-oriented reinforcement learning

In goal-oriented reinforcement learning, S2D is not a geometric lifting mechanism but a Sparse-to-Dense reward transition. The 2024 paper formalizes training as a sequence of MDPs

$5$2

with reward supports satisfying

$5$3

and optimal policy sets satisfying

$5$4

The dense stage is constructed with Potential-Based Reward Shaping (PBRS),

$5$5

using a distance-based potential $5$6, so that optimal policy is preserved. The paper studies 2-stage or 3-stage transitions, with candidate schedules near the first quarter of training. Examples given in the appendix are 100k, 200k, 400k frames for LunarLander-V2, 50k, 100k, 250k updates for ViZDoom, and 1k, 2k, 3k episodes for CartPole-Reacher and UR5-Reacher (Park et al., 2024).

The evaluated domains are LunarLander-V2, CartPole-Reacher, UR5-Reacher, ViZDoom-Seen, ViZDoom-Unseen, RWARE Shelf-Delivery, and Gridworld, using SAC, A3C, PPO, and DQN. The paper reports that S2D consistently outperforms Only Sparse, Only Dense, Dense-to-Sparse (D2S), and intrinsic motivation baselines such as NGU and RND in sample efficiency and success rate, and that S2D achieves the best robustness in ViZDoom-Unseen. Its mechanistic analysis uses the Cross-Density Visualizer, which perturbs policy parameters as $5$7, and a sharpness measure

$5$8

with $5$9 and $n_j^T n_i$ 0. The reported outcome is that S2D-guided agents tend to converge to wider minima and that S2D shows more pronounced smoothing than D2S or static reward settings (Park et al., 2024). This use of the acronym broadens the meaning of sparse-to-dense lifting from spatial densification to curriculum design over reward density.

Several adjacent works implement sparse-to-dense mechanisms without presenting a standalone S2D method in the same sense. "Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D" does not introduce “Sparse to Dense Lifting” as a named separate framework, but it does implement a sparse-to-dense style lifting capability inside BoxerNet. Sparse point clouds or dense depth are projected into the image plane, aggregated by patch-wise median depth encoding, and converted into a dense patch feature grid $n_j^T n_i$ 1, with empty patches set to $n_j^T n_i$ 2. The system then lifts open-vocabulary 2D detections into 7-DoF 3D boxes and fuses them by multi-view geometric and semantic filtering. The paper explicitly characterizes this as a representation strategy rather than a depth completion network (DeTone et al., 6 Apr 2026).

A more abstract sparse-to-dense bridge appears in "Transformation of Dense and Sparse Text Representations". Its Semantic Backward Transformation (SBT) reconstructs a dense representation from a sparse semantic vector:

$n_j^T n_i$ 3

The paper frames this as a bidirectional bridge between dense neural features and sparse interpretable semantics, with SFT for dense-to-sparse, SBT for sparse-to-dense, and SCSS for semantic combination in sparse space (Hu et al., 2019).

The depth-estimation literature contains yet another variant. " $n_j^T n_i$ 4: Learnable Sparse Signal Superdensity for Guided Depth Estimation" expands each sparse LiDAR or Radar cue into a local patch and predicts a confidence map, aggregating confidence by a maximum operator and fusing the expanded guidance with the depth network output through

$n_j^T n_i$ 5

The module can be inserted at the input, cost-volume, or output stage and is described as a learned local completion operator for sparse guidance rather than full depth completion per se (Huang et al., 2021).

These related mechanisms clarify the limits of the term. S2D is not restricted to one modality, one supervision regime, or one mathematical trick. It may refer to local coplanar propagation in SLAM, sparse supervision with dense prediction in segmentation, landmark-to-mesh deformation, reward-density curricula, or dense tokenization of sparse geometric evidence. The common denominator is a design choice: preserve the advantages of sparse signals, then densify them in a controlled way through geometry, learning, confidence weighting, or distillation.