Outpainting Synthetic Data Generation Pipeline

• Outpainting-based synthetic data generation pipelines are methods that extend visual content beyond original boundaries using geometric, mask, and temporal cues.
• They apply iterative augmentation techniques across domains like 3D reconstruction, advertising imagery, lens flare removal, and video enhancement.
• These pipelines leverage diffusion models, GANs, and tailored architectures to preserve scene alignment while generating synthetic periphery for downstream applications.

Searching arXiv for the cited papers to ground the article in the latest records. An outpainting-based synthetic data generation pipeline is a workflow in which a model expands an observed image, video frame, or camera view beyond its original support and then uses the synthesized periphery as augmentation, pseudo-supervision, or preprocessing for a downstream system. In the cited literature, this paradigm appears in several technically distinct forms: geometry-aware wide-field augmentation for sparse-view 3D reconstruction, iterative horizontal expansion for very long scenery images, mask-preserving background synthesis for personalized advertising, off-frame light-source reconstruction for flare removal, and temporally consistent edge completion for videos (Huang et al., 31 Dec 2025, Yang et al., 2019, Shilova et al., 2023, Tsai et al., 17 Oct 2025, Fan et al., 2023). Across these settings, the common objective is not merely image completion, but the controlled generation of synthetic observations that alter the effective training or inference distribution while preserving alignment with known content.

1. Conceptual scope and distinction from adjacent paradigms

Outpainting differs from inpainting by extending content beyond the known image boundary rather than only filling interior holes. It also differs from generic novel-view synthesis: in GaMO, the central reformulation is to expand the field of view from existing camera poses instead of generating images from sampled new poses, because this preserves alignment with the original camera geometry while increasing coverage beyond known view peripheries (Huang et al., 31 Dec 2025). In the scenery setting, the task is framed as multi-step generation in which newly synthesized regions can become spatially far away from the initial input, making long-range consistency a first-order concern rather than a by-product (Yang et al., 2019). In video, the additional requirement is temporal consistency across inferred clips and sparse guide frames, so the pipeline must manage jitter and artifact accumulation across time, not only within a single frame (Fan et al., 2023).

The practical role of outpainting also varies by domain. In sparse-view reconstruction, synthesized wide-FOV images serve as new supervisory views for 3D Gaussian Splatting or related pipelines. In personalized advertising, the generated background is itself the creative artifact, conditioned on user interests and a preserved product mask. In lens-flare removal, outpainting functions as a plug-and-play preprocessing stage that reconstructs off-frame light sources before a downstream Single Image Flare Removal model is applied. These examples show that the “synthetic data” produced by outpainting may be training data, refinement targets, selected outputs, or physically motivated intermediate observations (Shilova et al., 2023, Tsai et al., 17 Oct 2025).

A common misconception is that outpainting recovers arbitrary unseen content. The papers are more constrained. GaMO explicitly notes that occluded content invisible from all inputs cannot be reconstructed, and LightsOut relies on a learned light-source parameterization and conditioning rather than unconstrained hallucination (Huang et al., 31 Dec 2025, Tsai et al., 17 Oct 2025). This suggests that outpainting pipelines are most effective when the unknown region is strongly anchored by geometry, masks, temporal cues, or task-specific priors.

2. Geometric formulations and controllable unknown-region definition

The most explicit geometric formulation appears in GaMO. The inputs are posed RGB views $I_i \in \mathbb{R}^{H \times W \times 3}$, intrinsics $K_i \in \mathbb{R}^{3 \times 3}$, and extrinsics $R_i \in SO(3), t_i \in \mathbb{R}^3$, together with coarse geometry priors rendered from a coarse 3DGS initialized by DUSt3R point clouds (Huang et al., 31 Dec 2025). Wide-FOV targets are created by scaling the intrinsics with $S_k < 1$, keeping the principal point and extrinsics fixed:

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$

With the paper default $S_k = 0.6$, the canvas size remains unchanged while the camera covers a larger scene portion.

GaMO’s camera model is standard projective geometry. For a homogeneous 3D point $X \in \mathbb{R}^4$ and camera matrix $P = K[R|t]$,

$$\tilde{u} = PX,\qquad u = [\tilde{u}_x/\tilde{u}_z,\ \tilde{u}_y/\tilde{u}_z]^\top,$$

and back-projection with depth $d$ is

$K_i \in \mathbb{R}^{3 \times 3}$0

Warping from view $K_i \in \mathbb{R}^{3 \times 3}$1 to $K_i \in \mathbb{R}^{3 \times 3}$2 uses

$K_i \in \mathbb{R}^{3 \times 3}$3

The same section of the paper also provides the essential and fundamental matrix definitions,

$K_i \in \mathbb{R}^{3 \times 3}$4

which motivate epipolar-consistent conditioning even though GaMO operationally enforces consistency through warping and latent blending rather than explicit epipolar losses.

Unknown regions are defined through rendered priors. GaMO renders wide-FOV opacity with depth-ordered alpha blending,

$K_i \in \mathbb{R}^{3 \times 3}$5

and then thresholds it into an outpaint mask

$K_i \in \mathbb{R}^{3 \times 3}$6

This links the outpaint region directly to coarse scene occupancy rather than to an arbitrary border crop (Huang et al., 31 Dec 2025).

A parallel but task-specific form of controllable geometry appears in LightsOut. There, the unknown region is an extended canvas around the flare-corrupted image, paired with a binary mask $K_i \in \mathbb{R}^{3 \times 3}$7 and a learned light-source mask $K_i \in \mathbb{R}^{3 \times 3}$8. The regression module predicts $K_i \in \mathbb{R}^{3 \times 3}$9 circular source parameters $R_i \in SO(3), t_i \in \mathbb{R}^3$0 and confidences $R_i \in SO(3), t_i \in \mathbb{R}^3$1, which are rendered into

$R_i \in SO(3), t_i \in \mathbb{R}^3$2

In this case the outpainted synthetic content is constrained not by multiview camera geometry, but by an explicit spatial parameterization of off-frame light emitters (Tsai et al., 17 Oct 2025).

3. Canonical pipeline structure

Despite architectural differences, the papers converge on a recurring staged design: definition of known versus unknown support, assembly of structural conditioning, generation of extended content, and downstream consumption of the synthetic output.

Pipeline	Synthetic target	Downstream role
GaMO	Wide-FOV image $R_i \in SO(3), t_i \in \mathbb{R}^3$3 for each input camera	Refinement of 3DGS; similarly usable for NeRF/mesh pipelines
NS-Outpainting	Predicted right-half tiles in iterative 128-pixel steps	Construction of very long scenery images
AdBooster	Product-preserving background outpaint	Personalized ad creative selection and display
LightsOut	Off-frame light-source completion on an extended canvas	Preprocessing for SIFR, then crop back to original FOV
M3DDM	Edge-completed video clips and sparse keyframes	Temporal clip connection and long-video outpainting

In GaMO, the three stages are explicit. First, a DUSt3R point cloud is used to initialize a coarse 3DGS trained for 10,000 iterations; the model then renders wide-FOV opacity and coarse color. Second, zero-shot multi-view diffusion outpainting generates wide-FOV images conditioned on multi-view signals and coarse priors. Third, 3DGS is refined jointly on original and outpainted views, with alternating supervision and LPIPS on outpainted views; point re-initialization from outpainted views is optional (Huang et al., 31 Dec 2025).

The scenery pipeline is instead iterative and autoregressive. A single step takes a $R_i \in SO(3), t_i \in \mathbb{R}^3$4 input and produces a $R_i \in SO(3), t_i \in \mathbb{R}^3$5 output whose left half equals the input and whose right half is predicted; multi-step generation repeatedly crops the rightmost $R_i \in SO(3), t_i \in \mathbb{R}^3$6 tile and uses it as the next seed, concatenating outputs to form very long images (Yang et al., 2019). This pipeline is not framed as downstream augmentation in the 3D sense, but it is an explicit synthetic data generator with boundary blending, quality filtering, metadata tracking, and storage sharding.

AdBooster adds a dataset-construction stage before model fine-tuning. The automated pipeline uses U²-Net for mask extraction, filters images with monochromatic backgrounds using a standard-deviation threshold of 20, drops images with mask area ratio greater than 0.6, and uses BLIP captions to produce text conditions for fine-tuning Stable Diffusion outpainting. At inference, personalization is injected through prompts $R_i \in SO(3), t_i \in \mathbb{R}^3$7 or $R_i \in SO(3), t_i \in \mathbb{R}^3$8, and $R_i \in SO(3), t_i \in \mathbb{R}^3$9 candidates are generated and ranked by CLIP similarity to the user representation (Shilova et al., 2023).

M3DDM generalizes the pattern to video by replacing single-image generation with a hierarchical coarse-to-fine schedule. It first generates sparse keyframes and clips, then interpolates additional keyframes, and finally densifies to interval 1. The paper reports working interval schedules such as $S_k < 1$0 and $S_k < 1$1, with guide frames and global frames supplying temporal anchors throughout the process (Fan et al., 2023).

4. Conditioning mechanisms, denoising objectives, and architectural variants

Most of the pipelines use diffusion-style denoising objectives. GaMO adopts the standard forward process

$S_k < 1$2

with the simplified training objective

$S_k < 1$3

Its reverse process uses DDIM sampling with $S_k < 1$4, but the distinctive mechanism is geometry-aware latent blending at timesteps $S_k < 1$5:

$S_k < 1$6

Iterative Mask Scheduling progressively changes the mask support over denoising, hard masks outperform soft masks by $S_k < 1$7 dB PSNR, and noise resampling repeated $S_k < 1$8 times yields a further $S_k < 1$9 dB PSNR gain (Huang et al., 31 Dec 2025).

The conditioning stack in GaMO is also unusually structured. It combines Plücker ray embeddings, Canonical Coordinate Maps, warped RGB and CCM signals, and center-pasted downscaled originals. The denoiser is conditioned on both input and target fused features,

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$0

which anchors the generated periphery to existing camera geometry rather than relying solely on text or image priors (Huang et al., 31 Dec 2025).

AdBooster uses a Stable Diffusion outpainting variant conditioned on image, text, and mask tuples. Only the diffusion U-Net is updated during fine-tuning; the text encoder and VAE remain frozen. The reported loss is the standard latent diffusion noise-prediction objective,

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$1

with conditioning defined by the caption, product mask, and masked latent or image channels (Shilova et al., 2023). Personalization is introduced at generation time through prompt construction rather than through a separate ranking loss during training.

LightsOut also uses a diffusion objective,

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$2

augmented by a light-map alignment loss

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$3

Its diffusion backbone is Stable Diffusion v2 inpainting with LoRA injected into the denoising U-Net, while the multitask regression module is trained with Hungarian matching, Smooth L1 position loss, BCE confidence loss, and an uncertainty-weighted multitask objective (Tsai et al., 17 Oct 2025).

By contrast, the 2019 scenery system is GAN-based rather than diffusion-based. Its generator is an encoder-decoder with Recurrent Content Transfer, Skip Horizontal Connection, and Global Residual Blocks, and it is trained with a masked reconstruction loss plus global and local WGAN-GP discriminators. The reported hyperparameters are $f_x' = S_k f_x,\qquad f_y' = S_k f_y.$4, $f_x' = S_k f_x,\qquad f_y' = S_k f_y.$5, $f_x' = S_k f_x,\qquad f_y' = S_k f_y.$6, and $f_x' = S_k f_x,\qquad f_y' = S_k f_y.$7 (Yang et al., 2019). The architectural emphasis is on horizontal consistency over very long distances, rather than latent denoising.

M3DDM extends latent diffusion into spacetime. The 3D U-Net receives noisy raw latents, masked latents, and a binary mask as $f_x' = S_k f_x,\qquad f_y' = S_k f_y.$8 channels, uses pseudo-3D operations and cross-attention from a lightweight encoder over global frames, and trains with

$f_x' = S_k f_x,\qquad f_y' = S_k f_y.$9

Its two-condition classifier-free guidance is

$S_k = 0.6$0

with defaults $S_k = 0.6$1 and $S_k = 0.6$2 (Fan et al., 2023).

5. Downstream integration and reported empirical behavior

The downstream value of outpainted synthetic data is clearest in GaMO. The refined 3DGS is trained on both original and outpainted images with alternating supervision. For original views,

$S_k = 0.6$3

and for outpainted views,

$S_k = 0.6$4

The paper reports state-of-the-art reconstruction quality across 3, 6, and 9 input views on Replica and ScanNet++, with a $S_k = 0.6$5 speedup over state-of-the-art diffusion-based methods and processing time under 10 minutes. For 6-view Replica, GaMO reports PSNR 25.84 dB, SSIM 0.877, LPIPS 0.109, and FID 72.95; for 6-view ScanNet++, it reports PSNR 23.41 dB, SSIM 0.835, LPIPS 0.181, and FID 108.06 (Huang et al., 31 Dec 2025).

The scenery pipeline reports 1-step validation FID 13.713 for RCT+SHC, compared with 15.186 for FC+SHC, 14.825 for GLC, 19.040 for CA, and 19.734 for Pix2Pix. The paper also reports that 3 GRBs improve FID to 13.713 relative to 15.171 with 0 GRB, and it qualitatively attributes improved multi-step robustness to the RCT+SHC design (Yang et al., 2019).

AdBooster evaluates synthetic creatives by a CLIP-based reward proxy,

$S_k = 0.6$6

The fine-tuned Stable Diffusion outpainting model improves FID from 30.46 to 25.73 on the augmented fashion dataset. Relative CLIP uplift over baselines is reported across categories, including +12.41% to +21.5% for direct $S_k = 0.6$7 prompting and +15.16% to +27.87% for GPT-3.5 prompt generation, depending on category (Shilova et al., 2023).

LightsOut evaluates the usefulness of outpainted synthetic context through downstream flare removal rather than isolated image realism. With Flare7K++ SIFR on real no-light-source images, PSNR improves from 26.29 dB to 28.41 dB, SSIM from 0.8337 to 0.8956, and LPIPS from 0.0442 to 0.0397. The paper also reports that regression-based light masks outperform U-Net map prediction in mIoU and that RGB blending is clearly better than latent blending (Tsai et al., 17 Oct 2025).

For video outpainting, M3DDM reports on DAVIS PSNR 20.26, SSIM 0.7082, LPIPS 0.2026, and FVD 300.0, compared with SDM at PSNR 20.02, SSIM 0.7078, LPIPS 0.2165, and FVD 334.6. On YouTube-VOS it reports PSNR 20.20, SSIM 0.7312, LPIPS 0.1854, and FVD 66.62 (Fan et al., 2023).

6. Limitations, failure modes, and research directions

The principal limitations are domain-specific but structurally related. GaMO cannot reconstruct content that is occluded and invisible from all inputs; performance degrades when views are clustered or misaligned, and the paper notes that indoor scenes are prioritized, with adjustments needed for outdoor or large unbounded scenes (Huang et al., 31 Dec 2025). The scenery pipeline reports semantic drift, repetitive or implausible scenes, and texture blur over many iterative steps, particularly if SHC or GRB capacity is reduced (Yang et al., 2019). AdBooster identifies duplicated content, lighting inconsistencies, person generation artifacts, and subject/background ratio issues, while also noting that training hyperparameters, throughput, and safety-governance specifics are not reported (Shilova et al., 2023). LightsOut notes computational overhead from the added outpainting stage, and M3DDM reports rough faces, difficult text rendering, and sensitivity to initial Gaussian noise (Tsai et al., 17 Oct 2025, Fan et al., 2023).

Several corrective strategies recur across the papers. GaMO uses hard masks, center-pasted downscaled originals, geometry-aware warping, and repeated noise resampling to reduce drift and boundary artifacts. The scenery work uses cosine boundary masks, dual discriminators, and iterative cropping to keep semantics moving outward from a local anchor. LightsOut relies on explicit light-map conditioning and RGB-space compositing rather than latent blending. M3DDM uses multiple guide frames, global-frame prompts, and a three-level hybrid coarse-to-fine schedule to reduce temporal gaps and artifact accumulation (Huang et al., 31 Dec 2025, Yang et al., 2019, Tsai et al., 17 Oct 2025, Fan et al., 2023).

Future directions are stated explicitly in several sources. GaMO identifies adaptive outpaint scale selection, hybrid strategies combining limited novel-view generation with outpainting, and lightweight finetuning for challenging domains. LightsOut points to stronger conditioning and control through its existing conditioning structure, although the paper itself centers on LoRA and regression-driven light masks. AdBooster lists SDXL, ControlNet, and Composer as future work. M3DDM’s analysis implies that better handling of long-range dependencies, faces, and text would further strengthen video pipelines (Huang et al., 31 Dec 2025, Shilova et al., 2023, Fan et al., 2023).

Taken together, these works define outpainting-based synthetic data generation as a family of controlled extrapolation pipelines in which the crucial design problem is not merely how to generate missing pixels, but how to encode the structural invariants of the downstream task—camera geometry, product identity, light-source placement, or temporal continuity—so that the synthesized periphery is operationally useful rather than visually plausible alone.