---
title: Outpainting Synthetic Data Generation Pipeline
url: https://www.emergentmind.com/topics/outpainting-based-synthetic-data-generation-pipeline
type: topic
---

# Outpainting Synthetic Data Generation Pipeline

Searching arXiv for the cited papers to ground the article in the latest records.
An outpainting-based synthetic data generation pipeline is a workflow in which a model expands an observed image, video frame, or camera view beyond its original support and then uses the synthesized periphery as augmentation, pseudo-supervision, or preprocessing for a downstream system. In the cited literature, this paradigm appears in several technically distinct forms: geometry-aware wide-field augmentation for sparse-view 3D reconstruction, iterative horizontal expansion for very long scenery images, mask-preserving background synthesis for personalized advertising, off-frame light-source reconstruction for flare removal, and temporally consistent edge completion for videos [2512.25073] [1912.12688] [2309.11507] [2510.15868] [2309.02119]. Across these settings, the common objective is not merely image completion, but the controlled generation of synthetic observations that alter the effective training or inference distribution while preserving alignment with known content.

## 1. Conceptual scope and distinction from adjacent paradigms

Outpainting differs from inpainting by extending content beyond the known image boundary rather than only filling interior holes. It also differs from generic novel-view synthesis: in GaMO, the central reformulation is to expand the field of view from existing camera poses instead of generating images from sampled new poses, because this preserves alignment with the original camera geometry while increasing coverage beyond known view peripheries [2512.25073]. In the scenery setting, the task is framed as multi-step generation in which newly synthesized regions can become spatially far away from the initial input, making long-range consistency a first-order concern rather than a by-product [1912.12688]. In video, the additional requirement is temporal consistency across inferred clips and sparse guide frames, so the pipeline must manage jitter and artifact accumulation across time, not only within a single frame [2309.02119].

The practical role of outpainting also varies by domain. In sparse-view reconstruction, synthesized wide-FOV images serve as new supervisory views for 3D Gaussian Splatting or related pipelines. In personalized advertising, the generated background is itself the creative artifact, conditioned on user interests and a preserved product mask. In lens-flare removal, outpainting functions as a plug-and-play preprocessing stage that reconstructs off-frame light sources before a downstream Single Image Flare Removal model is applied. These examples show that the “synthetic data” produced by outpainting may be training data, refinement targets, selected outputs, or physically motivated intermediate observations [2309.11507] [2510.15868].

A common misconception is that outpainting recovers arbitrary unseen content. The papers are more constrained. GaMO explicitly notes that occluded content invisible from all inputs cannot be reconstructed, and LightsOut relies on a learned light-source parameterization and conditioning rather than unconstrained hallucination [2512.25073] [2510.15868]. This suggests that outpainting pipelines are most effective when the unknown region is strongly anchored by geometry, masks, temporal cues, or task-specific priors.

## 2. Geometric formulations and controllable unknown-region definition

The most explicit geometric formulation appears in GaMO. The inputs are posed RGB views $I_i \in \mathbb{R}^{H \times W \times 3}$, intrinsics $K_i \in \mathbb{R}^{3 \times 3}$, and extrinsics $R_i \in SO(3), t_i \in \mathbb{R}^3$, together with coarse geometry priors rendered from a coarse 3DGS initialized by DUSt3R point clouds [2512.25073]. Wide-FOV targets are created by scaling the intrinsics with $S_k < 1$, keeping the principal point and extrinsics fixed:
$$
f_x' = S_k f_x,\qquad f_y' = S_k f_y.
$$
With the paper default $S_k = 0.6$, the canvas size remains unchanged while the camera covers a larger scene portion.

GaMO’s camera model is standard projective geometry. For a homogeneous 3D point $X \in \mathbb{R}^4$ and camera matrix $P = K[R|t]$,
$$
\tilde{u} = PX,\qquad u = [\tilde{u}_x/\tilde{u}_z,\ \tilde{u}_y/\tilde{u}_z]^\top,
$$
and back-projection with depth $d$ is
$$
X = R^\top K^{-1}(\tilde{u}d) + C,\qquad C = -R^\top t.
$$
Warping from view $i$ to $j$ uses
$$
\omega_{i \to j}(u) = \pi_j(X) = K_j [R_j|t_j]\big(R_i^\top K_i^{-1}(\tilde{u}d_i(u)) + C_i\big).
$$
The same section of the paper also provides the essential and fundamental matrix definitions,
$$
E_{ij} = [t_{ij}]_\times R_{ij},\qquad F_{ij} = K_j^{-\top} E_{ij} K_i^{-1},
$$
which motivate epipolar-consistent conditioning even though GaMO operationally enforces consistency through warping and latent blending rather than explicit epipolar losses.

Unknown regions are defined through rendered priors. GaMO renders wide-FOV opacity with depth-ordered alpha blending,
$$
O(u)=\sum_{i\in N(u)} \sigma_i(u)\prod_{j=1}^{i-1}(1-\sigma_j(u)), \qquad \sigma_i(u)=\alpha_i G_i'(u),
$$
and then thresholds it into an outpaint mask
$$
M(u)=\mathbf{1}(O(u)<\eta_{\text{mask}}),\qquad \eta_{\text{mask}}=0.6.
$$
This links the outpaint region directly to coarse scene occupancy rather than to an arbitrary border crop [2512.25073].

A parallel but task-specific form of controllable geometry appears in LightsOut. There, the unknown region is an extended canvas around the flare-corrupted image, paired with a binary mask $M$ and a learned light-source mask $M_L$. The regression module predicts $N$ circular source parameters $(x_i,y_i,r_i)$ and confidences $c_i$, which are rendered into
$$
M_L(x,y)=\sum_{i=1}^{N} \tilde{c}_i \cdot \sigma\!\left(r_i-\sqrt{(x-x_i)^2+(y-y_i)^2}\right).
$$
In this case the outpainted synthetic content is constrained not by multiview camera geometry, but by an explicit spatial parameterization of off-frame light emitters [2510.15868].

## 3. Canonical pipeline structure

Despite architectural differences, the papers converge on a recurring staged design: definition of known versus unknown support, assembly of structural conditioning, generation of extended content, and downstream consumption of the synthetic output.

| Pipeline | Synthetic target | Downstream role |
|---|---|---|
| GaMO | Wide-FOV image $S_i^{out}$ for each input camera | Refinement of 3DGS; similarly usable for NeRF/mesh pipelines |
| NS-Outpainting | Predicted right-half tiles in iterative 128-pixel steps | Construction of very long scenery images |
| AdBooster | Product-preserving background outpaint | Personalized ad creative selection and display |
| LightsOut | Off-frame light-source completion on an extended canvas | Preprocessing for SIFR, then crop back to original FOV |
| M3DDM | Edge-completed video clips and sparse keyframes | Temporal clip connection and long-video outpainting |

In GaMO, the three stages are explicit. First, a DUSt3R point cloud is used to initialize a coarse 3DGS trained for 10,000 iterations; the model then renders wide-FOV opacity and coarse color. Second, zero-shot multi-view diffusion outpainting generates wide-FOV images conditioned on multi-view signals and coarse priors. Third, 3DGS is refined jointly on original and outpainted views, with alternating supervision and LPIPS on outpainted views; point re-initialization from outpainted views is optional [2512.25073].

The scenery pipeline is instead iterative and autoregressive. A single step takes a $128\times128$ input and produces a $128\times256$ output whose left half equals the input and whose right half is predicted; multi-step generation repeatedly crops the rightmost $128\times128$ tile and uses it as the next seed, concatenating outputs to form very long images [1912.12688]. This pipeline is not framed as downstream augmentation in the 3D sense, but it is an explicit synthetic data generator with boundary blending, quality filtering, metadata tracking, and storage sharding.

AdBooster adds a dataset-construction stage before model fine-tuning. The automated pipeline uses U²-Net for mask extraction, filters images with monochromatic backgrounds using a standard-deviation threshold of 20, drops images with mask area ratio greater than 0.6, and uses BLIP captions to produce text conditions for fine-tuning Stable Diffusion outpainting. At inference, personalization is injected through prompts $z_{QC}(u)$ or $z_P(u)=T_\phi(i,u)$, and $K=4$ candidates are generated and ranked by CLIP similarity to the user representation [2309.11507].

M3DDM generalizes the pattern to video by replacing single-image generation with a hierarchical coarse-to-fine schedule. It first generates sparse keyframes and clips, then interpolates additional keyframes, and finally densifies to interval 1. The paper reports working interval schedules such as $[30,15,1]$ and $[15,5,1]$, with guide frames and global frames supplying temporal anchors throughout the process [2309.02119].

## 4. Conditioning mechanisms, denoising objectives, and architectural variants

Most of the pipelines use diffusion-style denoising objectives. GaMO adopts the standard forward process
$$
x_t=\sqrt{\bar{\alpha}_t}x_0+\sqrt{1-\bar{\alpha}_t}\epsilon,\qquad \epsilon\sim\mathcal{N}(0,I),
$$
with the simplified training objective
$$
L_{\text{simple}}=\mathbb{E}_{t,x_0,\epsilon}\big[\|\epsilon-\epsilon_\theta(x_t,t,c)\|_2^2\big].
$$
Its reverse process uses DDIM sampling with $T=50$, but the distinctive mechanism is geometry-aware latent blending at timesteps $t_k\in\{0.7T,0.5T,0.3T\}$:
$$
z^{\text{blend}}_{t_k}=(1-M^{(k)}_{\text{latent}})\odot z^{\text{coarse}}_{t_k}+M^{(k)}_{\text{latent}}\odot z_{t_k}.
$$
Iterative Mask Scheduling progressively changes the mask support over denoising, hard masks outperform soft masks by $+0.64$ dB PSNR, and noise resampling repeated $R=3$ times yields a further $+0.24$ dB PSNR gain [2512.25073].

The conditioning stack in GaMO is also unusually structured. It combines Plücker ray embeddings, Canonical Coordinate Maps, warped RGB and CCM signals, and center-pasted downscaled originals. The denoiser is conditioned on both input and target fused features,
$$
p_\theta(z_t \mid z_r, P_r, C_r, I_r, P_t^*, C_{r\to t}^{aug}, I_{r\to t}^{aug}),
$$
which anchors the generated periphery to existing camera geometry rather than relying solely on text or image priors [2512.25073].

AdBooster uses a Stable Diffusion outpainting variant conditioned on image, text, and mask tuples. Only the diffusion U-Net is updated during fine-tuning; the text encoder and VAE remain frozen. The reported loss is the standard latent diffusion noise-prediction objective,
$$
L_\epsilon=\mathbb{E}_{z_0,\epsilon,t}\big[\|\epsilon-\epsilon_\theta(z_t,t,\text{cond})\|^2\big],
$$
with conditioning defined by the caption, product mask, and masked latent or image channels [2309.11507]. Personalization is introduced at generation time through prompt construction rather than through a separate ranking loss during training.

LightsOut also uses a diffusion objective,
$$
L=\mathbb{E}_{x,t,\epsilon,m}\big[\|\epsilon_\theta(x_t,t,p,M,I_M)-\epsilon\|_2^2\big],
$$
augmented by a light-map alignment loss
$$
L_{\text{light}}=\|\tilde{M}_L-M_L\|_2^2.
$$
Its diffusion backbone is Stable Diffusion v2 inpainting with LoRA injected into the denoising U-Net, while the multitask regression module is trained with Hungarian matching, Smooth L1 position loss, BCE confidence loss, and an uncertainty-weighted multitask objective [2510.15868].

By contrast, the 2019 scenery system is GAN-based rather than diffusion-based. Its generator is an encoder-decoder with Recurrent Content Transfer, Skip Horizontal Connection, and Global Residual Blocks, and it is trained with a masked reconstruction loss plus global and local WGAN-GP discriminators. The reported hyperparameters are $\lambda_{gp}=10$, $\beta=0.9$, $\lambda_{adv}=0.002$, and $\lambda_{rec}=0.998$ [1912.12688]. The architectural emphasis is on horizontal consistency over very long distances, rather than latent denoising.

M3DDM extends latent diffusion into spacetime. The 3D U-Net receives noisy raw latents, masked latents, and a binary mask as $C_{in}=9$ channels, uses pseudo-3D operations and cross-attention from a lightweight encoder over global frames, and trains with
$$
L_{DM}=\mathbb{E}_{x,\epsilon,t}\big[\|\epsilon-\epsilon_\theta(x_t,c,t)\|_2^2\big].
$$
Its two-condition classifier-free guidance is
$$
\hat{\epsilon}(z_t,c_1,c_2)=\epsilon(z_t,\emptyset,\emptyset)+s_1\big(\epsilon(z_t,c_1,\emptyset)-\epsilon(z_t,\emptyset,\emptyset)\big)+s_2\big(\epsilon(z_t,c_1,c_2)-\epsilon(z_t,c_1,\emptyset)\big),
$$
with defaults $s_1=2$ and $s_2=4$ [2309.02119].

## 5. Downstream integration and reported empirical behavior

The downstream value of outpainted synthetic data is clearest in GaMO. The refined 3DGS is trained on both original and outpainted images with alternating supervision. For original views,
$$
L_{\text{input}}=(1-\lambda_s)L1+\lambda_s L_{D\text{-}SSIM},\qquad \lambda_s=0.2,
$$
and for outpainted views,
$$
L_{\text{outpainted}}=L_{\text{recon}}+\lambda_{perc}\,LPIPS,\qquad \lambda_{perc}=0.1.
$$
The paper reports state-of-the-art reconstruction quality across 3, 6, and 9 input views on Replica and ScanNet++, with a $25\times$ speedup over state-of-the-art diffusion-based methods and processing time under 10 minutes. For 6-view Replica, GaMO reports PSNR 25.84 dB, SSIM 0.877, LPIPS 0.109, and FID 72.95; for 6-view ScanNet++, it reports PSNR 23.41 dB, SSIM 0.835, LPIPS 0.181, and FID 108.06 [2512.25073].

The scenery pipeline reports 1-step validation FID 13.713 for RCT+SHC, compared with 15.186 for FC+SHC, 14.825 for GLC, 19.040 for CA, and 19.734 for Pix2Pix. The paper also reports that 3 GRBs improve FID to 13.713 relative to 15.171 with 0 GRB, and it qualitatively attributes improved multi-step robustness to the RCT+SHC design [1912.12688].

AdBooster evaluates synthetic creatives by a CLIP-based reward proxy,
$$
R(u,i)=\langle CLIP(z(u)), CLIP(c(i))\rangle.
$$
The fine-tuned Stable Diffusion outpainting model improves FID from 30.46 to 25.73 on the augmented fashion dataset. Relative CLIP uplift over baselines is reported across categories, including +12.41% to +21.5% for direct $z_{QC}$ prompting and +15.16% to +27.87% for GPT-3.5 prompt generation, depending on category [2309.11507].

LightsOut evaluates the usefulness of outpainted synthetic context through downstream flare removal rather than isolated image realism. With Flare7K++ SIFR on real no-light-source images, PSNR improves from 26.29 dB to 28.41 dB, SSIM from 0.8337 to 0.8956, and LPIPS from 0.0442 to 0.0397. The paper also reports that regression-based light masks outperform U-Net map prediction in mIoU and that RGB blending is clearly better than latent blending [2510.15868].

For video outpainting, M3DDM reports on DAVIS PSNR 20.26, SSIM 0.7082, LPIPS 0.2026, and FVD 300.0, compared with SDM at PSNR 20.02, SSIM 0.7078, LPIPS 0.2165, and FVD 334.6. On YouTube-VOS it reports PSNR 20.20, SSIM 0.7312, LPIPS 0.1854, and FVD 66.62 [2309.02119].

## 6. Limitations, failure modes, and research directions

The principal limitations are domain-specific but structurally related. GaMO cannot reconstruct content that is occluded and invisible from all inputs; performance degrades when views are clustered or misaligned, and the paper notes that indoor scenes are prioritized, with adjustments needed for outdoor or large unbounded scenes [2512.25073]. The scenery pipeline reports semantic drift, repetitive or implausible scenes, and texture blur over many iterative steps, particularly if SHC or GRB capacity is reduced [1912.12688]. AdBooster identifies duplicated content, lighting inconsistencies, person generation artifacts, and subject/background ratio issues, while also noting that training hyperparameters, throughput, and safety-governance specifics are not reported [2309.11507]. LightsOut notes computational overhead from the added outpainting stage, and M3DDM reports rough faces, difficult text rendering, and sensitivity to initial Gaussian noise [2510.15868] [2309.02119].

Several corrective strategies recur across the papers. GaMO uses hard masks, center-pasted downscaled originals, geometry-aware warping, and repeated noise resampling to reduce drift and boundary artifacts. The scenery work uses cosine boundary masks, dual discriminators, and iterative cropping to keep semantics moving outward from a local anchor. LightsOut relies on explicit light-map conditioning and RGB-space compositing rather than latent blending. M3DDM uses multiple guide frames, global-frame prompts, and a three-level hybrid coarse-to-fine schedule to reduce temporal gaps and artifact accumulation [2512.25073] [1912.12688] [2510.15868] [2309.02119].

Future directions are stated explicitly in several sources. GaMO identifies adaptive outpaint scale selection, hybrid strategies combining limited novel-view generation with outpainting, and lightweight finetuning for challenging domains. LightsOut points to stronger conditioning and control through its existing conditioning structure, although the paper itself centers on LoRA and regression-driven light masks. AdBooster lists SDXL, ControlNet, and Composer as future work. M3DDM’s analysis implies that better handling of long-range dependencies, faces, and text would further strengthen video pipelines [2512.25073] [2309.11507] [2309.02119].

Taken together, these works define outpainting-based synthetic data generation as a family of controlled extrapolation pipelines in which the crucial design problem is not merely how to generate missing pixels, but how to encode the structural invariants of the downstream task—camera geometry, product identity, light-source placement, or temporal continuity—so that the synthesized periphery is operationally useful rather than visually plausible alone.

Source: https://www.emergentmind.com/topics/outpainting-based-synthetic-data-generation-pipeline