Pix2SG: Pixel-Conditioned Structured Generation

• Pix2SG is an umbrella term for multiple pixel-conditioned generation systems that output structured representations like scene graphs, 3D maps, and synthetic images.
• It encompasses methods ranging from location-free scene graph generation using autoregressive models to dual-guided 3D inference and image-to-image translation for binary SPC and SAR synthesis.
• A recurring design pattern is the use of intermediate structured priors and learned transformations to bridge gaps between raw pixel data and operational, semantic outputs.

Pix2SG is an ambiguous designation used in recent arXiv literature for several distinct pixel-conditioned generation systems rather than a single standardized method. In different contexts, it denotes a location-free scene graph generator, a semantic-geometric mapping framework for human-robot understanding, an image-to-3D scene graph component inside a dual spatial learning system, a two-stage Single Photon Camera reconstruction pipeline based on Pix2PixHD and 3D Gaussian Splatting, and, in one overview usage, a pix2pix-based method for glacier SAR synthesis from masks (Özsoy et al., 2023, Longo et al., 27 Jun 2025, Zhao et al., 2024, Sharma et al., 7 Jun 2025, Dietrich-Sussner et al., 2021).

1. Nomenclature and scope

The shared intuition behind the various Pix2SG usages is a transformation from image-derived inputs to structured outputs. What changes across papers is the target structure: relation quintuples for scene graphs, hierarchical semantic-geometric graphs for robot mapping, discrete 3D scene graphs for spatial reasoning, 3D Gaussian Splatting assets for novel-view synthesis, or synthetic SAR images conditioned on masks.

Usage of “Pix2SG”	Input $\rightarrow$ output	Core formulation
LF-SGG baseline (Özsoy et al., 2023)	Image $\rightarrow$ location-free scene graph	Autoregressive quintuple decoding
Pix2G / pixels-to-scene-graph (Longo et al., 27 Jun 2025)	RGB-D + LiDAR map $\rightarrow$ BIM-like 2D map, segmented 3D cloud, multi-layer graph	CPU-only semantic and structural fusion
SD$^3$ image$\rightarrow$3DSG path (Zhao et al., 2024)	Image $\rightarrow$ 3D scene graph	Discrete graph diffusion with dual guidance
SPC pipeline (Sharma et al., 7 Jun 2025)	Binary SPC frames $\rightarrow$ RGB views $\rightarrow$ 3DGS	Pix2PixHD followed by 3DGS
Glacier SAR synthesis (Dietrich-Sussner et al., 2021)	Binary mask $\rightarrow$ SAR image	pix2pix conditional GAN

A common misconception is that Pix2SG invariably means “pixels to scene graph.” That interpretation is exact for some works, especially the robotics and scene-graph papers, but not for all usages. In the SPC paper, Pix2SG denotes Pix2PixHD followed by 3D Gaussian Splatting; in the glacier SAR paper, the designation is an editorial shorthand rather than the paper’s own method name (Sharma et al., 7 Jun 2025, Dietrich-Sussner et al., 2021).

2. Location-free scene graph generation

In "Location-Free Scene Graph Generation" (Özsoy et al., 2023), Pix2SG is the first method proposed for location-free scene graph generation (LF-SGG). The task removes bounding boxes and segmentation masks from both training and inference. Given an image $I$, the goal is to predict a graph $\rightarrow$0, where each node carries a class label $\rightarrow$1 and each directed edge carries a relation label $\rightarrow$2. The paper frames this as autoregressive sequence modeling,

$\rightarrow$3

with the scene graph encoded as a sequence of relation quintuples $\rightarrow$4 (Özsoy et al., 2023).

The architecture combines an EfficientNet image encoder with a Transformer decoder. EfficientNet-B7 is used for Visual Genome with input resized to $\rightarrow$5, and EfficientNet-B5 is used for 4D-OR with input resized to $\rightarrow$6. The decoder has hidden size 256, 8 attention heads, 6 encoder layers, and 2 decoder layers. Training uses teacher forcing with maximum likelihood,

$\rightarrow$7

optimized with AdamW at learning rate $\rightarrow$8, weight decay $\rightarrow$9, batch size 16, and 200 epochs with early stopping. Decoding uses nucleus sampling with $\rightarrow$0, because greedy decoding led to token repetitions and poorer graph coverage (Özsoy et al., 2023).

A central technical issue is evaluation without locations. Matching predicted and ground-truth graphs becomes an NP-hard problem because instance identities must be aligned without IoU or masks. The paper addresses this with Heuristic Tree Search (HTS), a branching algorithm that starts from the highest-degree ground-truth node, ranks candidate matches of the same class by one-hop neighborhood overlap, and searches with branching factor $\rightarrow$1. The paper reports that $\rightarrow$2 is sufficient to reach near-optimal matching with sub-second runtime in C++ (Özsoy et al., 2023).

Empirically, Pix2SG achieves $\rightarrow$3, $\rightarrow$4, and $\rightarrow$5 on Visual Genome. On 4D-OR it reports Precision $\rightarrow$6, Recall $\rightarrow$7, and $\rightarrow$8, surpassing the cited prior location-based state of the art, which had Precision $\rightarrow$9, Recall $^3$0, and $^3$1. The paper also quantifies the annotation-cost motivation: with a median 42 seconds per bounding box, it estimates roughly 15,945 hours for 1.3M boxes in Visual Genome and 2,205 hours for 189K boxes in 4D-OR (Özsoy et al., 2023).

3. Semantic-geometric mapping for human-robot understanding

In "Pixels-to-Graph: Real-time Integration of Building Information Models and Scene Graphs for Semantic-Geometric Human-Robot Understanding" (Longo et al., 27 Jun 2025), the query term Pix2SG corresponds to Pix2G, a lightweight CPU-only framework for constructing semantic-geometric scene graphs from image pixels and LiDAR maps during autonomous exploration. The goal is to bridge human-readable 2D Building Information Model (BIM)-like representations and robot-centric 3D maps in unknown indoor environments. Inputs are a Velodyne VLP-16 LiDAR at 10 Hz, an Intel RealSense D455 RGB-D camera at 15 Hz, and robot pose from onboard LiDAR localization and mapping. Outputs are a de-noised 2D top-down environment map, a structure-segmented 3D point cloud, and a multi-layer graph spanning object, scene, room, and building levels (Longo et al., 27 Jun 2025).

The system has two parallel branches followed by graph rendering. The semantic branch uses MMDetection for instance masks and class IDs, projects masks onto LiDAR points, clusters associated points with DBSCAN to estimate 3D object centers, and uses AlexNet trained on Places365 for scene classification such as office or garage. Scene nodes are created at robot locations, and objects are linked to scenes based on temporal co-occurrence. The structural branch voxelizes the point cloud, projects it to a bird’s-eye-view image with adaptive height-density scoring,

$^3$2

updates a low-pass threshold according to

$^3$3

and generates a $^3$4 BEV image with 12 px padding. A GAN then performs denoising and inpainting with generator loss

$^3$5

Finally, Mask R-CNN with a ResNet101 backbone segments rooms and corridors on the BEV image, and pixel-to-voxel projection lifts the masks back into 3D (Longo et al., 27 Jun 2025).

For clarity, the method can be described as a graph $^3$6 whose node types are object nodes $^3$7, scene nodes $^3$8, room nodes $^3$9, and building nodes $\rightarrow$0. Edge types include object-in-scene, scene-in-room, room-in-building, and adjacency or containment relations among rooms and buildings. The paper does not specify adjacency matrices or node feature vectors; edges and attributes are managed procedurally within ROS/RViz (Longo et al., 27 Jun 2025).

The framework was evaluated on the NASA JPL NeBula-Spot legged robot in a cluttered garage of approximately $\rightarrow$1 and an urban office-like environment of approximately $\rightarrow$2. In the garage, the method produced clear structural segmentation of three large areas with consistent 3D colorization and integrated multi-layer graph rendering in RViz. In the office, it identified 21 distinct rooms despite noise and partial acquisitions. Structural ablations report mIoU $\rightarrow$3, Precision $\rightarrow$4, and Recall $\rightarrow$5 for JPL-only data; mIoU $\rightarrow$6, Precision $\rightarrow$7, and Recall $\rightarrow$8 when adding CubiCasa5K; and mIoU $\rightarrow$9, Precision $\rightarrow$0, and Recall $\rightarrow$1 when adding noisy augmentation and GAN preprocessing. CPU usage peaks at approximately $\rightarrow$2 on both the base station and the onboard system, while BEV generation and 3D segmentation scale linearly with map size; the paper proposes a sliding window as future work to bound latency (Longo et al., 27 Jun 2025).

4. Pix2SG as image-to-3D scene graph inference in dual spatial learning

In "Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image" (Zhao et al., 2024), Pix2SG is the image$\rightarrow$33D sub-task inside a dual SI2T/ST2I framework. The central representation is a shared 3D scene graph (3DSG) $\rightarrow$4 with object nodes $\rightarrow$5, attribute nodes $\rightarrow$6, and relationship nodes $\rightarrow$7. Nodes are parameterized by discrete tag embeddings $\rightarrow$8, and a Discrete Graph Auto-Encoder (DGAE) quantizes continuous graph embeddings into discrete latent codes through

$\rightarrow$9

The method treats Pix2SG not as metric 3D reconstruction with explicit boxes or camera inversion, but as the enrichment of an initial 2D visual scene graph into a 3D-aware graph using learned priors from 3DSG datasets (Zhao et al., 2024).

The system contains three discrete diffusion components: an $\rightarrow$0 graph diffusion shared by image and text inputs, a $\rightarrow$1Image diffusion for ST2I, and a $\rightarrow$2Text diffusion for SI2T. Images are encoded by a VQ-VAE/VQ-GAN that maps $\rightarrow$3 images to $\rightarrow$4 code tokens with codebook size $\rightarrow$5; CLIP provides global conditioning features; GPT-2 decodes text; and a UNet is used as the denoiser for all three diffusions. For Pix2SG specifically, an off-the-shelf VSG parser such as Neural Motifs produces an initial 2D scene graph from the image, the DGAE encodes and quantizes it, and graph diffusion denoises it into a 3DSG conditioned on global image features $\rightarrow$6 and intermediate diffusion states from the easier $\rightarrow$7image and $\rightarrow$8text paths (Zhao et al., 2024).

The discrete diffusion formulation uses the standard quantization operator

$\rightarrow$9

with forward and reverse discrete Markov transitions, KL-based training, and alignment losses

$\rightarrow$0

$\rightarrow$1

These terms couple the hard image$\rightarrow$23D and text$\rightarrow$33D inferences to the easier $\rightarrow$4 paths. The full training schedule has four stages: DGAE pretraining, spatial alignment of visual and text decoders, 2DSG$\rightarrow$53DSG diffusion training, and overall dual tuning (Zhao et al., 2024).

Evaluation on the VSD benchmark reports, for ST2I on VSDv1, FID $\rightarrow$6, IS $\rightarrow$7, and CLIP $\rightarrow$8, versus the cited Frido baseline at FID $\rightarrow$9, IS $\rightarrow$0, and CLIP $\rightarrow$1. On VSDv2, ST2I reaches FID $\rightarrow$2, IS $\rightarrow$3, and CLIP $\rightarrow$4. For SI2T, the same framework yields BLEU4 $\rightarrow$5 and SPICE $\rightarrow$6 on VSDv1, and BLEU4 $\rightarrow$7 and SPICE $\rightarrow$8 on VSDv2. Human evaluation reports best Spatial Accuracy of $\rightarrow$9 for ST2I and $I$0 for SI2T, and Diversity $I$1 for SI2T. The paper also measures 3DSG generation quality by Triplet Recall, with Initial VSG at $I$2, 3D$I$3 without dual intermediate sharing at $I$4, and Full 3D$I$5 at $I$6, indicating the importance of the dual guidance used by the Pix2SG path (Zhao et al., 2024).

5. Pix2SG for binary Single Photon Camera novel-view synthesis

In "SPC to 3D: Novel View Synthesis from Binary SPC via I2I translation" (Sharma et al., 7 Jun 2025), Pix2SG denotes a specific two-stage pipeline: Pix2PixHD translates binary Single Photon Camera frames into plausible RGB images, and 3D Gaussian Splatting reconstructs and renders the scene. The motivation is that binary SPC frames carry only 1 bit per channel and therefore lose intensity and color gradations that are required by standard NeRF or 3DGS pipelines. The paper models binary detection probability per pixel as

$I$7

and uses a simulator with quantum efficiency $I$8 and dead-time $I$9 to generate thresholded 1-bit-per-channel color SPC frames from RGB data (Sharma et al., 7 Jun 2025).

Stage 1 uses Pix2PixHD with a coarse-to-fine generator and multi-scale discriminators. Training combines adversarial, $\rightarrow$00, perceptual, and feature-matching losses,

$\rightarrow$01

and the paper emphasizes that the feature-matching term was empirically crucial for SPC$\rightarrow$02RGB translation. Training data are drawn from NeRF-LLFF, with about 15,000 augmented images created by zooming, rotation, shearing, and flipping. Because the SPC frames are simulated from posed RGB images, the translated RGB outputs remain aligned to the original camera poses (Sharma et al., 7 Jun 2025).

Stage 2 uses 3D Gaussian Splatting as the preferred reconstruction backend. The scene is represented as anisotropic Gaussians with means $\rightarrow$03, covariances $\rightarrow$04, opacities $\rightarrow$05, and colors $\rightarrow$06. Projection and rasterization follow the standard 3DGS formulation,

$\rightarrow$07

and optimization minimizes photometric loss against the translated RGB views. The paper reports 30,000 iterations for 3DGS, versus approximately 50,000 iterations for NeRF, and identifies 3DGS as faster and more resilient to moderate imperfections in the translated images (Sharma et al., 7 Jun 2025).

On IBRNet-Collected-2 scenes, the main two-stage results are Pix2PixHD + NeRF with PSNR $\rightarrow$08, SSIM $\rightarrow$09, and LPIPS $\rightarrow$10, and Pix2PixHD + 3DGS (Pix2SG) with PSNR $\rightarrow$11, SSIM $\rightarrow$12, and LPIPS $\rightarrow$13. Directly training NeRF or 3DGS on binary SPC frames performs worse: NeRF yields $\rightarrow$14, $\rightarrow$15, $\rightarrow$16, while 3DGS yields $\rightarrow$17, $\rightarrow$18, $\rightarrow$19. Qualitatively, the paper attributes the improvement to the image prior learned by Pix2PixHD, which restores color and texture consistency across views and thereby stabilizes geometry and radiance estimation (Sharma et al., 7 Jun 2025).

The paper also states several limitations: a domain gap between simulated and real SPC sensors, hallucination risks in the image-to-image translation stage, and failure under extreme photon scarcity or in scenes with repetitive patterns and specularities. Denoising-first alternatives such as Restormer and DiffBIR underperformed before 3D reconstruction (Sharma et al., 7 Jun 2025).

6. Pix2SG as pix2pix-based SAR synthesis from masks

In "Synthetic Glacier SAR Image Generation from Arbitrary Masks Using Pix2Pix Algorithm" (Dietrich-Sussner et al., 2021), the paper itself does not name its method Pix2SG; in the overview associated with the present term, Pix2SG denotes the paper’s pix2pix-based conditional GAN that synthesizes SAR glacier imagery from binary masks. The input masks separate “glacier and rocks” from “ocean,” effectively encoding calving-front geometry. Preprocessed SAR acquisitions are cropped to fjord areas of interest, converted into non-overlapping $\rightarrow$20 mask-image patch pairs, and split into 2003 training patches and 223 validation patches. The SAR archive spans ERS-1/2, Envisat ASAR, RADARSAT-1, ALOS PALSAR, TerraSAR-X, and TanDEM-X from 1995 to 2014. Multi-looking is applied to reduce speckle, and geocoding and orthorectification use the ASTER DEM (Dietrich-Sussner et al., 2021).

The generator is a U-Net with 16 layers total, comprising 8 encoder and 8 decoder stages with skip connections. Encoder blocks use Conv-BatchNorm-LeakyReLU with slope 0.2, except the first encoder layer, which omits BatchNorm. Decoder blocks use Conv-BatchNorm-Dropout-ReLU in the first three layers, then Conv-BatchNorm-LeakyReLU, and a final Conv + Tanh. A “gen5” variant enlarges generator kernels to $\rightarrow$21. The discriminator is a 5-layer PatchGAN with $\rightarrow$22 kernels and stride 2, producing a $\rightarrow$23 output grid over a $\rightarrow$24 input, so that each decision corresponds to a $\rightarrow$25 receptive field. A “dis3” variant reduces discriminator kernels to $\rightarrow$26 (Dietrich-Sussner et al., 2021).

Training follows the standard pix2pix objective,

$\rightarrow$27

with an $\rightarrow$28 reconstruction term,

$\rightarrow$29

and full objective

$\rightarrow$30

using the original pix2pix weighting $\rightarrow$31, Gaussian weight initialization with mean 0 and standard deviation 0.02, Adam at learning rate 0.0002, and 300 epochs with checkpoints every 15 epochs (Dietrich-Sussner et al., 2021).

Evaluation uses non-binary Dice coefficient and SSIM. At epoch 210, the original model (“orig”) reports Dice $\rightarrow$32 and SSIM $\rightarrow$33; “gen5” reports Dice $\rightarrow$34 and SSIM $\rightarrow$35; “dis3” reports Dice $\rightarrow$36 and SSIM $\rightarrow$37; “l11,gan100” reports Dice $\rightarrow$38 and SSIM $\rightarrow$39; and “l150,gan50” reports Dice $\rightarrow$40 and SSIM $\rightarrow$41. Qualitatively, the paper observes that gen5 yields more natural textures and better-defined glacier boundaries, orig can be overly noisy, and dis3 can produce repetitive structures in homogeneous regions. The method is proposed as a way to augment downstream segmentation or calving-front detection datasets, but the paper does not report downstream mIoU, F1, or overall accuracy gains (Dietrich-Sussner et al., 2021).

The limitations are explicitly remote-sensing-specific: mixed sensors and acquisition geometries induce domain shift, physical realism is learned statistically rather than through a physics-informed loss, and no explicit radiometric calibration to $\rightarrow$42 or $\rightarrow$43 is described. Suggested remedies include physics-aware priors, radiometry normalization, incidence-angle correction, speckle-aware losses, and multi-modal conditioning such as DEM or elevation channels (Dietrich-Sussner et al., 2021).

7. Comparative interpretation and recurring design patterns

Across these papers, Pix2SG names methods that differ in output ontology, supervision, and evaluation protocol. The LF-SGG version predicts symbolic relation quintuples without localization and is evaluated by Recall@K or graph-level Precision/Recall/F1 after heuristic graph matching (Özsoy et al., 2023). The robotics version produces a multi-layer semantic-geometric graph anchored to a de-noised 2D BIM-like map and a segmented 3D point cloud, with structural segmentation measured by mIoU, Precision, and Recall (Longo et al., 27 Jun 2025). The SD$\rightarrow$44 version infers discrete 3DSGs inside a dual image-text generative system and is assessed by downstream SI2T/ST2I metrics together with Triplet Recall for 3DSG fidelity (Zhao et al., 2024). The SPC version targets novel-view synthesis and is evaluated by PSNR, SSIM, and LPIPS (Sharma et al., 7 Jun 2025). The glacier SAR version is not a scene-graph model at all and instead synthesizes radar imagery, evaluated by non-binary Dice and SSIM (Dietrich-Sussner et al., 2021).

These distinctions matter because direct comparison across Pix2SG usages is generally invalid. A frequent source of confusion is to treat the term as if it identified one benchmark family or one algorithmic lineage. The papers do not support that reading. Instead, they suggest a naming pattern in which “Pix2SG” or closely related forms are attached to pipelines that convert pixel-space observations into more structured, semantically richer, or more generative representations. A plausible implication is that the term is best understood historically and contextually: in scene-graph work it refers to graph prediction from images; in robotics it refers to real-time semantic-geometric map-graph integration; in spatial generative modeling it refers to image-conditioned 3DSG inference; in SPC reconstruction it refers to Pix2PixHD plus 3DGS; and in SAR synthesis it is an editorial shorthand for mask-conditioned pix2pix generation (Özsoy et al., 2023, Longo et al., 27 Jun 2025, Zhao et al., 2024, Sharma et al., 7 Jun 2025, Dietrich-Sussner et al., 2021).

Another recurring pattern is the use of intermediate structured priors to compensate for missing information. LF-SGG removes location labels and relies on autoregressive consistency plus HTS matching. Pix2G reduces 3D complexity by segmenting in the 2D BEV domain and lifting back to 3D. SD$\rightarrow$45 regularizes image$\rightarrow$463D inference with easier $\rightarrow$47 paths. The SPC pipeline restores plausible RGB appearance before 3D reconstruction. The glacier SAR method uses mask-conditioned adversarial synthesis to compensate for limited labeled radar data. This suggests that, despite the terminological ambiguity, Pix2SG-labeled systems repeatedly occupy the same methodological niche: they use learned priors and structured intermediate representations to bridge a gap between impoverished image observations and a more operational output space.