Non-Semantic Scene Composition

• Non-semantic scene composition is a paradigm that models visual scenes using spatial, geometric, and appearance cues rather than explicit object identities.
• It underpins techniques in generative modeling, unsupervised learning, and 3D editing, improving layout fidelity and robustness across varied domains.
• Evaluation frameworks like SCSSIM quantify its structural consistency, with applications ranging from image retrieval to autonomous scene understanding.

Non-semantic scene composition refers to the modeling, manipulation, or assessment of visual scenes based solely on structural, spatial, geometric, or appearance cues, strictly decoupled from semantic object identities or categories. This paradigm prioritizes the arrangement, relationships, and harmonious placement of scene elements—such as objects, regions, or color blocks—without leveraging or encoding explicit object-level semantic information. It arises in diverse settings, including compositional generative modeling, abstract scene retrieval, 3D-object insertion, and quantitative evaluation of scene layout fidelity.

1. Conceptual Foundations and Motivation

Non-semantic scene composition addresses the limitations of methods overly reliant on high-level object semantics for inferring or manipulating the structure of visual scenes. Rather than recognizing “what” is in a scene, these approaches encode “how” scene elements are spatially organized, their geometric attributes, color distributions, and composition structure.

Fundamental motivations include:

• Generalization across scene types and content: By decoupling from object classes, non-semantic composition systems support reasoning across unknown or novel domains, as demonstrated by the “things syntax” abstraction and block illustration retrieval paradigm (Kordumova et al., 2016).
• Aesthetic and functional requirements: Many downstream tasks, such as image generation or robotic navigation, demand preservation or enhancement of geometric structure and layout irrespective of content (Haque et al., 7 Aug 2025, Zou et al., 6 May 2026).
• Robustness: Structure-based representations exhibit invariance to textural, color, or minor pixel-level variations, making them robust to noise, blur, or imperfect segmentation (Haque et al., 7 Aug 2025).

This concept underpins advances in fully unsupervised object-centric learning (Chen et al., 2022, Yuan et al., 2021), efficient 3D scene editing (Gao et al., 9 Oct 2025), reference-based and reference-free aesthetic image generation (Zou et al., 6 May 2026), and analytical quality assessment metrics (Haque et al., 7 Aug 2025).

2. Representations and Formalisms

(a) Geometric and Layout-Based Representations

• Things syntax: Encodes each candidate scene element as a vector of observable properties—normalized position (horizontal/vertical), normalized size, aspect ratio, and dominant color label—yielding a scene matrix $$W \in \mathbb{R}^{n \times 4} \times \{1,\dots,11\}$$ for $n$ proposals (Kordumova et al., 2016). This non-semantic encoding omits object identity entirely.
• Scene composition structure (SCS): Models the spatial arrangement and hierarchical partitioning of an image via recursive splitting along strong horizontal/vertical boundaries. This hierarchy is captured by the Cuboidal hierarchical partitioning (CuPID) tree (Haque et al., 7 Aug 2025).
• Intrinsic/extrinsic latent factorization: Scene elements are represented via intrinsic (canonical, context-invariant) embeddings and extrinsic (position, scale, orientation) latents, as seen in GOCL (Chen et al., 2022).
• Surface octahedral probes (SOPs): In 3D, SOPs store local illumination and occlusion data at dense surface points, supporting appearance- and light-transfer without object semantics (Gao et al., 9 Oct 2025).
• Low-frequency composition-aware feature maps: In generative diffusion models, composition is distilled into spatial structure (local contrast maps) and large-scale color distributions, both semantic-agnostic (Zou et al., 6 May 2026).

(b) Layered and Mixture Models

• Pixel-wise Mixtures: Scene generation is modeled as a mixture over $K$ objects plus background, where each mixture weight and appearance is derived from non-semantic, slot-based or spatially transformed features (Chen et al., 2022, Yuan et al., 2021).
• Block-based Abstract Illustrations: Human-drawn colored rectangles encode size/position/layout for querying or retrieval, represented as orderless collections of things-syntax vectors (Kordumova et al., 2016).

3. Learning, Inference, and Composition Strategies

(a) Unsupervised Binding and Decomposition

• Slot Attention and Iterative Integration: Latent slots are inferred from one or more images, factorizing object-centric attributes (viewpoint-invariant) from viewpoint-dependent parameters, and associating scene elements across views via slot-pixel attention or patch-matching (Yuan et al., 2021, Chen et al., 2022).
• Patch-Matching/Prototype Assignment: Intrinsic encodings are matched to a learned dictionary of prototypes using local features and occlusion weights, enabling identification and fusion of occluded or deformed elements without reference to semantic labels (Chen et al., 2022).

(b) Reference-driven and Theme-driven Generation

• Composition-aware guidance: Low-frequency, semantic-agnostic feature maps extracted from reference images are jointly encoded and injected as conditions into generative diffusion pipelines (Union-ControlNet), enabling spatial and color composition control (Zou et al., 6 May 2026).
• LVLM-aided retrieval: When explicit reference images are unavailable, Large Vision-LLMs retrieve compositionally appropriate exemplars using in-context examples and chain-of-thought planning (Zou et al., 6 May 2026). This indirection preserves semantic-agnosticism in the final composition guidance.
• Text-to-composition planning: Fine-tuning on large text-image pairs enables models to infer plausible semantic-free structure and layout directly from textual themes (Zou et al., 6 May 2026).

(c) Surface Lighting and Occlusion for 3D Composition

• Surface Octahedral Probes (SOPs): Instead of per-point ray-tracing, KNN interpolation among SOPs stores indirect illumination and occlusion over discrete hemispheres. This supports relightable compositing and realistic shadowing in 3D Gaussian Splatting fields, decoupled from object class (Gao et al., 9 Oct 2025).

4. Evaluation Methodologies and Metrics

Accurately quantifying non-semantic scene composition demands metrics that focus on layout and structural fidelity.

• Scene Composition Structure Similarity (SCSSIM): Measures the correspondence between the hierarchical spatial partitioning (as defined by CuPID trees) of two images. SCSSIM is invariant to textural or color perturbations and highly sensitive to rearrangements, rotations, or structural manipulations, filling the gap left by pixelwise (MSE, PSNR) and perceptual (SSIM, LPIPS, CLIP) scores (Haque et al., 7 Aug 2025).
• Segmentation and Object Consistency Metrics: ARI, AMI, IoU, OCA, and IACC quantify segmentation, object count/order accuracy, and prototype identification consistency without recourse to semantics (Chen et al., 2022, Yuan et al., 2021).
• Aesthetic and Diversity Metrics: In generative settings, aesthetic score, FID, and CLIP alignment are reported, though only ablations that isolate layout or compositional cycle-consistency losses are diagnostic w.r.t. structural composition (Zou et al., 6 May 2026).

5. Applications

Non-semantic scene composition supports a spectrum of applications:

Domain	Example Mechanism	Principal Citation
Content-agnostic image retrieval	Abstract block illustrations, things syntax histograms	(Kordumova et al., 2016)
Object-centric unsupervised learning	Slot/factor separation, patch-matching	(Chen et al., 2022, Yuan et al., 2021)
3D object-scene compositing	Relightable SOP-based editing, multiview alignment	(Gao et al., 9 Oct 2025)
Generative image modeling	Reference/structure-guided diffusion, text-to-composition	(Zou et al., 6 May 2026)
Structural metric for GenAI	SCSSIM index and layout-fidelity diagnostics	(Haque et al., 7 Aug 2025)

Beyond these, practical applications extend to robotics (scene understanding without semantic priors), video stitching (structure alignment), and automated surveillance or measurement (structural consistency checks).

6. Experimental Findings and Comparative Insights

Key empirical findings consolidate the value of non-semantic composition:

• Retrieval: Scene retrieval based solely on things syntax or block illustrations (mAP up to 12.39%) outperforms, or is competitive with, several object-attribute and off-the-shelf deep semantic features in zero-shot settings (Kordumova et al., 2016).
• Unsupervised Consistency: Models such as GOCL and OCLOC achieve near-perfect object identity consistency (IACC ≈ 0.99) and high ARI for segmentation, without using object-class labels (Chen et al., 2022, Yuan et al., 2021).
• 3D Editing Quality: SOP-based real-time composition yields physically plausible shadows, multi-view consistency, and improved editing throughput compared to baselines (28 FPS, full composition ≈36 s, PSNR ≈24.46 dB) (Gao et al., 9 Oct 2025).
• Aesthetic Image Generation: Composer yields higher LAION-tracked Aesthetic Score (Aes), better FID, and is preferred by humans over baseline and state-of-the-art composition-aware systems, even in reference-free text-to-image settings (Zou et al., 6 May 2026).
• Metric Robustness: SCSSIM displays invariance (>0.98) to non-compositional distortions and monotonic decrease under compositional alteration, outperforming other similarity indices (Haque et al., 7 Aug 2025).

7. Limitations and Future Directions

Principal limitations stem from assumptions about scene stationarity, probe granularity, and coverage:

• 3D SOPs: Assume quasi-static/local occlusion; dynamic scenes or large-scale objects challenge KNN interpolation and require hierarchical or adaptive probe hierarchies (Gao et al., 9 Oct 2025).
• Reference Coverage: Diffusion-based local lighting completion degrades with insufficient panoramic coverage (<40%) (Gao et al., 9 Oct 2025).
• Ambiguity in non-semantic cues: For certain classes of scenes, position/size/aspect alone may be insufficient for fine-grained discrimination or manipulation, motivating hybrid approaches.
• Composition metric scope: SCSSIM and similar metrics abstract away textural or subtle perceptual cues that, while semantically innocuous, affect downstream human or task relevance (Haque et al., 7 Aug 2025).

Future research aims include integrating learned light-transport surrogates for enhanced global illumination, per-texel or region confidence quantification for probes, end-to-end finetuning of composition predictors, and extending structural metrics to multi-modal or hierarchical data. A plausible implication is growing adoption of non-semantic structure priors in generative AI, 3D vision, and autonomous system pipelines.