Spatially-Structured Co-Generation

• Spatially-structured co-generation is a family of methods that jointly generate spatial configurations and auxiliary channels such as appearance or dynamics to improve overall model coherence.
• It typically employs dual-stream architectures or region-wise diffusion schedules to integrate spatial data with other modalities, addressing issues like structural collapse and physical inconsistencies.
• This approach has diverse applications in HOI video synthesis, 3D scene generation, power grid modeling, and optical systems, while balancing trade-offs between efficiency and generalizability.

Spatially-structured co-generation denotes a family of methods in which spatial organization is generated jointly with another channel—appearance, geometry, dynamics, interference, or design alternatives—rather than added after the fact. The phrase is used explicitly in CoInteract for human-object interaction video synthesis, where an RGB appearance stream and an auxiliary HOI structure stream are jointly trained inside a shared Diffusion Transformer (Luo et al., 21 Apr 2026). Across adjacent literatures, closely related ideas appear in joint region-wise diffusion schedules (Papa et al., 5 Jun 2026), spatial-context-driven 3D scene generation (Liu et al., 26 May 2025), graph-and-dynamics synthesis for power grids (He et al., 29 Aug 2025), and structured optical or quantum state engineering in which phase, polarization, orbital angular momentum, or two-photon interference are produced together (Schiano et al., 2023). Taken together, these works suggest a broader paradigm in which spatial structure is treated as a first-class generative variable rather than a downstream constraint.

1. Conceptual scope and defining characteristics

In the machine learning literature, spatially-structured co-generation most often refers to architectures that jointly model a primary output and an auxiliary spatial representation under shared parameters. In CoInteract, the primary output is an RGB HOI video, while the auxiliary channel is a pixel-aligned HOI structure representation built from projected human mesh and fused object mask; both streams are tokenized separately, passed through the same DiT blocks, and supervised jointly so that interaction geometry becomes part of the backbone’s internal representation (Luo et al., 21 Apr 2026). In AsyncPatch Diffusion, the shared object is a single image whose spatial tokens are assigned different diffusion times, so that clean, partially corrupted, and highly noisy regions coexist within one joint state vector rather than following one global denoising clock (Papa et al., 5 Jun 2026).

In scene and graph generation, the same pattern appears at a larger structural scale. Agentic 3D scene generation introduces a spatial context $C=(S,P,G)$ composed of a scene portrait, a semantically labeled point cloud, and a scene hypergraph, and the VLM repeatedly reads and updates that context during generation and editing (Liu et al., 26 May 2025). PowerGrow factorizes the joint distribution of grid topology, node features, edge attributes, and temporal dynamics into a staged chain so that structure and dynamics are not generated independently (He et al., 29 Aug 2025). Luminate extends the idea into human-AI co-creation by arranging generated alternatives on a spatial canvas organized by generated dimensions and values, thereby turning design-space exploration into a structured spatial process rather than a serial chat interaction (Suh et al., 2023).

A plausible cross-domain interpretation is that spatially-structured co-generation is defined less by a single formalism than by three recurring commitments: spatial variables are explicitly represented, coupled objects are generated jointly rather than post hoc stitched together, and the coupling is designed to improve coherence, plausibility, or controllability.

2. Dual-stream and region-wise formulations in generative modeling

CoInteract provides the most explicit formulation. It is motivated by two failure modes in HOI video synthesis: structural collapse in sensitive human regions, especially hands and faces, and physical violations such as hand-object interpenetration. Its response is a dual-stream training architecture embedded in a shared Diffusion Transformer, with RGB latent $\mathbf{z}_r$ and HOI-structure latent $\mathbf{z}_h$, modality-specific patch embeddings of the same patch size, shared transformer parameters, and stream-specific adaptive layer-normalization scale and shift parameters (Luo et al., 21 Apr 2026). The auxiliary stream is described as “a silhouette-like 3-channel rendering obtained by projecting the recovered human mesh to the image plane and fusing the projected object mask,” constructed from object masks extracted by SAM3 and human mesh recovered with SAM3D-body.

The joint objective is

$$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$

with $\lambda_h=1$ unless otherwise stated, so the HOI branch regularizes shared DiT weights through direct structure supervision. Communication between streams is handled by Two-Stage Asymmetric Co-Attention. Stage 1 uses bidirectional attention; Stage 2 uses an asymmetric mask in which RGB queries attend only to RGB tokens, while HOI queries attend to both RGB and HOI tokens. This design preserves train-time structural regularization while allowing the HOI branch to be removed at inference for “zero-overhead RGB generation” in the sense specific to the auxiliary branch. The reported ablation is sharp: removing co-generation reduces VLM-QA from $0.72$ to $0.48$, described as a $33.3\%$ decrease, while the full model attains the highest VLM-QA $=0.72$ and HQ $=0.724$ among compared methods (Luo et al., 21 Apr 2026).

AsyncPatch Diffusion addresses a related limitation from the perspective of diffusion time rather than auxiliary supervision. It defines a tokenized sample

$\mathbf{z}_r$0

assigns each token its own diffusion time $\mathbf{z}_r$1, and uses the forward marginal

$\mathbf{z}_r$2

The model learns a joint score network over the entire asynchronously corrupted image, and the paper proves what it describes as the first valid ELBO for this process (Papa et al., 5 Jun 2026). The point is not merely masked denoising: each region can follow a different schedule while remaining part of one coherent joint distribution. Empirically, AsyncPatch preserves unconditional generation quality close to standard diffusion while being natively suited for inpainting and other spatially adaptive schedules; for ImageNet-64 generation, baseline diffusion reports FID $\mathbf{z}_r$3 and AsyncPatch sampling reports $\mathbf{z}_r$4, while inpainting improves substantially relative to RePaint on several mask types (Papa et al., 5 Jun 2026).

These two systems instantiate complementary versions of the same idea. CoInteract injects an auxiliary geometric stream into a shared backbone and then discards that stream at test time. AsyncPatch keeps a single stream but promotes diffusion time itself to a spatial field. This suggests two major design lineages within spatially-structured co-generation: explicit multi-stream supervision and intrinsic region-wise state parameterization.

3. Holistic articulated motion and 3D world modeling

In articulated generation, spatially-structured co-generation appears as joint modeling of multiple correlated body channels or scene components. 3DGesPolicy formulates holistic co-speech gesture generation as a continuous trajectory-control problem. The generated state per frame is

$\mathbf{z}_r$5

combining $\mathbf{z}_r$6 joints in Rot6D, $\mathbf{z}_r$7 FLAME parameters, $\mathbf{z}_r$8 foot-contact labels, and $\mathbf{z}_r$9 global translations (Sha et al., 26 Jan 2026). Rather than directly regressing poses, the method predicts an action sequence $\mathbf{z}_h$0 and reconstructs motion by temporal accumulation,

$\mathbf{z}_h$1

The paper’s claim is that part-based or frame-level formulations weaken spatial coherence, whereas a unified action space lets body pose, facial expression, translation, and contact evolve as one articulated trajectory. Quantitatively, the full model reports FGD $\mathbf{z}_h$2, DIV $\mathbf{z}_h$3, MSE $\mathbf{z}_h$4, and LVD $\mathbf{z}_h$5; the ablation without the action formulation degrades to FGD $\mathbf{z}_h$6, MSE $\mathbf{z}_h$7, and LVD $\mathbf{z}_h$8 (Sha et al., 26 Jan 2026).

Agentic 3D scene generation generalizes the principle from articulated bodies to full environments. Its spatial context

$\mathbf{z}_h$9

contains a scene portrait $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$0, a semantically labeled point cloud

$$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$1

and a scene hypergraph $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$2 (Liu et al., 26 May 2025). The point cloud is reconstructed with Fast3R and labeled with Grounded-SAM; the hypergraph stores unary, binary, and ternary spatial relations, with the explicitly named set

$$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$3

When an instance $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$4 is modified, the point cloud is updated locally as

$$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$5

Generated meshes are then aligned to instance geometry by optimizing scale, rotation, and translation against nearest neighbors in $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$6, and ergonomic adjustment is posed as a hypergraph-constrained optimization: $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$7 The reported comparisons show the method achieving CLIP $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$8, BLIP $$\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,$$9, LPIPS $\lambda_h=1$0, and the best aesthetic-quality and functional-plausibility rankings among the listed baselines (Liu et al., 26 May 2025).

A notable commonality between 3DGesPolicy and the agentic 3D scene system is that both treat the generated object as a structured state evolving under updates rather than as a stack of independent local predictions. This suggests that, in embodied domains, spatially-structured co-generation often takes the form of stateful trajectory or world-model generation.

4. Graph-structured infrastructure and co-generation of dynamics

PowerGrow formulates power-system synthesis as a joint problem over a spatial graph with attached dynamics: $\lambda_h=1$1 where $\lambda_h=1$2 is adjacency, $\lambda_h=1$3 node features, $\lambda_h=1$4 edge features, and $\lambda_h=1$5 nodal time series (He et al., 29 Aug 2025). Its central mechanism is dependence decomposition. Rather than learning one monolithic reverse model over structure and dynamics, it factorizes the generation process into

$\lambda_h=1$6

with $\lambda_h=1$7 a latent embedding of the temporal load trajectories. The corresponding factorization is

$\lambda_h=1$8

followed by deterministic decoding of $\lambda_h=1$9 back to $0.72$0. The structural stages use hierarchical graph beta-diffusion, while the temporal stage uses an LSTM autoencoder with latent dimension $0.72$1 in the reported experiments.

The practical motivation is feasibility. PowerGrow does not impose hard power-flow constraints inside diffusion; instead, it narrows the feasible space by staged conditioning and by training on feasible systems. Evaluation is then carried out with AC power-flow constraints on voltages, line thermal limits, and generator limits. On the $0.72$2-bus system, PowerGrow reports Deg. MMD $0.72$3, Clus. MMD $0.72$4, Orbit MMD $0.72$5, Spec. MMD $0.72$6, Time. MMD $0.72$7, Attr. MMD $0.72$8, convergence rate $0.72$9, and feasibility score $0.48$0. Its N-1 contingency resilience is reported as $0.48$1, compared with $0.48$2 for the reference true $0.48$3-bus grid and $0.48$4 for random walk (He et al., 29 Aug 2025).

An earlier precursor in synthetic power-grid generation approached a related problem from the angle of joint spatial embedding and topology. The Geographical Network Learner and Generator first fits a Gaussian Mixture Model to node positions, samples new coordinates, then connects them using TWST and a reinforcement procedure in which edge probabilities depend on Euclidean distance, local density, and degree (Soltan et al., 2015). The method was evaluated on WI, SERC, and FRCC, explicitly comparing average path length, clustering coefficient, degree distributions, and Euclidean line-length distributions to those of the original grids. Although this work does not use the phrase co-generation, it models node placement and graph construction as coupled rather than independent processes.

Within infrastructure synthesis, spatially-structured co-generation therefore denotes not only joint generation of topology and attributes, but also the conditioning of temporal dynamics on the generated graph scaffold. A plausible implication is that such staged couplings are especially useful when physical feasibility depends on interactions across several heterogeneous variable types.

5. Optical and quantum implementations

In photonics and quantum optics, spatially-structured co-generation concerns the joint engineering of spatial profile, polarization, orbital angular momentum, frequency, and multi-photon interference. The “structured quantum eraser” experiment provides a clear quantum example. Two photons are prepared in orthogonal vector-vortex modes, interfere at a 50:50 beam splitter, and are projected by polarization analyzers so that distinguishability becomes a function of transverse position. The coincidence probabilities

$0.48$5

differ by coherent subtraction versus incoherent addition of the exchanged amplitudes, and the measured local visibilities include

$0.48$6

showing position-dependent bunching, antibunching, and zero-interference sectors across the beam profile (Schiano et al., 2023). The paper characterizes this as spatially structured coalescence along the transverse profile.

Frequency-conversion papers show a related co-generation of wavelength and structured field content. In high-harmonic generation, structured near-IR drivers produce EUV vector beams, EUV vortex beams, and EUV vector-vortex beams in a $0.48$7 mm argon cell, with experimentally confirmed scaling laws

$0.48$8

for scalar-vortex and vector-vortex driving, respectively (Pandey et al., 2022). In third-harmonic generation with a quasi-periodically poled KTP crystal inside a nonlinear Sagnac interferometer, vector optical fields with space-varying polarization are converted so that OAM is tripled and the vectorial structure is preserved in the harmonic field; for a scalar vortex input, the generated TH field follows $0.48$9, while the more general vector-beam mapping carries cubic amplitude weighting and polarization remapping through the interferometric geometry (Lou et al., 2022).

A plasma-based variant appears in forward Raman amplification. Here a weak structured seed—LG, Bessel-Gaussian, or Airy—is co-propagated with a stronger pump in plasma, and the paper reports $33.3\%$0–$33.3\%$1-fold intensity amplification, output intensities around $33.3\%$2, and sub-cycle or few-cycle compression after nonlinear evolution and self-phase modulation (Lei et al., 27 Jun 2025). In the LG case, the matching includes OAM conservation,

$33.3\%$3

so the plasma wave can carry compensating OAM, and additional orthogonal-polarization OAM components can be generated under circularly polarized pumping.

The broader review literature places these mechanisms in a common frame. Vector beams are represented as two independently structured polarization components,

$33.3\%$4

and many practical conversion schemes proceed by splitting the vector field into scalar polarization components, processing them separately through SHG, THG, FWM, or memories, and recombining them coherently (Wang et al., 2020). In that literature, co-generation often means that spatial, polarization, and angular-momentum structure are transformed together rather than sequentially.

6. Design-space exploration as spatial co-generation

Luminate extends the idea from physical space to structured design spaces in human-AI co-creation. Instead of returning one answer or an unstructured list, the system first asks an LLM to generate categorical dimensions and values, then samples requirement lists by choosing one value from each dimension, generates responses in parallel, summarizes them for semantic zoom, and places them on a zoomable exploration canvas (Suh et al., 2023). Responses initially appear as a cluster; selecting one dimension organizes them along one axis, and selecting two dimensions organizes them in a 2D layout under value labels. The system’s semantic zoom levels are DOT, TITLE, KEYWORD, SUMMARY, and FULL TEXT.

The study involved $33.3\%$5 professional writers, with system settings of up to $33.3\%$6 nominal dimensions, up to $33.3\%$7 ordinal dimensions, response length fixed at $33.3\%$8 words, and $33.3\%$9 responses per prompt. Reported usage averages were $=0.72$0 prompts, $=0.72$1 responses explored, $=0.72$2 dimension-button selections, $=0.72$3 unique 1D spaces, and $=0.72$4 unique 2D spaces. The Creativity Support Index was $=0.72$5 with standard deviation $=0.72$6, and the highest factor was Exploration $=0.72$7 with standard deviation $=0.72$8 (Suh et al., 2023).

Although this work does not concern physical geometry, it is spatially structured in two precise senses given in the paper: responses occupy positions on a literal 2D canvas, and each response is also treated as occupying a point in a multidimensional conceptual space induced by generated dimensions and values. The system therefore demonstrates a non-physical analogue of spatially-structured co-generation: the joint production of alternatives and the space in which those alternatives are explored.

7. Recurrent principles, trade-offs, and limitations

Several trade-offs recur across the literature. First, spatial structure often improves realism or controllability by introducing explicit auxiliary channels, but this does not always come without cost. In CoInteract, the asymmetric mask allows RGB-only inference, yet the ablation that keeps bidirectional attention and retains the HOI branch at inference achieves slightly higher VLM-QA $=0.72$9 and HQ $=0.724$0 at $=0.724$1 inference cost, indicating an explicit efficiency–accuracy trade-off rather than a free improvement (Luo et al., 21 Apr 2026). In AsyncPatch, supervised inpainting models still outperform the asynchronous model on in-distribution masks, even though AsyncPatch is competitive or stronger on some out-of-distribution cases and does not require task-specific fine-tuning (Papa et al., 5 Jun 2026).

Second, many of the strongest results depend on explicit representations that are not present in baseline systems. These include HOI structure renderings in CoInteract, hypergraphs and point clouds in agentic 3D generation, latent temporal embeddings in PowerGrow, and spatially resolved mode projections in the quantum-erasure protocol (Liu et al., 26 May 2025). A plausible implication is that spatially-structured co-generation is often less about adding another condition than about redesigning the state space the model or experiment actually operates in.

Third, spatially structured effects can disappear under inappropriate aggregation. In the quantum-erasure experiment, some structured interference patterns vanish under bucket detection and become visible only under spatially resolved conditioned detection (Schiano et al., 2023). Luminate shows an HCI analogue: $=0.724$2 responses could become overwhelming, so visible structure itself can create cognitive load unless accompanied by progressive disclosure or other navigation aids (Suh et al., 2023).

Finally, current systems remain domain-limited. Agentic 3D generation lists failures for scenes with many or tiny objects and dependence on the geometric foundation model in multi-image settings (Liu et al., 26 May 2025). PowerGrow is demonstrated on $=0.724$3-bus and $=0.724$4-bus systems, leaving large-scale grids open (He et al., 29 Aug 2025). 3DGesPolicy explicitly notes that hand and finger detail remain future work (Sha et al., 26 Jan 2026). These limitations suggest that spatially-structured co-generation is best understood as an emerging design pattern with strong empirical value, but not yet as a universally solved methodology.