---
title: Spatially-Structured Co-Generation
url: https://www.emergentmind.com/topics/spatially-structured-co-generation
type: topic
---

# Spatially-Structured Co-Generation

Spatially-structured co-generation denotes a family of methods in which spatial organization is generated jointly with another channel—appearance, geometry, dynamics, interference, or design alternatives—rather than added after the fact. The phrase is used explicitly in CoInteract for human-object interaction video synthesis, where an RGB appearance stream and an auxiliary HOI structure stream are jointly trained inside a shared Diffusion Transformer [2604.19636]. Across adjacent literatures, closely related ideas appear in joint region-wise diffusion schedules [2606.07079], spatial-context-driven 3D scene generation [2505.20129], graph-and-dynamics synthesis for power grids [2509.12212], and structured optical or quantum state engineering in which phase, polarization, orbital angular momentum, or two-photon interference are produced together [2306.13821]. Taken together, these works suggest a broader paradigm in which spatial structure is treated as a first-class generative variable rather than a downstream constraint.

## 1. Conceptual scope and defining characteristics

In the machine learning literature, spatially-structured co-generation most often refers to architectures that jointly model a primary output and an auxiliary spatial representation under shared parameters. In CoInteract, the primary output is an RGB HOI video, while the auxiliary channel is a pixel-aligned HOI structure representation built from projected human mesh and fused object mask; both streams are tokenized separately, passed through the same DiT blocks, and supervised jointly so that interaction geometry becomes part of the backbone’s internal representation [2604.19636]. In AsyncPatch Diffusion, the shared object is a single image whose spatial tokens are assigned different diffusion times, so that clean, partially corrupted, and highly noisy regions coexist within one joint state vector rather than following one global denoising clock [2606.07079].

In scene and graph generation, the same pattern appears at a larger structural scale. Agentic 3D scene generation introduces a spatial context \(C=(S,P,G)\) composed of a scene portrait, a semantically labeled point cloud, and a scene hypergraph, and the VLM repeatedly reads and updates that context during generation and editing [2505.20129]. PowerGrow factorizes the joint distribution of grid topology, node features, edge attributes, and temporal dynamics into a staged chain so that structure and dynamics are not generated independently [2509.12212]. Luminate extends the idea into human-AI co-creation by arranging generated alternatives on a spatial canvas organized by generated dimensions and values, thereby turning design-space exploration into a structured spatial process rather than a serial chat interaction [2310.12953].

A plausible cross-domain interpretation is that spatially-structured co-generation is defined less by a single formalism than by three recurring commitments: spatial variables are explicitly represented, coupled objects are generated jointly rather than post hoc stitched together, and the coupling is designed to improve coherence, plausibility, or controllability.

## 2. Dual-stream and region-wise formulations in generative modeling

CoInteract provides the most explicit formulation. It is motivated by two failure modes in HOI video synthesis: structural collapse in sensitive human regions, especially hands and faces, and physical violations such as hand-object interpenetration. Its response is a dual-stream training architecture embedded in a shared Diffusion Transformer, with RGB latent \(\mathbf{z}_r\) and HOI-structure latent \(\mathbf{z}_h\), modality-specific patch embeddings of the same patch size, shared transformer parameters, and stream-specific adaptive layer-normalization scale and shift parameters [2604.19636]. The auxiliary stream is described as “a silhouette-like 3-channel rendering obtained by projecting the recovered human mesh to the image plane and fusing the projected object mask,” constructed from object masks extracted by SAM3 and human mesh recovered with SAM3D-body.

The joint objective is
\[
\mathcal{L}_{flow}=\mathcal{L}_r+\lambda_h\mathcal{L}_h,
\]
with \(\lambda_h=1\) unless otherwise stated, so the HOI branch regularizes shared DiT weights through direct structure supervision. Communication between streams is handled by Two-Stage Asymmetric Co-Attention. Stage 1 uses bidirectional attention; Stage 2 uses an asymmetric mask in which RGB queries attend only to RGB tokens, while HOI queries attend to both RGB and HOI tokens. This design preserves train-time structural regularization while allowing the HOI branch to be removed at inference for “zero-overhead RGB generation” in the sense specific to the auxiliary branch. The reported ablation is sharp: removing co-generation reduces VLM-QA from \(0.72\) to \(0.48\), described as a \(33.3\%\) decrease, while the full model attains the highest VLM-QA \(=0.72\) and HQ \(=0.724\) among compared methods [2604.19636].

AsyncPatch Diffusion addresses a related limitation from the perspective of diffusion time rather than auxiliary supervision. It defines a tokenized sample
\[
\mathbf{x}=\{\mathbf{x}_1,\mathbf{x}_2,\dots,\mathbf{x}_N\},
\]
assigns each token its own diffusion time \(\mathbf{t}=(t_1,\dots,t_N)\in[0,1]^N\), and uses the forward marginal
\[
q(\mathbf{z}_{\mathbf{t}}\mid \mathbf{x})=\prod_{i=1}^N \mathcal{N}(\alpha_{t_i}\mathbf{x}_i,\sigma_{t_i}^2\mathbf{I}).
\]
The model learns a joint score network over the entire asynchronously corrupted image, and the paper proves what it describes as the first valid ELBO for this process [2606.07079]. The point is not merely masked denoising: each region can follow a different schedule while remaining part of one coherent joint distribution. Empirically, AsyncPatch preserves unconditional generation quality close to standard diffusion while being natively suited for inpainting and other spatially adaptive schedules; for ImageNet-64 generation, baseline diffusion reports FID \(1.74\) and AsyncPatch sampling reports \(1.77\), while inpainting improves substantially relative to RePaint on several mask types [2606.07079].

These two systems instantiate complementary versions of the same idea. CoInteract injects an auxiliary geometric stream into a shared backbone and then discards that stream at test time. AsyncPatch keeps a single stream but promotes diffusion time itself to a spatial field. This suggests two major design lineages within spatially-structured co-generation: explicit multi-stream supervision and intrinsic region-wise state parameterization.

## 3. Holistic articulated motion and 3D world modeling

In articulated generation, spatially-structured co-generation appears as joint modeling of multiple correlated body channels or scene components. 3DGesPolicy formulates holistic co-speech gesture generation as a continuous trajectory-control problem. The generated state per frame is
\[
g^{1:N}\in \mathbb{R}^{55\times 6+100+4+3},
\]
combining \(55\) joints in Rot6D, \(100\) FLAME parameters, \(4\) foot-contact labels, and \(3\) global translations [2601.18451]. Rather than directly regressing poses, the method predicts an action sequence \(a^{1:N}_0\) and reconstructs motion by temporal accumulation,
\[
g^n_0=g_{temp}+\sum_{i=1}^{n} a^i_0\cdot \Delta t.
\]
The paper’s claim is that part-based or frame-level formulations weaken spatial coherence, whereas a unified action space lets body pose, facial expression, translation, and contact evolve as one articulated trajectory. Quantitatively, the full model reports FGD \(=3.190\), DIV \(=14.03\), MSE \(=7.020\), and LVD \(=2.200\); the ablation without the action formulation degrades to FGD \(23.563\), MSE \(82.530\), and LVD \(18.030\) [2601.18451].

Agentic 3D scene generation generalizes the principle from articulated bodies to full environments. Its spatial context
\[
C=(S,P,G)
\]
contains a scene portrait \(S\), a semantically labeled point cloud
\[
P=\{(\mathbf{x}_i,\mathbf{c}_i,l_i)\}_{i=1}^{N},
\]
and a scene hypergraph \(G=(V,E)\) [2505.20129]. The point cloud is reconstructed with Fast3R and labeled with Grounded-SAM; the hypergraph stores unary, binary, and ternary spatial relations, with the explicitly named set
\[
R=\{\text{clearance},\text{contact},\text{alignment},\text{equidistance},\text{symmetry}\}.
\]
When an instance \(v\) is modified, the point cloud is updated locally as
\[
P \leftarrow (P\setminus P_v)\cup \hat{P}_v.
\]
Generated meshes are then aligned to instance geometry by optimizing scale, rotation, and translation against nearest neighbors in \(P_v\), and ergonomic adjustment is posed as a hypergraph-constrained optimization:
\[
\min_{\{R_v,\mathbf{t}_v\}_{v\in V}} \sum_{e\in E}\lambda_{r_e}\cdot L_{r_e}(\{R_v,\mathbf{t}_v\}_{v\in e}).
\]
The reported comparisons show the method achieving CLIP \(0.385\), BLIP \(0.737\), LPIPS \(0.571\), and the best aesthetic-quality and functional-plausibility rankings among the listed baselines [2505.20129].

A notable commonality between 3DGesPolicy and the agentic 3D scene system is that both treat the generated object as a structured state evolving under updates rather than as a stack of independent local predictions. This suggests that, in embodied domains, spatially-structured co-generation often takes the form of stateful trajectory or world-model generation.

## 4. Graph-structured infrastructure and co-generation of dynamics

PowerGrow formulates power-system synthesis as a joint problem over a spatial graph with attached dynamics:
\[
\mathcal{G}=(X,A,E,D),
\]
where \(A\) is adjacency, \(X\) node features, \(E\) edge features, and \(D\) nodal time series [2509.12212]. Its central mechanism is dependence decomposition. Rather than learning one monolithic reverse model over structure and dynamics, it factorizes the generation process into
\[
(A,X)\rightarrow E\rightarrow H\rightarrow D,
\]
with \(H\) a latent embedding of the temporal load trajectories. The corresponding factorization is
\[
p(A,X,E,H)=p_\theta(A,X)\,p_\gamma(E\mid A,X)\,p_\phi(H\mid A,X,E),
\]
followed by deterministic decoding of \(H\) back to \(D\). The structural stages use hierarchical graph beta-diffusion, while the temporal stage uses an LSTM autoencoder with latent dimension \(d_4=32\) in the reported experiments.

The practical motivation is feasibility. PowerGrow does not impose hard power-flow constraints inside diffusion; instead, it narrows the feasible space by staged conditioning and by training on feasible systems. Evaluation is then carried out with AC power-flow constraints on voltages, line thermal limits, and generator limits. On the \(36\)-bus system, PowerGrow reports Deg. MMD \(=0.0006\), Clus. MMD \(=0.0037\), Orbit MMD \(=0.0018\), Spec. MMD \(=0.0061\), Time. MMD \(=0.0891\), Attr. MMD \(=0.0077\), convergence rate \(=98.9\%\), and feasibility score \(=0.967\). Its N-1 contingency resilience is reported as \(73.19\%\), compared with \(72.91\%\) for the reference true \(36\)-bus grid and \(65.24\%\) for random walk [2509.12212].

An earlier precursor in synthetic power-grid generation approached a related problem from the angle of joint spatial embedding and topology. The Geographical Network Learner and Generator first fits a Gaussian Mixture Model to node positions, samples new coordinates, then connects them using TWST and a reinforcement procedure in which edge probabilities depend on Euclidean distance, local density, and degree [1508.04447]. The method was evaluated on WI, SERC, and FRCC, explicitly comparing average path length, clustering coefficient, degree distributions, and Euclidean line-length distributions to those of the original grids. Although this work does not use the phrase co-generation, it models node placement and graph construction as coupled rather than independent processes.

Within infrastructure synthesis, spatially-structured co-generation therefore denotes not only joint generation of topology and attributes, but also the conditioning of temporal dynamics on the generated graph scaffold. A plausible implication is that such staged couplings are especially useful when physical feasibility depends on interactions across several heterogeneous variable types.

## 5. Optical and quantum implementations

In photonics and quantum optics, spatially-structured co-generation concerns the joint engineering of spatial profile, polarization, orbital angular momentum, frequency, and multi-photon interference. The “structured quantum eraser” experiment provides a clear quantum example. Two photons are prepared in orthogonal vector-vortex modes, interfere at a 50:50 beam splitter, and are projected by polarization analyzers so that distinguishability becomes a function of transverse position. The coincidence probabilities
\[
C_{(In)}^{\alpha,\beta}(\varphi_1;\varphi_2)
\quad\text{and}\quad
C_{(Out)}^{\alpha,\beta}(\varphi_1;\varphi_2)
\]
differ by coherent subtraction versus incoherent addition of the exchanged amplitudes, and the measured local visibilities include
\[
\mathcal{V}_{HH}=1,\quad \mathcal{V}_{HV}=-1,\quad \mathcal{V}_{AH}=\cos(2\varphi_1),\quad \mathcal{V}_{AV}=-\cos(2\varphi_1),
\]
showing position-dependent bunching, antibunching, and zero-interference sectors across the beam profile [2306.13821]. The paper characterizes this as spatially structured coalescence along the transverse profile.

Frequency-conversion papers show a related co-generation of wavelength and structured field content. In high-harmonic generation, structured near-IR drivers produce EUV vector beams, EUV vortex beams, and EUV vector-vortex beams in a \(15\) mm argon cell, with experimentally confirmed scaling laws
\[
\ell_q=q\,\ell_1
\quad\text{and}\quad
\ell_{pq}=q\,\ell_{p1},
\]
for scalar-vortex and vector-vortex driving, respectively [2207.10730]. In third-harmonic generation with a quasi-periodically poled KTP crystal inside a nonlinear Sagnac interferometer, vector optical fields with space-varying polarization are converted so that OAM is tripled and the vectorial structure is preserved in the harmonic field; for a scalar vortex input, the generated TH field follows \(|V\rangle|m\rangle \mapsto |V\rangle|3m\rangle\), while the more general vector-beam mapping carries cubic amplitude weighting and polarization remapping through the interferometric geometry [2204.04173].

A plasma-based variant appears in forward Raman amplification. Here a weak structured seed—LG, Bessel-Gaussian, or Airy—is co-propagated with a stronger pump in plasma, and the paper reports \(10^4\)–\(10^5\)-fold intensity amplification, output intensities around \(10^{17}\,\mathrm{W/cm^2}\), and sub-cycle or few-cycle compression after nonlinear evolution and self-phase modulation [2506.21979]. In the LG case, the matching includes OAM conservation,
\[
l_0=l_1+l_2,
\]
so the plasma wave can carry compensating OAM, and additional orthogonal-polarization OAM components can be generated under circularly polarized pumping.

The broader review literature places these mechanisms in a common frame. Vector beams are represented as two independently structured polarization components,
\[
\vec{u}(\vec{r}_\perp)=
\begin{pmatrix}
u_h(\vec{r}_\perp)e^{i\Phi_h(\vec{r}_\perp)}\\
u_v(\vec{r}_\perp)e^{i\Phi_v(\vec{r}_\perp)}
\end{pmatrix},
\]
and many practical conversion schemes proceed by splitting the vector field into scalar polarization components, processing them separately through SHG, THG, FWM, or memories, and recombining them coherently [2007.11316]. In that literature, co-generation often means that spatial, polarization, and angular-momentum structure are transformed together rather than sequentially.

## 6. Design-space exploration as spatial co-generation

Luminate extends the idea from physical space to structured design spaces in human-AI co-creation. Instead of returning one answer or an unstructured list, the system first asks an LLM to generate categorical dimensions and values, then samples requirement lists by choosing one value from each dimension, generates responses in parallel, summarizes them for semantic zoom, and places them on a zoomable exploration canvas [2310.12953]. Responses initially appear as a cluster; selecting one dimension organizes them along one axis, and selecting two dimensions organizes them in a 2D layout under value labels. The system’s semantic zoom levels are `DOT`, `TITLE`, `KEYWORD`, `SUMMARY`, and `FULL TEXT`.

The study involved \(14\) professional writers, with system settings of up to \(5\) nominal dimensions, up to \(3\) ordinal dimensions, response length fixed at \(150\) words, and \(40\) responses per prompt. Reported usage averages were \(4.3\) prompts, \(13.8\) responses explored, \(1.8\) dimension-button selections, \(1.1\) unique 1D spaces, and \(1.2\) unique 2D spaces. The Creativity Support Index was \(82.16\) with standard deviation \(5.14\), and the highest factor was Exploration \(=18.21\) with standard deviation \(1.76\) [2310.12953].

Although this work does not concern physical geometry, it is spatially structured in two precise senses given in the paper: responses occupy positions on a literal 2D canvas, and each response is also treated as occupying a point in a multidimensional conceptual space induced by generated dimensions and values. The system therefore demonstrates a non-physical analogue of spatially-structured co-generation: the joint production of alternatives and the space in which those alternatives are explored.

## 7. Recurrent principles, trade-offs, and limitations

Several trade-offs recur across the literature. First, spatial structure often improves realism or controllability by introducing explicit auxiliary channels, but this does not always come without cost. In CoInteract, the asymmetric mask allows RGB-only inference, yet the ablation that keeps bidirectional attention and retains the HOI branch at inference achieves slightly higher VLM-QA \(0.76\) and HQ \(0.738\) at \(4.13\times\) inference cost, indicating an explicit efficiency–accuracy trade-off rather than a free improvement [2604.19636]. In AsyncPatch, supervised inpainting models still outperform the asynchronous model on in-distribution masks, even though AsyncPatch is competitive or stronger on some out-of-distribution cases and does not require task-specific fine-tuning [2606.07079].

Second, many of the strongest results depend on explicit representations that are not present in baseline systems. These include HOI structure renderings in CoInteract, hypergraphs and point clouds in agentic 3D generation, latent temporal embeddings in PowerGrow, and spatially resolved mode projections in the quantum-erasure protocol [2505.20129]. A plausible implication is that spatially-structured co-generation is often less about adding another condition than about redesigning the state space the model or experiment actually operates in.

Third, spatially structured effects can disappear under inappropriate aggregation. In the quantum-erasure experiment, some structured interference patterns vanish under bucket detection and become visible only under spatially resolved conditioned detection [2306.13821]. Luminate shows an HCI analogue: \(40\) responses could become overwhelming, so visible structure itself can create cognitive load unless accompanied by progressive disclosure or other navigation aids [2310.12953].

Finally, current systems remain domain-limited. Agentic 3D generation lists failures for scenes with many or tiny objects and dependence on the geometric foundation model in multi-image settings [2505.20129]. PowerGrow is demonstrated on \(14\)-bus and \(36\)-bus systems, leaving large-scale grids open [2509.12212]. 3DGesPolicy explicitly notes that hand and finger detail remain future work [2601.18451]. These limitations suggest that spatially-structured co-generation is best understood as an emerging design pattern with strong empirical value, but not yet as a universally solved methodology.

Source: https://www.emergentmind.com/topics/spatially-structured-co-generation