---
title: 'Anatomy-to-Anatomy Generation: Methods & Applications'
url: https://www.emergentmind.com/topics/anatomy-to-anatomy-generation
type: topic
---

# Anatomy-to-Anatomy Generation: Methods & Applications

Anatomy-to-anatomy generation denotes a class of generative and predictive methods in which the learned object is anatomy itself rather than image intensity. The output may be a segmentation mask, a surface mesh, a signed distance field, a skeletal description, a multi-class probability field, or a future anatomical state; the input may be noise, partial anatomy, interpretable geometric controls, or a baseline anatomy to be evolved forward. In contemporary medical-imaging research, the term therefore covers unconditional mask synthesis, controllable shape editing, completion of missing structures, sequential multi-organ generation, and longitudinal anatomical evolution under disease or treatment [2602.10167] [2504.03313] [2604.12969] [2509.25280].

## 1. Definition and conceptual scope

In its strict sense, anatomy-to-anatomy generation means that the model operates in a semantic anatomical domain at both training and sampling time. A canonical example is unconditional synthesis of multi-class brain segmentation masks from pure Gaussian noise, with no CT image provided at inference; in that formulation, the model learns a distribution $p_\theta(x)$ over anatomical masks, optionally conditioned on a scalar indicating lesion presence [2602.10167]. The same logic appears in thyroid shape synthesis from signed distance functions, where a latent code is edited to change volume, isthmus area, or symmetry while preserving the rest of the anatomy [2504.03313].

The scope is broader than unconditional sampling. Some systems map incomplete anatomy to complete anatomy, as in 3D completion of missing organs, vessels, muscles, and bones from partial segmentations [2309.04956]. Others compose full assemblies from heterogeneous part datasets, producing “virtual chimaeras” from unpaired or partially overlapping cardiac substructures [2210.01607]. A further branch predicts future anatomy from baseline anatomy and subject information, either through semi-parametric deformation models conditioned on genetic and clinical data or through cross-diffusion digital twins that evolve a multi-class anatomical state forward in time [2010.04757] [2509.25280].

A useful distinction in the literature is between anatomy-to-anatomy generation and adjacent anatomy-guided generation. In the former, the primary output remains anatomical; in the latter, anatomy acts as control for CT, X-ray, or report generation. This distinction is explicit in work that generates brain masks rather than CT images and is also implicit in joint structure-aware image models that treat anatomy as the central latent variable [2602.10167] [2310.03559] [2403.09240].

## 2. Anatomical representations

The field is organized less by modality than by representation. One common representation is the discrete segmentation mask. In brain NCCT, anatomy can be encoded as one-hot tensors $X \in \{0,1\}^{C \times H \times W}$ with $C=7$ classes at $256 \times 256$ resolution, including a dedicated ischemic infarct class [2602.10167]. In abdominal synthesis, organ masks are converted into signed distance fields on a fixed $128^3$ grid, allowing smooth geometric conditioning and diffusion over shape rather than label maps [2604.12969].

A second major representation is the implicit field. Thyroid synthesis uses a coordinate-based MLP $f_\theta(\mathbf{x}, \mathbf{z}_i) \approx SDF_i(\mathbf{x})$ with a 64-dimensional latent code; the zero level set defines the organ surface and naturally supports topology changes such as agenesis of the isthmus [2504.03313]. A related development is skeletal latent diffusion, in which surface point clouds are converted into differentiable skeletons with radii, then encoded into skeletal latent sets and decoded back into signed distance fields and meshes [2603.07504].

Mesh- and graph-based representations are equally prominent. Whole-heart “virtual chimaeras” are modeled as assemblies of triangular surface meshes with shared connectivity, treated as graphs and composed by affine and non-rigid graph-convolutional registration modules [2210.01607]. In cardiac factor models, anatomy is represented as a spatial tensor $\mathbf{C} \in \{0,1\}^{K \times H \times W}$ whose channels correspond to disentangled cardiac structures such as LV cavity, myocardium, and RV [2107.01748].

Some methods use discrete latent anatomy tokens rather than explicit masks or meshes. High-resolution 3D brain MRI synthesis employs a 3D VQ-VAE to quantize anatomy into a discrete latent grid, followed by an autoregressive Performer that models the token sequence [2209.03177]. At the opposite end of the spectrum are continuous probabilistic state fields. In anatomy-oriented digital twins, anatomy is represented as $p : \Omega \times [0,T] \to \Delta^{K-1}$, with each voxel carrying a simplex-constrained distribution over tissue classes [2509.25280].

These representational choices encode different anatomical priors. Segmentation masks favor discrete semantics and adjacency constraints; SDFs favor surface smoothness and topology flexibility; mesh graphs favor correspondence and assembly; simplex fields favor exclusivity and temporal evolution.

## 3. Generative mechanisms

A dominant pattern is the two-stage latent generative model. In brain mask synthesis, a VAE first learns a 256-dimensional anatomical latent space from segmentation masks alone, and a latent DDPM-like model then generates new latent codes from noise before a frozen decoder maps them back to masks [2602.10167]. AbdomenGen follows a related structure organ by organ: each organ has its own conditional 3D diffusion U-Net that predicts an SDF given a body-mask SDF, a context SDF aggregating previously generated organs, and an interpretable volume control scalar [2604.12969].

Implicit neural representations provide a second mechanism. Thyroid synthesis uses a single MLP over coordinates and latent codes, with part of the latent fixed to explicit anatomical measurements and the remaining dimensions trainable [2504.03313]. Skeletal latent diffusion extends this by inserting a differentiable skeletonization module and training diffusion directly in a structurally constrained latent point set, followed by implicit SDF decoding and Marching Cubes [2603.07504].

Autoregression remains important for large 3D data. The morphology-preserving brain model compresses $160 \times 224 \times 160$ T1w MRI volumes into approximately 1400 discrete latent tokens and models them with a 24-layer Performer using linear attention, local attention heads, and ReZero stabilization [2209.03177]. This design trades sampling speed for high-resolution morphology preservation.

Not all anatomy-to-anatomy generators are diffusion-based. Anatomy Completor formulates completion as a 3D denoising auto-encoder mapping incomplete multi-anatomy volumes to either full anatomy or residual missing anatomy, with aggregated Dice losses across multiple corruption patterns [2309.04956]. Cardiac disentangled anatomy arithmetic constructs new anatomies by swapping channels in a spatial anatomy tensor and using a localized noise injection network to repair overlaps and restore plausibility [2107.01748]. Whole-heart compositional generation separates part-wise gcVAE or mcVAE shape modeling from a self-supervised spatial composition network that assembles parts into coherent multi-structure hearts [2210.01607].

A distinct line replaces purely statistical generation with dynamical evolution. Anatomy-DT evolves anatomical probability fields with a cross-diffusion reaction-diffusion PDE, implicit-explicit time integration, simplex projection, and topology-aware losses, thereby treating anatomy generation as spatiotemporal state propagation rather than one-shot sampling [2509.25280].

## 4. Control, editing, and compositionality

Controllability ranges from coarse prompts to directly interpretable geometric parameters. The brain mask model offers binary lesion control through a learned embedding $p(y)$ with $y \in \{0,1\}$, concatenated with the latent and normalized timestep in the denoiser input; this yields lesion-present or lesion-free samples while leaving lesion size and location to the learned conditional distribution [2602.10167]. A more structured approach appears in thyroid synthesis, where three fixed latent dimensions encode volume, isthmus area, and symmetry, allowing edits such as increasing volume or reducing isthmus area while holding the trainable residual code constant [2504.03313].

Cardiac models make the control spatially explicit. Disentangled anatomy arithmetic swaps selected channels of a cardiac anatomy tensor between subjects, optionally combined with morphological erosion or dilation, and then refines the hybrid anatomy with a localized noise injection network before re-entangling it with imaging factors [2107.01748]. Whole-heart virtual chimaeras achieve compositionality at the part level: independently generated LV, RV, LA, RA, and aortic root meshes are assembled by learned affine and non-rigid composition networks using shared boundary vertices as weak labels [2210.01607].

Abdominal synthesis introduces organ-wise scalar control through the Volume Control Scalar. For organ $o_i$, expected organ volume is regressed from body volume, and the standardized residual
$$
v_i = \frac{r_{o_i} - \mu_i}{\sigma_i}
$$
acts as a habitus-adjusted control variable injected by FiLM into the organ-specific diffusion model [2604.12969]. The resulting control is continuous, organ-specific, and calibrated over approximately $[-3,+3]$.

Inference-time guidance provides another form of control without conditional retraining. CardioComposer trains an unconditional latent diffusion model on 3D multi-tissue cardiac segmentations and, at sampling time, applies losses on geometric moments—mass, centroid, and scale-normalized covariance—of selected tissues. Because the guidance loss is decomposed into size, position, and shape terms, these attributes can be activated independently or composed across multiple structures [2509.08015].

Longitudinal models use a different control signal: subject state. Predictive deformation modeling estimates future anatomy from a single baseline scan, time interval, and kernels over genetic, clinical, and baseline image features [2010.04757]. Anatomy-DT conditions future anatomical evolution on baseline anatomy, treatment variables, and PDE parameters, enabling anatomy-to-anatomy generation under intervention scenarios such as surgery, radiotherapy, or chemotherapy [2509.25280].

## 5. Evaluation and empirical evidence

The literature repeatedly shows that generic image metrics are insufficient for anatomical generation. Brain mask synthesis therefore reports qualitative preservation of skull–brain configuration and class semantics, class-distribution matching, and a mask-adapted FID that decreases from 64.43 at epoch 100 to 61.88 at epoch 800 [2602.10167]. Thyroid synthesis evaluates both reconstruction and steerability: the correlation model achieves PCC 0.98 for target volume, 0.93 for isthmus area, and 0.43 for symmetry, while reconstruction Chamfer distance remains $1.60 \pm 0.06$ mm [2504.03313].

For organ generation, geometric fidelity is often measured directly on surfaces or masks. AbdomenGen reports, for liver on 200 held-out test cases, Dice $0.83 \pm 0.05$, ASSD $5.85 \pm 2.34$ mm, HD95 $17.58 \pm 9.35$ mm, and Chamfer $11.70 \pm 4.68$ mm; it also uses Wasserstein-1 matching to tune hepatomegaly synthesis, reducing distributional distance to a MERLIN cohort by 73.6% at liver VCS = 4.9 [2604.12969]. Anatomy Completor reports best global DSC $0.931$ on the 40% threshold test set when residual learning and loss aggregation are combined [2309.04956].

Morphological validation can be population-level rather than instance-level. The autoregressive brain model evaluates synthetic cohorts with voxel-based morphometry and GIF-derived tissue volumes, reporting that synthetic VQ-VAE brains reproduce canonical ventricular and Alzheimer-related atrophy patterns and show no significant differences for most tissue types and subgroups relative to real data [2209.03177]. Virtual chimaeras are assessed with generalisability, specificity, and a clinical acceptance rate based on chamber volume intervals, with VAE-based models outperforming PCA-based shape models even under partial-overlap training [2210.01607].

Topology-sensitive tasks require topology-sensitive metrics. Anatomy-DT combines Dice and HD95 with clDice and an overlap regularizer; on the synthetic Voronoi benchmark, the full model reaches DSC $95.70 \pm 0.30$ and HD95 $1.56 \pm 0.14$, while also preserving topology [2509.25280]. Skeletal latent diffusion evaluates medical shapes with CD, EMD, HD, F1, FID, KID, MMD, coverage, and 1-NNA, showing superior reconstruction and generation quality on MedSDF and vessel datasets [2603.07504].

Downstream utility is another empirical criterion. Cardiac disentangled anatomy arithmetic improves balanced ACDC classification accuracy from 88.3% to 91.4% and segmentation Dice from 84.9 to 86.5 when its synthetic samples are added to training, with similar gains for underrepresented pathologies and vendor domains in M\&Ms [2107.01748].

## 6. Applications, boundaries, and limitations

Applications follow directly from the fact that the generated object is anatomically interpretable. Several papers explicitly target data augmentation, shape-prior learning, or image synthesis conditioned on generated masks [2602.10167] [2107.01748]. Others target virtual imaging trials and simulation. Thyroid shape synthesis is framed around virtual cohorts with controlled anatomy for imaging studies [2504.03313], while virtual chimaeras are designed for in-silico evaluation of cardiovascular devices [2210.01607]. AbdomenGen is aimed at controllable abdominal phantoms for scanner optimization, dosimetry, and cohort-matched simulation [2604.12969]. A separate workflow based on physical prostate phantoms, automated ultrasound segmentation, 3D reconstruction, and a 3D GAN extends the idea to surgical training and post-resection anatomy generation [2511.19198].

A common boundary in the literature is between anatomy-to-anatomy generation and anatomy-guided image or language generation. MedSyn jointly models CT and anatomical masks as a 4-channel diffusion process over $(x,l,a,v)$ and supports marginalization-based structure-to-image synthesis [2310.03559]. XReal uses anatomy and pathology masks to control chest X-ray diffusion without fine-tuning the backbone [2403.09240]. Anatomy-guided radiology report generation restructures the visual representation into 29 anatomical regions with pathology-aware prompts for report synthesis [2411.10789]. These works share the same anatomical control philosophy, but the primary output is no longer anatomy itself.

Limitations recur across paradigms. Slice-wise generators may lack explicit 3D consistency [2602.10167]. Some controls remain weak or partial, as with thyroid symmetry compared with volume and isthmus area [2504.03313]. Sequential generators can accumulate error across organs [2604.12969]. Completion and assembly methods depend strongly on segmentation quality or on fixed correspondence and topology assumptions [2309.04956] [2210.01607]. Dynamical digital twins remain deterministic and do not yet provide explicit uncertainty quantification [2509.25280]. High-resolution brain generation has demonstrated morphology preservation, but explicit conditional control, privacy auditing, and faster sampling remain open issues [2209.03177].

Taken together, the literature suggests that anatomy-to-anatomy generation is becoming a general design paradigm rather than a single model family. Its unifying principle is that anatomy is treated as the primary state variable: learned directly, edited directly, composed directly, or evolved directly. A plausible implication is that future systems will increasingly combine these elements—structured representations, interpretable control variables, topology-aware constraints, and downstream simulation—to move from anatomy synthesis toward anatomy programming.

Source: https://www.emergentmind.com/topics/anatomy-to-anatomy-generation