Anatomy-to-Anatomy Generation: Methods & Applications

• The paper outlines a framework where the model learns a distribution over anatomical representations, enabling unconditional synthesis and controlled shape editing from latent anatomical variables.
• Anatomy-to-anatomy generation employs diverse representations such as segmentation masks, signed distance fields, implicit fields, and mesh graphs to capture detailed anatomical structures.
• Practical applications include data augmentation, diagnostic simulation, and longitudinal prediction, although challenges like 3D consistency and uncertainty quantification persist.

Anatomy-to-anatomy generation denotes a class of generative and predictive methods in which the learned object is anatomy itself rather than image intensity. The output may be a segmentation mask, a surface mesh, a signed distance field, a skeletal description, a multi-class probability field, or a future anatomical state; the input may be noise, partial anatomy, interpretable geometric controls, or a baseline anatomy to be evolved forward. In contemporary medical-imaging research, the term therefore covers unconditional mask synthesis, controllable shape editing, completion of missing structures, sequential multi-organ generation, and longitudinal anatomical evolution under disease or treatment (Borrego et al., 10 Feb 2026, Wilde et al., 4 Apr 2025, Bhandari et al., 14 Apr 2026, Bhattacharya et al., 29 Sep 2025).

1. Definition and conceptual scope

In its strict sense, anatomy-to-anatomy generation means that the model operates in a semantic anatomical domain at both training and sampling time. A canonical example is unconditional synthesis of multi-class brain segmentation masks from pure Gaussian noise, with no CT image provided at inference; in that formulation, the model learns a distribution $p_\theta(x)$ over anatomical masks, optionally conditioned on a scalar indicating lesion presence (Borrego et al., 10 Feb 2026). The same logic appears in thyroid shape synthesis from signed distance functions, where a latent code is edited to change volume, isthmus area, or symmetry while preserving the rest of the anatomy (Wilde et al., 4 Apr 2025).

The scope is broader than unconditional sampling. Some systems map incomplete anatomy to complete anatomy, as in 3D completion of missing organs, vessels, muscles, and bones from partial segmentations (Li et al., 2023). Others compose full assemblies from heterogeneous part datasets, producing “virtual chimaeras” from unpaired or partially overlapping cardiac substructures (Dou et al., 2022). A further branch predicts future anatomy from baseline anatomy and subject information, either through semi-parametric deformation models conditioned on genetic and clinical data or through cross-diffusion digital twins that evolve a multi-class anatomical state forward in time (Dalca et al., 2020, Bhattacharya et al., 29 Sep 2025).

A useful distinction in the literature is between anatomy-to-anatomy generation and adjacent anatomy-guided generation. In the former, the primary output remains anatomical; in the latter, anatomy acts as control for CT, X-ray, or report generation. This distinction is explicit in work that generates brain masks rather than CT images and is also implicit in joint structure-aware image models that treat anatomy as the central latent variable (Borrego et al., 10 Feb 2026, xu et al., 2023, Hashmi et al., 2024).

2. Anatomical representations

The field is organized less by modality than by representation. One common representation is the discrete segmentation mask. In brain NCCT, anatomy can be encoded as one-hot tensors $X \in \{0,1\}^{C \times H \times W}$ with $C=7$ classes at $256 \times 256$ resolution, including a dedicated ischemic infarct class (Borrego et al., 10 Feb 2026). In abdominal synthesis, organ masks are converted into signed distance fields on a fixed $128^3$ grid, allowing smooth geometric conditioning and diffusion over shape rather than label maps (Bhandari et al., 14 Apr 2026).

A second major representation is the implicit field. Thyroid synthesis uses a coordinate-based MLP $$f_\theta(\mathbf{x}, \mathbf{z}_i) \approx SDF_i(\mathbf{x})$$ with a 64-dimensional latent code; the zero level set defines the organ surface and naturally supports topology changes such as agenesis of the isthmus (Wilde et al., 4 Apr 2025). A related development is skeletal latent diffusion, in which surface point clouds are converted into differentiable skeletons with radii, then encoded into skeletal latent sets and decoded back into signed distance fields and meshes (Zhang et al., 8 Mar 2026).

Mesh- and graph-based representations are equally prominent. Whole-heart “virtual chimaeras” are modeled as assemblies of triangular surface meshes with shared connectivity, treated as graphs and composed by affine and non-rigid graph-convolutional registration modules (Dou et al., 2022). In cardiac factor models, anatomy is represented as a spatial tensor $\mathbf{C} \in \{0,1\}^{K \times H \times W}$ whose channels correspond to disentangled cardiac structures such as LV cavity, myocardium, and RV (Thermos et al., 2021).

Some methods use discrete latent anatomy tokens rather than explicit masks or meshes. High-resolution 3D brain MRI synthesis employs a 3D VQ-VAE to quantize anatomy into a discrete latent grid, followed by an autoregressive Performer that models the token sequence (Tudosiu et al., 2022). At the opposite end of the spectrum are continuous probabilistic state fields. In anatomy-oriented digital twins, anatomy is represented as $p : \Omega \times [0,T] \to \Delta^{K-1}$, with each voxel carrying a simplex-constrained distribution over tissue classes (Bhattacharya et al., 29 Sep 2025).

These representational choices encode different anatomical priors. Segmentation masks favor discrete semantics and adjacency constraints; SDFs favor surface smoothness and topology flexibility; mesh graphs favor correspondence and assembly; simplex fields favor exclusivity and temporal evolution.

3. Generative mechanisms

A dominant pattern is the two-stage latent generative model. In brain mask synthesis, a VAE first learns a 256-dimensional anatomical latent space from segmentation masks alone, and a latent DDPM-like model then generates new latent codes from noise before a frozen decoder maps them back to masks (Borrego et al., 10 Feb 2026). AbdomenGen follows a related structure organ by organ: each organ has its own conditional 3D diffusion U-Net that predicts an SDF given a body-mask SDF, a context SDF aggregating previously generated organs, and an interpretable volume control scalar (Bhandari et al., 14 Apr 2026).

Implicit neural representations provide a second mechanism. Thyroid synthesis uses a single MLP over coordinates and latent codes, with part of the latent fixed to explicit anatomical measurements and the remaining dimensions trainable (Wilde et al., 4 Apr 2025). Skeletal latent diffusion extends this by inserting a differentiable skeletonization module and training diffusion directly in a structurally constrained latent point set, followed by implicit SDF decoding and Marching Cubes (Zhang et al., 8 Mar 2026).

Autoregression remains important for large 3D data. The morphology-preserving brain model compresses $160 \times 224 \times 160$ T1w MRI volumes into approximately 1400 discrete latent tokens and models them with a 24-layer Performer using linear attention, local attention heads, and ReZero stabilization (Tudosiu et al., 2022). This design trades sampling speed for high-resolution morphology preservation.

Not all anatomy-to-anatomy generators are diffusion-based. Anatomy Completor formulates completion as a 3D denoising auto-encoder mapping incomplete multi-anatomy volumes to either full anatomy or residual missing anatomy, with aggregated Dice losses across multiple corruption patterns (Li et al., 2023). Cardiac disentangled anatomy arithmetic constructs new anatomies by swapping channels in a spatial anatomy tensor and using a localized noise injection network to repair overlaps and restore plausibility (Thermos et al., 2021). Whole-heart compositional generation separates part-wise gcVAE or mcVAE shape modeling from a self-supervised spatial composition network that assembles parts into coherent multi-structure hearts (Dou et al., 2022).

A distinct line replaces purely statistical generation with dynamical evolution. Anatomy-DT evolves anatomical probability fields with a cross-diffusion reaction-diffusion PDE, implicit-explicit time integration, simplex projection, and topology-aware losses, thereby treating anatomy generation as spatiotemporal state propagation rather than one-shot sampling (Bhattacharya et al., 29 Sep 2025).

4. Control, editing, and compositionality

Controllability ranges from coarse prompts to directly interpretable geometric parameters. The brain mask model offers binary lesion control through a learned embedding $p(y)$ with $X \in \{0,1\}^{C \times H \times W}$0, concatenated with the latent and normalized timestep in the denoiser input; this yields lesion-present or lesion-free samples while leaving lesion size and location to the learned conditional distribution (Borrego et al., 10 Feb 2026). A more structured approach appears in thyroid synthesis, where three fixed latent dimensions encode volume, isthmus area, and symmetry, allowing edits such as increasing volume or reducing isthmus area while holding the trainable residual code constant (Wilde et al., 4 Apr 2025).

Cardiac models make the control spatially explicit. Disentangled anatomy arithmetic swaps selected channels of a cardiac anatomy tensor between subjects, optionally combined with morphological erosion or dilation, and then refines the hybrid anatomy with a localized noise injection network before re-entangling it with imaging factors (Thermos et al., 2021). Whole-heart virtual chimaeras achieve compositionality at the part level: independently generated LV, RV, LA, RA, and aortic root meshes are assembled by learned affine and non-rigid composition networks using shared boundary vertices as weak labels (Dou et al., 2022).

Abdominal synthesis introduces organ-wise scalar control through the Volume Control Scalar. For organ $X \in \{0,1\}^{C \times H \times W}$1, expected organ volume is regressed from body volume, and the standardized residual

$X \in \{0,1\}^{C \times H \times W}$2

acts as a habitus-adjusted control variable injected by FiLM into the organ-specific diffusion model (Bhandari et al., 14 Apr 2026). The resulting control is continuous, organ-specific, and calibrated over approximately $X \in \{0,1\}^{C \times H \times W}$3.

Inference-time guidance provides another form of control without conditional retraining. CardioComposer trains an unconditional latent diffusion model on 3D multi-tissue cardiac segmentations and, at sampling time, applies losses on geometric moments—mass, centroid, and scale-normalized covariance—of selected tissues. Because the guidance loss is decomposed into size, position, and shape terms, these attributes can be activated independently or composed across multiple structures (Kadry et al., 8 Sep 2025).

Longitudinal models use a different control signal: subject state. Predictive deformation modeling estimates future anatomy from a single baseline scan, time interval, and kernels over genetic, clinical, and baseline image features (Dalca et al., 2020). Anatomy-DT conditions future anatomical evolution on baseline anatomy, treatment variables, and PDE parameters, enabling anatomy-to-anatomy generation under intervention scenarios such as surgery, radiotherapy, or chemotherapy (Bhattacharya et al., 29 Sep 2025).

5. Evaluation and empirical evidence

The literature repeatedly shows that generic image metrics are insufficient for anatomical generation. Brain mask synthesis therefore reports qualitative preservation of skull–brain configuration and class semantics, class-distribution matching, and a mask-adapted FID that decreases from 64.43 at epoch 100 to 61.88 at epoch 800 (Borrego et al., 10 Feb 2026). Thyroid synthesis evaluates both reconstruction and steerability: the correlation model achieves PCC 0.98 for target volume, 0.93 for isthmus area, and 0.43 for symmetry, while reconstruction Chamfer distance remains $X \in \{0,1\}^{C \times H \times W}$4 mm (Wilde et al., 4 Apr 2025).

For organ generation, geometric fidelity is often measured directly on surfaces or masks. AbdomenGen reports, for liver on 200 held-out test cases, Dice $X \in \{0,1\}^{C \times H \times W}$5, ASSD $X \in \{0,1\}^{C \times H \times W}$6 mm, HD95 $X \in \{0,1\}^{C \times H \times W}$7 mm, and Chamfer $X \in \{0,1\}^{C \times H \times W}$8 mm; it also uses Wasserstein-1 matching to tune hepatomegaly synthesis, reducing distributional distance to a MERLIN cohort by 73.6% at liver VCS = 4.9 (Bhandari et al., 14 Apr 2026). Anatomy Completor reports best global DSC $X \in \{0,1\}^{C \times H \times W}$9 on the 40% threshold test set when residual learning and loss aggregation are combined (Li et al., 2023).

Morphological validation can be population-level rather than instance-level. The autoregressive brain model evaluates synthetic cohorts with voxel-based morphometry and GIF-derived tissue volumes, reporting that synthetic VQ-VAE brains reproduce canonical ventricular and Alzheimer-related atrophy patterns and show no significant differences for most tissue types and subgroups relative to real data (Tudosiu et al., 2022). Virtual chimaeras are assessed with generalisability, specificity, and a clinical acceptance rate based on chamber volume intervals, with VAE-based models outperforming PCA-based shape models even under partial-overlap training (Dou et al., 2022).

Topology-sensitive tasks require topology-sensitive metrics. Anatomy-DT combines Dice and HD95 with clDice and an overlap regularizer; on the synthetic Voronoi benchmark, the full model reaches DSC $C=7$0 and HD95 $C=7$1, while also preserving topology (Bhattacharya et al., 29 Sep 2025). Skeletal latent diffusion evaluates medical shapes with CD, EMD, HD, F1, FID, KID, MMD, coverage, and 1-NNA, showing superior reconstruction and generation quality on MedSDF and vessel datasets (Zhang et al., 8 Mar 2026).

Downstream utility is another empirical criterion. Cardiac disentangled anatomy arithmetic improves balanced ACDC classification accuracy from 88.3% to 91.4% and segmentation Dice from 84.9 to 86.5 when its synthetic samples are added to training, with similar gains for underrepresented pathologies and vendor domains in M&Ms (Thermos et al., 2021).

6. Applications, boundaries, and limitations

Applications follow directly from the fact that the generated object is anatomically interpretable. Several papers explicitly target data augmentation, shape-prior learning, or image synthesis conditioned on generated masks (Borrego et al., 10 Feb 2026, Thermos et al., 2021). Others target virtual imaging trials and simulation. Thyroid shape synthesis is framed around virtual cohorts with controlled anatomy for imaging studies (Wilde et al., 4 Apr 2025), while virtual chimaeras are designed for in-silico evaluation of cardiovascular devices (Dou et al., 2022). AbdomenGen is aimed at controllable abdominal phantoms for scanner optimization, dosimetry, and cohort-matched simulation (Bhandari et al., 14 Apr 2026). A separate workflow based on physical prostate phantoms, automated ultrasound segmentation, 3D reconstruction, and a 3D GAN extends the idea to surgical training and post-resection anatomy generation (Müller et al., 24 Nov 2025).

A common boundary in the literature is between anatomy-to-anatomy generation and anatomy-guided image or language generation. MedSyn jointly models CT and anatomical masks as a 4-channel diffusion process over $C=7$2 and supports marginalization-based structure-to-image synthesis (xu et al., 2023). XReal uses anatomy and pathology masks to control chest X-ray diffusion without fine-tuning the backbone (Hashmi et al., 2024). Anatomy-guided radiology report generation restructures the visual representation into 29 anatomical regions with pathology-aware prompts for report synthesis (Gao et al., 2024). These works share the same anatomical control philosophy, but the primary output is no longer anatomy itself.

Limitations recur across paradigms. Slice-wise generators may lack explicit 3D consistency (Borrego et al., 10 Feb 2026). Some controls remain weak or partial, as with thyroid symmetry compared with volume and isthmus area (Wilde et al., 4 Apr 2025). Sequential generators can accumulate error across organs (Bhandari et al., 14 Apr 2026). Completion and assembly methods depend strongly on segmentation quality or on fixed correspondence and topology assumptions (Li et al., 2023, Dou et al., 2022). Dynamical digital twins remain deterministic and do not yet provide explicit uncertainty quantification (Bhattacharya et al., 29 Sep 2025). High-resolution brain generation has demonstrated morphology preservation, but explicit conditional control, privacy auditing, and faster sampling remain open issues (Tudosiu et al., 2022).

Taken together, the literature suggests that anatomy-to-anatomy generation is becoming a general design paradigm rather than a single model family. Its unifying principle is that anatomy is treated as the primary state variable: learned directly, edited directly, composed directly, or evolved directly. A plausible implication is that future systems will increasingly combine these elements—structured representations, interpretable control variables, topology-aware constraints, and downstream simulation—to move from anatomy synthesis toward anatomy programming.