Controllable Synthesis

• Controllable synthesis is the design of generative processes that are steered by explicit conditions rather than implicit data sampling.
• It employs decoupled architectures and explicit control representations to manage variables like pose, prosody, and style in diverse applications.
• Advances in evaluation metrics and supervision strategies enhance fidelity while ensuring that edits remain intuitively aligned with user intent.

Controllable synthesis is the design of generative or constructive processes in which outputs are steered by explicit conditions rather than sampled only from an implicit data distribution. In current research, the concept spans multimodal speech synthesis, talking-head generation, person and semantic image synthesis, singing voice synthesis, human motion and character video generation, 3D face synthesis, medical image generation, and even water-assisted materials growth. Across these settings, the controlled variables include speaker-related style, prosody, dynamics, pose, gaze, emotion, semantic layout, lesion morphology, 3D shape, scene context, and reagent delivery; a recurring problem is to expose these variables in a form that is editable, semantically meaningful, and compatible with high-fidelity synthesis (Niu et al., 26 Jun 2025, Li et al., 2024, Ryu et al., 8 Sep 2025, Zhao et al., 2019).

1. Problem formulations and scope

In visual synthesis, controllability is often framed as a constrained mapping from a structural description to an image or video. Pose-transfer and person synthesis methods take a source appearance and a target pose, then require the output to preserve identity- or garment-level appearance while conforming to the new configuration. Semantic image synthesis makes the constraint even more explicit: a semantic layout determines what should appear where, while appearance is separately controlled through semantic-part style tokens or user-specified colors. In these formulations, controllability is fundamentally spatial; it binds generation to target parsing maps, pose heatmaps, or semantic masks rather than leaving structure to a latent prior (Zhou et al., 2022, Men et al., 2020, Kim et al., 2023).

In speech and singing, the problem is usually posed as conditional generation of acoustics from text or score while exposing hidden expressive variables. Hierarchical TTS models treat speaking style, accent, background noise, and recording condition as latent attributes that must be inferred and then manipulated. Prosody control operates at frame level through $F_0$ trajectories, while recent singing work targets dynamics control through explicit energy sequences. Multimodal controllable speech synthesis extends this further by allowing face images, text prompts, and speech embeddings to act as interchangeable control modalities for speaker-related style (Hsu et al., 2018, Kumar et al., 2021, Morrison et al., 2020, Niu et al., 26 Jun 2025, Ryu et al., 8 Sep 2025).

In 3D-aware generation, motion synthesis, and physical growth, controllability is defined over geometric or process variables. NeRF-based face models use 3D Morphable Model conditions to control shape, expression, and pose; motion models condition future human motion on past motion and external controls such as locomotion velocities; character video systems separate identity, motion, and scene into distinct codes; and water-assisted chemical vapor transport treats water-vapor delivery as a tunable control variable governing precursor transport, supersaturation, and heterostructure formation (Sun et al., 2022, Men et al., 2024, Yin et al., 2023, Zhao et al., 2019).

2. Control representations

A central design choice in controllable synthesis is the representation of the control variable itself. Recent work consistently moves away from opaque global latents toward explicit, semantically structured intermediate spaces. In talking-head synthesis, Ditto defines an identity-agnostic motion representation

$$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$

where $\boldsymbol{\delta}$ is expression deformation, $\mathbf{R}$ is head rotation, and $\mathbf{t}$ is translation. The deformation term is further analyzed as a 63-dimensional vector corresponding to the $x,y,z$ coordinates of 21 implicit 3D keypoints, which makes regional and magnitude-constrained editing possible (Li et al., 2024).

In singing voice synthesis, dynamics are represented either per frame or per phoneme. The phoneme-level energy sequence compresses framewise loudness into one scalar per phoneme: $$e_i = \frac{1}{T_i} \sum_{t=t_{\mathrm{start}_i}}^{t_{\mathrm{end}_i}} \mathbf E[t].$$ This representation trades exact temporal resolution for a practical control interface aligned with the symbolic score (Ryu et al., 8 Sep 2025).

Speech synthesis work based on hierarchical VAEs uses latent variables that are structured rather than monolithic. A categorical latent $z_y$ indexes attribute groups, while a Gaussian latent $z_l$ models fine-grained variation within a group, yielding a Gaussian-mixture prior over speech attributes. In the supervised extension, an additional continuous latent $z_o$ is tied to observed labels such as speaker identity. This explicitly separates coarse mode selection from continuous attribute control (Hsu et al., 2018).

In image synthesis, the dominant representations are spatial. Person synthesis uses target pose heatmaps, parsing maps, and per-semantic-part style tokens; semantic artwork and semantic image synthesis similarly bind control to segmentation maps. RUCGAN compresses style to one representative color per semantic region, while CASD models appearance as a set of semantic style tokens routed to target locations through attention (Zhou et al., 2022, Kim et al., 2023, Men et al., 2020).

Medical synthesis adopts clinically structured controls. LF-VAR conditions generation on lesion-region masks, lesion type labels, and quantified lesion measurements extracted from the lesion region with PyRadiomics. In 3D face synthesis, the condition is geometric: 3DMM coefficients define a mesh

$$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$0

so shape and expression are injected as explicit surfaces rather than as image-space attributes (Sun et al., 27 Aug 2025, Sun et al., 2022).

3. Architectural patterns for enforcing control

A recurring architectural pattern is decoupling. Instead of asking a single end-to-end generator to learn both control alignment and synthesis from scarce matched data, recent systems isolate alignment, representation construction, and rendering into separate stages. In multimodal controllable speech synthesis, this appears as a three-stage pipeline: face-speech alignment, text prompt alignment, and speech synthesis model training. The face encoder is aligned to a frozen ECAPA-TDNN speech encoder using weight-shared AM-Softmax classification, knowledge distillation, and contrastive learning, while the speech synthesizer is trained separately on high-quality speech data (Niu et al., 26 Jun 2025).

Ditto applies the same principle to audiovisual generation by moving diffusion into a compact motion space and separating motion generation from identity-specific rendering. Audio, emotion labels, eye state, canonical keypoints, and initial motion are combined through a structured conditioning interface in which eye state, geometry, and emotion act as Enhanced Conditional Signals and reference initial motion acts as an Initial Conditional Signal. The renderer then converts generated motion into frames for a fixed identity, making both realtime inference and semantic motion editing feasible (Li et al., 2024).

MIMO decomposes video along a different axis: it first separates each clip into human, underlying scene, and floating occlusion using depth-aware masks, then factorizes the human into a canonical identity code and a structured motion code based on SMPL-anchored latent features. Scene and occlusion are encoded separately and fused into a full scene code. These codes enter a Stable Diffusion 1.5 plus AnimateDiff backbone through different pathways, so identity, motion, and scene remain distinct control channels rather than collapsing into a single conditioning tensor (Men et al., 2024).

In 3D face synthesis, control is enforced directly on the volumetric representation. CGOF++ uses a mesh-guided volume sampler to concentrate NeRF samples near a conditioning 3DMM surface and a distance-aware density regularizer to suppress density away from that surface. A 3D landmark loss and a volume warping loss then refine local semantic structure and cross-expression consistency. The effect is to move from image-space attribute guidance to explicit geometric control of the underlying radiance field (Sun et al., 2022).

For sequential motion, MoDiff combines a cross-modal Transformer encoder with an autoregressive diffusion decoder. Motion context and control context are encoded separately, fused through cross-modal attention, and then used at every denoising step. The additional diffusion data dropout corrupts motion context via the diffusion forward process, forcing the model to rely more strongly on external controls and improving robustness under imperfect inputs (Yin et al., 2023).

4. Supervision, alignment, and data construction

Controllable synthesis rarely has access to fully annotated or fully matched supervision, so many systems redesign the training problem to use weaker forms of data. The multimodal speech framework explicitly avoids fully matched speech-face-text triplets by learning from multiple kinds of paired data instead. This reformulation treats triplet scarcity as the central bottleneck and replaces it with cross-modal embedding alignment plus separate speech synthesis training (Niu et al., 26 Jun 2025).

Dataset construction can itself become part of the control pipeline. ArtSem addresses the absence of semantic artwork datasets by generating semantic maps from landscape photography, performing unpaired photo-to-art translation, and then filtering the resulting semantic-art pairs with a learned quality assessor. The final CMSAS model is therefore trained on synthetic but paired semantic-art data, which makes semantic controllability possible in a domain where direct annotation is largely unavailable (Huang et al., 2023).

Several speech models create controllability by simulating user intervention during training. Controllable neural prosody synthesis provides ground-truth $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$1 as an input feature on random subsequences between 10 and 1000 milliseconds, so the model learns conditional completion rather than only unconditional prediction. RTI-VAE improves latent robustness by minimizing mutual information between latent variables and reordering Transformer layers so that the encoder learns a generic speech representation before style-specific deviations, reducing cluster overlap of speaker attributes (Morrison et al., 2020, Kumar et al., 2021).

Medical and motion synthesis also rely on supervision that is structured but not manually authored. LF-VAR extracts lesion measurements from image-mask pairs and injects them through a lightweight embedding network, while a lesion-focused VQ-VAE loss preserves non-lesion structure across scales. MoDiff’s diffusion data dropout plays a related role: it creates corrupted motion contexts during training so the model can later reconstruct or synthesize under incomplete observations without retraining (Sun et al., 27 Aug 2025, Yin et al., 2023).

5. Evaluation and empirical regimes

Because controllability is not identical to fidelity, evaluation in this area typically separates realism metrics from control-adherence metrics. Different subfields have therefore developed different metric families.

Domain	Control variables	Representative evaluation
Talking heads and video	audio, emotion, eye state, pose, scene	FID, FVD, CSIM, Sync-C, Sync-D, RTF, FFD; LPIPS, SSIM, PSNR, ADE, recall (Li et al., 2024, Chen et al., 2023)
Speech and singing	latent style/noise, $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$2, energy	MOS, SNR, MCD, FFE, Energy MAE, F0 MAE (Hsu et al., 2018, Morrison et al., 2020, Ryu et al., 8 Sep 2025)
Image and 3D face synthesis	pose, semantic style, 3DMM shape/expression/pose	SSIM, FID, LPIPS, PSNR; DS, LD, LC, APD (Zhou et al., 2022, Sun et al., 2022)
Medical imaging and materials growth	lesion type, mask, measurements; water vapor	FID, IS, cross-dataset FID; Raman and photoluminescence mappings (Sun et al., 27 Aug 2025, Zhao et al., 2019)

Several results illustrate how recent systems turn controllability into a first-class empirical target. Ditto reports head-only offline inference at $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$3 with RTF $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$4, online streaming at RTF $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$5, and first-frame delay 385 ms, while maintaining strong synchronization metrics even with only 10 denoising steps (Li et al., 2024). In controllable singing, phoneme-level energy conditioning reduces Energy MAE from $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$6 to $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$7 and yields MOS $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$8, while frame-level energy reaches Energy MAE $$\mathbf{m}=\{\boldsymbol{\delta}, \mathbf{R}, \mathbf{t}\},$$9 (Ryu et al., 8 Sep 2025). LF-VAR achieves average FID $\boldsymbol{\delta}$0 across seven lesion types, improving on vanilla VAR’s $\boldsymbol{\delta}$1, and preserves its advantage under cross-dataset evaluation on ISIC2017 and Dermofit (Sun et al., 27 Aug 2025). CGOF++ reports $\boldsymbol{\delta}$2, $\boldsymbol{\delta}$3, $\boldsymbol{\delta}$4, $\boldsymbol{\delta}$5, $\boldsymbol{\delta}$6, and $\boldsymbol{\delta}$7, showing that geometric control accuracy is measured explicitly rather than inferred from image realism alone (Sun et al., 2022).

A consistent empirical pattern is that explicit controls outperform implicit surrogates when the task requires faithful adherence. The singing study finds that an energy predictor only reduces Energy MAE from $\boldsymbol{\delta}$8 to $\boldsymbol{\delta}$9, whereas direct phoneme-level energy input reduces it to $\mathbf{R}$0 (Ryu et al., 8 Sep 2025). MCDiff similarly shows that sparse strokes are too ambiguous as a direct diffusion condition; dense motion completion substantially improves FVD, LPIPS, SSIM, and PSNR relative to conditioning the diffusion model on sparse flows alone (Chen et al., 2023).

6. Limitations and current research directions

The main limitations are now less about raw fidelity than about the structure and reliability of the control interface. Ditto depends on the quality of an off-the-shelf motion extractor; its deformation semantics are discovered empirically from implicit keypoints rather than from a fully explicit FACS or landmark parameterization; emotion control is clip-level rather than frame-level; and disentanglement is improved but not perfect (Li et al., 2024). LF-VAR leaves key reproducibility details underdescribed, including the source of lesion masks and the normalization of measurement features; its “language prompt” capability is closer to lesion-category prompting than to rich free-form language control; and it does not establish disentangled control over individual lesion attributes (Sun et al., 27 Aug 2025).

Compression of the control space can also become a bottleneck. RUCGAN shows that one representative color per semantic region enables reference-free controllability, but this representation is too compressed for complex scenes such as Cityscapes, where performance drops relative to stronger baselines (Kim et al., 2023). CGOF++ achieves strong geometric control, yet still lacks precise texture and illumination control, relies on an off-the-shelf 3DMM predictor, and struggles with some non-neutral expressions such as closed eyes or frowning (Sun et al., 2022). MCDiff remains bounded by learned motion semantics and by the quality of tracking-derived motion annotations, so out-of-distribution edits such as moving a scoreboard graphic into the middle of a scene or collapsing shelves via background strokes remain difficult (Chen et al., 2023).

Taken together, these results suggest that controllability improves when the control variable is explicit, semantically structured, and injected throughout the synthesis pipeline rather than inferred only as a hidden latent. They also suggest that fidelity alone is an inadequate endpoint: the field increasingly evaluates whether a model obeys a specified motion, energy contour, semantic region, 3D shape, or chemical transport condition. In that sense, controllable synthesis is evolving from a question of conditional generation to a question of representation design—how to choose an intermediate space in which human intent, physical constraints, and learned generative priors can be made commensurate.