Neural-Guided Video Synthesis Framework

• Neural-guided video synthesis is a framework that employs structured neural, geometric, and semantic signals to control video generation beyond frame-wise predictions.
• It integrates diverse guidance modalities such as 3D morphable models, skeletons, scene graphs, and optical flow to decouple control signals from appearance while ensuring temporal coherence.
• The approach leverages staged decomposition, latent forecasting, and spatial priors to enhance realism, consistency, and 3D or world-level control in synthesized videos.

Searching arXiv for recent and foundational papers on neural-guided video synthesis and closely related frameworks. Neural-guided video synthesis denotes a class of video generation methods in which the synthesis process is steered by explicit neural, geometric, semantic, or physically grounded signals rather than by frame-wise pixel prediction alone. In the cited literature, the guidance variable may be a 3D morphable-model expression code, facial action units and head pose, 2D skeletons, semantic video scene graphs, quantized latent tokens, optical flow, a persistent 3D world estimate, or even a dynamic brain encoding model that scores candidate videos by predicted regional activation. This family therefore spans conditional GANs, NeRF-based neural rendering, transformer-based latent forecasting, diffusion models, and neural implicit representations, with a recurring objective: to decouple control from appearance while preserving temporal coherence, realism, and, in several systems, 3D or world consistency (Athar et al., 2021, Wang et al., 2018, Moing et al., 2021).

1. Historical formation and scope

Early formulations in this area cast video synthesis as conditional translation from a structured source video to a photorealistic target video. “Video-to-Video Synthesis” models a source sequence such as semantic segmentation masks, sketches, or poses and learns a conditional mapping to a target-domain video, explicitly emphasizing that frame-wise image synthesis is insufficient because it does not enforce temporal coherence (Wang et al., 2018). This line of work established the idea that an overview framework should not merely render frames, but should exploit structure in the conditioning signal and the temporal organization of the sequence.

Subsequent systems broadened what counts as guidance. “Head2Head” uses 3D Morphable Model tracking, gaze estimation, and dense conditioning images for full-head reenactment, transferring pose, expression, gaze direction, and blinking dynamics while preserving target identity (Koujan et al., 2020). “Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation” separates dynamic texture prediction in UV space from screen-space rendering, thereby guiding synthesis through a body-centric representation rather than sparse pose alone (Liu et al., 2020). “Render In-between” makes motion guidance primary by first inferring high-frame-rate skeletal motion from low-frame-rate video, then using a pose-guided renderer to generate RGB frames (Ho et al., 2021).

A later development is the incorporation of 3D scene representations and canonical spaces. “FLAME-in-NeRF” conditions a radiance field on low-dimensional facial expression parameters from a 3DMM and adds a 3DMM-derived spatial prior to localize expression-dependent effects to the face region (Athar et al., 2021). “Towards 4D Human Video Stylization” reconstructs a human NeRF and a scene NeRF, stylizing rendered feature maps so that the output can support style transfer, novel view synthesis, and animation within one framework (Wang et al., 2023). “3D-Aware Video Generation” extends 3D-aware image GANs into a 4D setting by combining a NeRF-like generator with a time-aware discriminator, thereby supporting joint camera control and motion generation from monocular videos (Bahmani et al., 2022).

The phrase also covers frameworks whose guidance is not geometric in the classical sense. “CCVS: Context-aware Controllable Video Synthesis” uses a doubly autoregressive loop, quantized latent tokens, and a learnable optical-flow module so that future generation can be conditioned on sample frames, audio, trajectories, class labels, layouts, or start/end frames (Moing et al., 2021). “SSGVS: Semantic Scene Graph-to-Video Synthesis” uses semantic video scene graphs because labels and text do not explicitly specify when events occur (Cong et al., 2022). At the far end of the design spectrum, “NEvo” performs evolutionary search over a structured prompt space and uses a dynamic encoding model to guide synthesis toward videos that maximize predicted responses in a target visual-cortex ROI (Tang et al., 2 Jul 2026). This suggests that “neural-guided video synthesis” is best understood as a methodological family defined by structured guidance, not by a single architecture.

2. Guidance modalities and control variables

A central axis of variation is the form of the control signal. In many systems, the control is intentionally low-dimensional or semantically structured so that synthesis remains editable. In “FLAME-in-NeRF,” the radiance field is conditioned on an expression code $\mathbf{e}$ estimated from a 3DMM, yielding a function

$$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$

where $\mathbf{x}$ is a 3D point, $\mathbf{d}$ is viewing direction, and $\mathbf{e}$ is the facial expression representation. This explicitly makes expression a control variable rather than an entangled byproduct of the training video (Athar et al., 2021).

In talking-head reenactment, guidance is often hybrid. “Head2Head” rasterizes tracked 3D face geometry into Normalised Mean Face Coordinates and supplements them with a gaze image, so the renderer is conditioned on a dense, geometry-aware map plus eye motion (Koujan et al., 2020). “A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models” translates Word2Vec embeddings into a 20-D AU+PS representation, comprising 17-D action units and 3-D head pose, then concatenates AU+PS with average facial landmarks and the former $n$ synthesized frames before image generation (Wang et al., 2019). In this case, semantic text guidance is converted into motion guidance before rendering.

In human-body synthesis, the control signal is frequently skeletal. “Render In-between” represents each pose as $\bar{\mathbf p}_t \in \mathbb{R}^{2\times J}$ with $J=19$, denoises the low-FPS sequence, linearly interpolates it, and then predicts the nonlinear residual with a transformer. The resulting high-frame-rate pose sequence is the primary condition for the renderer (Ho et al., 2021). “Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation” instead uses a partial normal map in UV space, not a sparse skeleton, so the pose guidance already lives in the same domain as the predicted texture (Liu et al., 2020).

Other frameworks use guidance that is temporally or semantically richer than pose alone. “SSGVS” conditions generation on an initial image and a non-fixed number of semantic video scene graphs; its VSG encoder maps discrete graph annotations into continuous embeddings and predicts graph representations for frames without labels (Cong et al., 2022). “CCVS” inserts a quantizer between the encoder and the transformer so that multimodal ancillary information can be tokenized and concatenated, supporting control by sample frames, audio tracks, trajectories in image space, class labels, semantic layouts, and start/end frames (Moing et al., 2021). “NEvo” searches a structured prompt space $$\mathcal{P} = \mathcal{A}_1 \times \cdots \times \mathcal{A}_M$$, where each axis is an interpretable visual-attribute category such as subject, mood, lighting, event structure, camera motion, temporal rhythm, interaction type, or motion profile (Tang et al., 2 Jul 2026).

Framework	Guidance signal	Synthesis target
FLAME-in-NeRF	3DMM expression code and 3DMM spatial prior	Controllable portrait novel-view rendering
Head2Head	NMFC and gaze images from 3DMM tracking	Full-head reenactment
Virtual Anchor	Word2Vec $\rightarrow$ AU+PS, average FLM, previous $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$0 frames	Text-driven anchor face video
Render In-between	Denoised and interpolated 2D skeletons	Human action frame upsampling
SSGVS	Semantic video scene graphs and initial frame	Scene-graph-to-video synthesis
NEvo	Structured prompt genes scored by a dynamic encoding model	ROI-optimized dynamic stimuli

A common misconception is that guidance in video synthesis is synonymous with text conditioning. The literature does not support that view. Text is only one possibility; guidance can be geometric, kinematic, structural, world-based, neural-latent, or neurobiological, and several of the most controllable systems rely on representations such as 3DMM coefficients, skeletons, scene graphs, or guidance images rather than free-form language (Wang et al., 2019, Cong et al., 2022, Mallya et al., 2020).

3. Recurrent architectural patterns

A recurring design pattern is staged decomposition. Rather than mapping input guidance directly to pixels, many frameworks introduce an intermediate representation that isolates one difficult aspect of the problem. “A Neural Virtual Anchor Synthesizer” decomposes text-driven face generation into text $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$1 AU+PS with Seq2Seq, followed by AU+PS + average FLM + previous frames $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$2 image synthesis with Pix2PixHD (Wang et al., 2019). “Render In-between” splits the task into motion inference and neural rendering, training the motion model on AMASS and the renderer on low-FPS target videos (Ho et al., 2021). “Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation” separates UV-space dynamic-texture prediction from screen-space refinement, arguing that high-frequency detail should be learned in a body-centric domain (Liu et al., 2020).

Autoregression appears in several distinct forms. “Video-to-Video Synthesis” factorizes the conditional video distribution with a Markov assumption and recursively synthesizes each frame from the current source frame, past $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$3 source frames, and past $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$4 generated frames, using $$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$5 in experiments (Wang et al., 2018). “Head2Head” generates

$$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$6

so the next output depends on the current and two previous conditioning frames plus the two previous synthesized frames (Koujan et al., 2020). “CCVS” is doubly autoregressive: the transformer predicts future latent tokens, and newly synthesized images are fed back into the image-space context window (Moing et al., 2021). This suggests that guidance is frequently paired with explicit temporal recursion, because control alone does not guarantee sequence stability.

Tokenization and latent forecasting constitute another major pattern. “CCVS” quantizes encoder latents with a learned codebook and uses an autoregressive transformer in token space, enabling multimodal conditioning and top-$$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$7 sampling for multiple futures (Moing et al., 2021). “SSGVS” uses a frozen VQ-VAE to produce discrete visual latents, into which graph embeddings are inserted before a GPT-like transformer models the joint prior (Cong et al., 2022). “NeRV-Diffusion” moves even further from frame-wise tokenization: a hypernetwork-based tokenizer maps an entire raw video to latent INR weights, and an implicit diffusion transformer denoises directly in that weight space rather than in frame-wise feature space (Ren et al., 29 Sep 2025).

The choice of decoder also varies with the representation. GAN-based image translators remain common when the control is already spatially aligned, as in Pix2PixHD variants, vid2vid-style sequential generators, and SPADE-like renderers (Wang et al., 2019, Wang et al., 2018, Ho et al., 2021). NeRF-based decoders dominate when 3D consistency, canonical pose spaces, or viewpoint control are required (Athar et al., 2021, Wang et al., 2023, Bahmani et al., 2022). Diffusion enters where sample quality or generative flexibility is central, but the biomedical cell-video study shows that adding dense optical flow to a diffusion denoiser is not automatically beneficial; the modified model became noisy in practice (Serna-Aguilera et al., 2022). The architectural lesson is not that one generator family is universally superior, but that the decoder must match the structure of the chosen guidance.

4. Spatial priors, 3D representations, and world models

One of the strongest trends in neural-guided video synthesis is the use of spatial priors to localize where guidance should act. In “FLAME-in-NeRF,” the 3DMM provides not only a low-dimensional expression parameterization but also a spatial prior that restricts expression-dependent effects to the face region. The paper motivates this as protection against spurious correlations between expression and viewpoint in handheld portrait videos. The radiance field uses standard volume rendering,

$$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$8

but the expression conditioning is effectively spatially gated by the 3DMM-derived prior (Athar et al., 2021). This is a paradigmatic example of neural guidance serving both as semantic control and as a disentangling constraint.

Canonicalization is another central device. “Towards 4D Human Video Stylization” maps observed human points into a canonical space with SMPL-guided inverse skinning,

$$F_\theta : (\mathbf{x}, \mathbf{d}, \mathbf{e}) \mapsto (\sigma, \mathbf{c}),$$9

and then adds an error-correction network $\mathbf{x}$0 to compensate for imperfect warping (Wang et al., 2023). The human and scene are represented by two separate NeRFs, and stylization is applied to rendered feature maps rather than RGB frames. This organization permits stylized original views, stylized novel camera views, and stylized human animation in novel poses within one framework (Wang et al., 2023).

Other works externalize the same principle beyond NeRF. “World-Consistent Video-to-Video Synthesis” introduces the guidance image: a sparse estimate of the current frame produced by back-projecting already rendered imagery into a reconstructed 3D world and projecting those colored 3D points into the current view (Mallya et al., 2020). The generator then conditions on semantic labels, the previous frame, flow information, and the guidance image through Multi-SPADE modulation. The goal is not merely short-term temporal smoothness but consistency across the entire rendered 3D world, especially when the camera revisits a previously observed location (Mallya et al., 2020).

Unconditional generation can also be 3D-aware. “3D-Aware Video Generation” combines a NeRF-like implicit generator with separate image and video discriminators, using distinct latent variables for content and motion, each of dimension 512, and a time-conditioned motion generator to learn a 4D scene distribution from monocular videos only (Bahmani et al., 2022). It supports yaw rotation, pitch rotation, zoom-out, and depth-map extraction from the expected ray termination distance (Bahmani et al., 2022). A plausible implication is that explicit world or scene structure is increasingly treated not as an optional add-on but as a direct carrier of controllability.

5. Temporal coherence and objective design

Temporal consistency is the dominant optimization concern across the literature. “Video-to-Video Synthesis” addresses it by decomposing each new frame into a warped previous frame and hallucinated content: $\mathbf{x}$1 where $\mathbf{x}$2 is an occlusion mask, $\mathbf{x}$3 is predicted optical flow, and $\mathbf{x}$4 is the hallucinated image (Wang et al., 2018). The model further combines conditional image and video discriminators, multi-scale temporal discrimination, flow supervision, and VGG-based feature matching (Wang et al., 2018). Removing flow warping hurts most in the reported ablation, which indicates that explicit reuse of earlier content remains a strong baseline for temporal stabilization.

Several frameworks specialize the temporal loss to the domain. “Head2Head” adds a mouth discriminator for cropped mouth regions and a multi-scale dynamics discriminator that evaluates clips of $\mathbf{x}$5 consecutive frames together with optical flow, using a FlowNet2 model fine-tuned on 4DFAB for facial motion (Koujan et al., 2020). “Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation” follows vid2vid-style training with a per-frame discriminator loss, a video discriminator loss over local windows of three frames, and an optical-flow-related term $\mathbf{x}$6 for both TexNet and RefNet (Liu et al., 2020). “CCVS” uses a learnable flow module to warp context features into the decoder at multiple resolutions and pairs this with 2D appearance and 3D temporal adversarial training (Moing et al., 2021).

In motion-guided human interpolation, temporal consistency is achieved by combining explicit motion reasoning with compositing. “Render In-between” generates a foreground image $\mathbf{x}$7, a mask $\mathbf{x}$8, and uses a separately interpolated background $\mathbf{x}$9, composing them as

$\mathbf{d}$0

The full image loss contains reconstruction, VGG perceptual terms, foreground-specific terms, mask regularization, and an adversarial term (Ho et al., 2021). “World-Consistent Video-to-Video Synthesis” adds a dedicated world-consistency loss

$\mathbf{d}$1

where $\mathbf{d}$2 is the guidance image, thereby extending temporal consistency into a longer-horizon spatial memory constraint (Mallya et al., 2020).

Diffusion-based systems revise the same problem in denoising form. “Neural Cell Video Synthesis via Optical-Flow Diffusion” conditions the denoiser on grayscale video plus horizontal and vertical Horn–Schunck flow fields, training with an $\mathbf{d}$3 reconstruction term. The intention is explicit motion-aware denoising, although the resulting model did not improve realism in the reported experiments (Serna-Aguilera et al., 2022). “NeRV-Diffusion” instead denoises in INR-weight space with a standard noise-prediction loss and adds Min-SNR-$\mathbf{d}$4 loss weighting and scheduled sampling for stabilization (Ren et al., 29 Sep 2025). Across these variants, the common pattern is clear: guidance affects not only the forward architecture but also the loss geometry by constraining where, when, and how information should persist.

6. Applications, empirical behavior, and limitations

The application space is unusually broad. Portrait and head synthesis are represented by “FLAME-in-NeRF,” “Head2Head,” and the virtual-anchor framework. The first can render portrait videos from arbitrary viewpoints while permitting explicit facial-expression editing from a single short mobile-captured video and preserving details such as hair, beard, teeth, and glasses (Athar et al., 2021). “Head2Head” reports an average per-pixel error of 9.3 in cycle-reenactment on a 1K-frame test sequence, and an automated detector from FaceForensics++ achieved only 1.88% accuracy on its generated fake videos (Koujan et al., 2020). The virtual-anchor framework is primarily qualitative: it states that synthesized images are realistic and mouth movements are “almost consistent” with corresponding words, but it does not report PSNR, SSIM, FID, LPIPS, or AV-sync metrics (Wang et al., 2019).

Human-body synthesis and interpolation are evaluated more quantitatively. “Render In-between” introduces HumanSloMo, consisting of 80 sequences recorded at 30–240 FPS with 10 subjects, and reports for its main method PSNR 30.75, SSIM 0.9648, LPIPS 0.0395, and FVD 123.04. The stronger variant Ours$\mathbf{d}$5 improves to PSNR 31.00, SSIM 0.9658, LPIPS 0.0409, and FVD 121.86, while Ours w. GT pose reaches PSNR 31.37, SSIM 0.9665, LPIPS 0.0384, and FVD 109.73 (Ho et al., 2021). “Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation” reports held-out foreground-only results in which the full pipeline achieves L2 8.675 and SSIM 0.784, outperforming its ablations, and user-preference shares of 92.59% vs. Esser, 98.15% vs. Ma, 80.61% vs. Liu, 68.22% vs. Chan, and 62.96% vs. Wang overall (Liu et al., 2020).

Controllable general video synthesis shows similar diversity. “CCVS” reports FVD $\mathbf{d}$6 on BAIR and FVD $\mathbf{d}$7 on Kinetics future prediction at $\mathbf{d}$8, while also supporting audio-conditioned synthesis, layout-conditioned synthesis, point-to-point synthesis, and unconditional generation (Moing et al., 2021). “SSGVS” evaluates on Action Genome and improves over MoCoGAN, LVT, VideoGPT, and CCVS, reporting FVD $\mathbf{d}$9 and SSIM 0.565; without VSG pre-training, FVD worsens to 457.1 and SSIM drops to 0.509 (Cong et al., 2022). “World-Consistent Video-to-Video Synthesis” improves Cityscapes FID from 69.07 for vid2vid to 49.89, and reports strong forward-backward consistency gains, for example RGB 8.73 vs 14.90 on Cityscapes and RGB 11.85 vs 46.30 on ScanNet (Mallya et al., 2020). “Video-to-Video Synthesis” itself reports 2K-resolution videos up to 30 seconds long and future-prediction FID 3.44 / 0.18 on Cityscapes, compared with 11.18 / 0.59 for PredNet and 10.00 / 0.43 for MCNet (Wang et al., 2018).

More recent work extends the framework idea into specialized scientific settings. “Towards 4D Human Video Stylization” reports temporal-consistency scores of 0.165 on the authors’ dataset and 0.214 on NeuMan, compared with 0.293 and 0.387 for StyleRF respectively, and reports that the geometry-guided tri-plane improves temporal consistency from 0.207 to 0.182 over vanilla tri-plane (Wang et al., 2023). “3D-Aware Video Generation” reports a chamfer distance of 1.03 on FaceForensics versus 1.27 for StyleNeRF, and finds that multiplicative time conditioning gives FVD 158.3 on TaiChi compared with 175.4 for positional time encoding and 164.7 for concatenation (Bahmani et al., 2022). “NeRV-Diffusion” reports UCF-101 gFVD values of 184, 133, and 97 for its Small, Base, and Large models, and Kinetics-600 gFVD values of 46, 30, and 22, respectively, while emphasizing smooth interpolation in INR-weight space (Ren et al., 29 Sep 2025).

The limitations are correspondingly heterogeneous. Target-specific systems such as “Head2Head” require a training video of the target subject and about 5 hours on an NVIDIA Titan V (Koujan et al., 2020). UV- or pose-guided human synthesis remains person-specific and degrades when test poses are far outside the training motion range; fingers are a known failure mode in the dynamic-texture framework (Liu et al., 2020). “World-Consistent Video-to-Video Synthesis” depends on reasonably valid structure-from-motion and is therefore best suited to mostly static scenes (Mallya et al., 2020). “3D-Aware Video Generation” notes high memory cost, about 2.4 GB / batch size compared to 0.8 GB / batch size for StyleGAN-V, and difficulty with multi-object scenes (Bahmani et al., 2022). “Neural Cell Video Synthesis via Optical-Flow Diffusion” is notable because the proposed guidance did not improve quality: most outputs were noisy and cells were not discernable, underscoring that adding a motion signal does not by itself guarantee a better guidance mechanism (Serna-Aguilera et al., 2022). “NEvo” identifies a different boundary condition: the framework is only as good as its encoding model and requires eventual closed-loop neuroimaging validation, even though it achieved the top 99.8% of MiT responses and 95.8% of localizer responses on average (Tang et al., 2 Jul 2026).

Taken together, these results indicate that neural-guided video synthesis is not a single solved problem but a set of design principles. The most persistent principles are structured control spaces, intermediate representations aligned with the control variable, explicit temporal objectives, and, increasingly, 3D or world-level memory. The literature also shows that guidance can be introduced at many levels—conditioning variable, latent tokenizer, renderer, discriminator, search objective, or world prior—and that the effectiveness of any such guidance is task-dependent rather than automatic (Athar et al., 2021, Moing et al., 2021, Tang et al., 2 Jul 2026).