---
title: Generative Face Video Coding (GFVC)
url: https://www.emergentmind.com/topics/generative-face-video-coding-gfvc
type: topic
---

# Generative Face Video Coding (GFVC)

Searching arXiv for recent GFVC papers to ground the article in current literature.
Generative Face Video Coding (GFVC) is a model-based compression paradigm for talking-head sequences in which inter frames are represented by compact facial priors or latent codes and reconstructed by a deep generative decoder conditioned on one or more conventionally coded key-reference frames. In the unified formulation reported in recent reviews, a key frame is coded by a standard codec such as VVC, an analysis module extracts a low-dimensional representation for each non-key frame, and a generator synthesizes the reconstructed frame from the decoded key frame and decoded representation [2506.07369][2311.02649]. Across the literature, GFVC is characterized by semantic-aware representation, perceptual optimization, and operation in ultra-low-bitrate regimes; several works also extend the paradigm toward layered scalability, multi-reference prediction, joint audio-video coding, standardization through Supplemental Enhancement Information (SEI), and low-complexity deployment [2410.15105][2502.17085].

## 1. Paradigm and formal definition

A generalized GFVC system partitions a face-only sequence into key-reference frames and generative inter frames. For key frames, a conventional codec produces reconstructed references. For each inter frame, the encoder extracts a compact facial prior \(s_t\) or \(\theta_1^t\), entropy-codes it, and the decoder synthesizes the frame with a generator conditioned on the decoded key frame and decoded prior [2311.02649][2506.07369]. The formal decomposition reported in the surveys is
\[
\hat X_0=\mathrm{Dec}(\mathrm{Enc}(X_0)),
\]
\[
\theta_1^t=\mathcal{E}(X_1^t), \quad R\approx \text{bits}(\theta_1^t),
\]
\[
\hat X_1^t=\zeta\!\Bigl(\hat X_0,\;\varpi\!\bigl(\hat\theta_1^t,\mathcal{E}(\hat X_0)\bigr)\Bigr)\equiv G(\hat\theta_1^t;\hat X_0).
\]
The associated training criterion is a Lagrangian rate–distortion objective,
\[
\mathcal{L}_{\rm tot}=\sum_t D(X_1^t,\hat X_1^t)+\lambda R(\theta_1^t),
\]
or, in the alternate review formulation,
\[
\min_{E_\phi,G_\theta} R+\lambda\cdot D,
\]
with an optional perception term \(P\) in \(R+\lambda D+\beta P\) [2506.07369][2311.02649].

This abstraction generalizes early Model-Based Coding by replacing analytic synthesis with deep generative networks. The encoder no longer transmits dense block motion fields and residual transforms as in hybrid coding; instead it transmits sparse or compact face representations such as 2D landmarks, 3D landmarks, semantic maps, temporal-evolution features, or latent codes, and relies on a learned synthesizer for dense motion recovery and image generation [2410.08485][2410.15105]. A plausible implication is that GFVC shifts the principal coding burden from pixel-domain residual transmission to representation design and decoder prior quality.

## 2. Representation families and decoder architectures

The surveys organize GFVC methods primarily by representation type. One major class uses **2D keypoints**. In this setting, the encoder transmits a small set of facial keypoints, optionally with local affine motions or Jacobians, and the decoder predicts dense flow and occlusion before synthesizing the frame through a U-Net- or SPADE-style generator [2506.07369][2311.02649]. In "Neural Face Video Compression using Multiple Views" [2203.15401], the single-view baseline follows the First Order Motion Model (FOMM): a keypoint detector \(D\) outputs sparse 2D keypoints, a dense motion predictor \(M\) refines a coarse flow into a dense field \(F(x)\) and predicts an occlusion map \(O(x)\), an encoder \(E\) extracts source-view features, warp-and-occlusion \(W\) bilinearly samples the feature map, and a ResNet-based generator \(G\) produces the reconstruction. The coarse flow is
\[
f_{\rm coarse}(x)=\sum_{k=1}^N \alpha_k(x)\,\bigl(T_k(x-p_k^s)+p_k^t-x\bigr),
\]
with refinement
\[
F(x)=f_{\rm coarse}(x)+\Delta f(x),
\]
and warped features
\[
\tilde z(x)=O(x)\sum_{y\in\Omega} z(y)\,k(y,x+F(x)).
\]
In the multi-view extension, \(K\) source views are stored, each view is warped independently, and the warped features are fused by a permutation-invariant aggregation module \(A(\cdot)\) before decoding [2203.15401].

A second class uses **3D semantic or morphable-model representations**. "Interactive Face Video Coding: A Generative Compression Framework" [2302.09919] represents each inter frame by a compact set of 3D-face semantic parameters after an Internal Dimension Increase (IDI) stage. The regressed latent \(\delta_{reg}\) contains appearance and motion terms, while the transmitted compact semantics are \(\delta_{com}=\{\delta_{mouth},\delta_{eye},\delta_{rot},\delta_{trans},\delta_{loc}\}\in\mathbb{R}^{14}\). A 3D mesh reconstruction module produces a dense 2D facial mesh and eye-blink map, a mesh-based motion estimation network generates coarse and fine flow, and a CSSFT-GAN generator synthesizes the output frame [2302.09919]. This line of work emphasizes editable semantics and direct control over mouth motion, eye blinking, head rotation, and head translation.

A third class uses **learned compact features or latent spaces**. The reviews list CFTE as a representative compact latent-matrix method and note that learned feature vectors or temporal-evolution features can replace explicit landmarks [2506.07369][2311.02649]. "Video Coding Using Learned Latent GAN Compression" [2207.04324] represents frames by latent codes in the StyleGAN2 \(W^+\) space and learns an invertible proxy space \(W^*\) through a RealNVP normalizing flow,
\[
z=T(w), \qquad w=T^{-1}(z).
\]
The encoder inverts each frame into \(w\in W^+\), maps it to \(z\in W^*\), quantizes and entropy-encodes \(z\), and reconstructs by \(G(T^{-1}(\hat z))\) using fixed StyleGAN2 encoder and generator components [2207.04324]. This establishes a distinct GFVC branch in which synthesis quality derives from a pretrained GAN prior rather than explicit motion warping alone.

Across these representation families, decoder design recurrently combines dense motion estimation, warping, occlusion handling, and a conditional generator. Depending on the method, realism is enforced by perceptual losses, adversarial losses, feature matching, equivariance constraints, or latent-space reconstruction objectives [2203.15401][2207.04324][2302.09919].

## 3. Coding objectives, losses, and rate–distortion behavior

GFVC training is consistently expressed as rate–distortion optimization, but the distortion term is often perceptual rather than pixel-fidelity oriented. Reviews summarize this as
\[
\mathcal{L}_{RD}=D(X,\hat X)+\lambda R(\theta),
\]
while standardization-oriented descriptions write
\[
\mathcal{L}=R+\lambda D
\]
with \(R\) including both base-picture bits and generative side information [2410.15105][2311.02649]. Typical distortion terms include \(\ell_1\) or \(\ell_2\) reconstruction, VGG-based perceptual loss, adversarial loss, and task-specific regularizers such as equivariance or identity preservation [2410.15105][2311.02649].

In the multi-view FOMM extension, the paper uses a multi-scale perceptual loss on VGG features and optionally a pixel-wise \(\ell_1\) or \(\ell_2\) term; no adversarial loss is used in that work [2203.15401]. The perceptual term is
\[
\mathcal L_{\rm perc}=\sum_{\ell}\|\phi_\ell(\hat f)-\phi_\ell(f)\|_1.
\]
Its per-frame rate is explicitly derived from keypoint transmission: with \(N=10\) 2D keypoints encoded as 16-bit floats, the inter-frame cost is \(320\) bits/frame [2203.15401].

By contrast, SGANC defines both rate and distortion in latent space. For intra coding,
\[
R=-E_{w,\epsilon}\sum_{i=1}^D \log_2 p_i(z_i+\epsilon_i),
\]
with \(D=18\cdot512\), and the distortion term is mean-squared error in \(W^+\),
\[
D=E_w[\|w-T^{-1}(z+\epsilon)\|_2^2].
\]
For inter coding, successive latent differences are quantized, optional residuals are inserted every \(g\) frames, and the loss is
\[
\mathcal L^{IC}=R+\lambda E[\|w_t-\hat w_t\|^2].
\]
The paper justifies latent-space distortion by arguing that \(W^+\) encodes multi-scale, semantically disentangled face attributes and reports that latent-space MSE yields a better rate–distortion trade-off than image-space MSE or LPIPS within that framework [2207.04324].

The empirical rate–distortion literature consistently notes that GFVC targets perceptual quality more than classical PSNR optimality. The 2025 survey reports average perceptual BD-rate savings at \(256\times256\) of \(-44.7\%\) for FOMM, \(-55.6\%\) for CFTE, \(-52.3\%\) for FV2V, \(-67.0\%\) for DAC, and \(-15.4\%\) for TPS relative to VVC in Rate-DISTS; corresponding Rate-LPIPS savings are \(-41.1\%\), \(-49.4\%\), \(-46.5\%\), \(-65.7\%\), and \(-11.4\%\) [2506.07369]. The same survey also states that GFVC lags in PSNR/SSIM and that this reflects the semantic-perceptual target rather than a failure of the paradigm [2506.07369]. This addresses a recurrent misconception: lower pixel-domain fidelity does not preclude superior perceptual quality under GFVC’s design objective.

## 4. Multi-reference, layered, and bandwidth-intelligent extensions

Several works extend baseline GFVC to address instability, narrow bitrate range, or limited bandwidth adaptation. One line of development introduces **multiple references**. The multi-view extension of FOMM stores \(K\) source views, computes a dense flow and warped feature map for each, and fuses them with either pooling or cross-view self-attention [2203.15401]. The self-attention formulation computes
\[
Q_{i,j}=Z_{i,j}W_q,\quad K_{i,j}=Z_{i,j}W_k,\quad V_{i,j}=Z_{i,j}W_v,
\]
followed by
\[
A_{i,j}=\mathrm{softmax}(Q_{i,j}K_{i,j}^\top/\sqrt d)\,V_{i,j},
\]
and weighted merging
\[
z_{\rm agg}(i,j)=\sum_{k=1}^K w_{i,j,k} z_{i,j}^{(k)}.
\]
On VoxCeleb2, the single-view baseline reported \(L1=13.16\), \(PSNR=22.79\) dB, \(SSIM=0.758\), and \(MS\text{-}SSIM=0.828\), whereas the 3-view model MAX-RS-3 reported \(L1=10.00\), \(PSNR=24.87\) dB, \(SSIM=0.795\), and \(MS\text{-}SSIM=0.872\) [2203.15401].

A more explicit multi-reference animation framework is MRDAC, which augments a conventional HEVC or VVC pipeline by carrying model side information in SEI, maintaining a small buffer of decoded references, predicting dense flow and occlusion from each reference in parallel, and aggregating warped deep features through weighted max-pooling [2409.01029]. Its per-reference warping is
\[
\hat\varepsilon_r=O^m_{r\to t}\cdot (O^f_{r\to t}\circ \varepsilon_r),
\]
and aggregation is
\[
\varepsilon^*=\max_{r=0\ldots N}(\lambda_r\hat\varepsilon_r),\qquad \sum_r\lambda_r=1,\;\lambda_r\propto \exp(-\kappa |t-r|).
\]
MRDAC additionally introduces a contrastive loss on warped representations from different references for the same target frame, with cosine similarity and temperature \(\tau\), to align multi-reference features and reduce reconstruction drift [2409.01029]. The paper states that, in standard long-GOP open-loop tests, MRDAC improves LPIPS by \(15\text{–}20\%\) across \(2\text{–}8\) kbps, improves MS-SSIM by \(0.05\text{–}0.10\) over DAC/MVAC, improves VMAF by \(5\text{–}12\) points at \(5\) kbps, and yields approximate BD-rate gains of \(-22\%\) versus DAC and \(-8\%\) versus MVAC; bi-directional mode provides an additional \(\sim 3\text{–}5\%\) MS-SSIM gain at the cost of \(\sim 2\) s added latency [2409.01029].

A second extension introduces **layered or progressive coding** to widen bitrate coverage. "A Hybrid Deep Animation Codec for Low-bitrate Video Conferencing" [2207.13530] combines a deep animation branch with a low-bitrate HEVC base-layer video and a learned fusion module. Keypoints and Jacobians are transmitted for animation, while HEVC P-frames at \(QP=50\) provide a \(5\text{–}20\) kbps auxiliary stream for \(256\times256@15\) fps video. Over VoxCeleb2 and Xiph.org sequences, the method reports BD-Rate gains over HEVC of \(-33.4\%\) and \(-30.7\%\) in PSNR, \(-33.4\%\) and \(-28.3\%\) in SSIM, and \(-48.8\%\) and \(-41.6\%\) in msVGG, and the authors state that H-DAC extends DAC’s operational range up to \(\approx 30\) kbps [2207.13530].

The notion of **bandwidth intelligence** is developed further by PGen and PFVC. PGen adds optional enhancement layers of auxiliary facial features at progressively coarser spatial resolutions such as \(32\times32\), \(16\times16\), and \(8\times8\), yielding a layered bitstream of base GFVC latents plus scalable feature maps [2502.17085]. The enhancement path includes a U-Net + GDN feature descriptor, a learned hyperprior and autoregressive entropy model, attention-guided signal enhancement via Hadamard product, and coarse-to-fine GAN-based generation with SPADE and multi-scale warping [2502.17085]. The paper reports that PGen extends GFVC’s useful perceptual rate range from \(<10\) kbps up to \(\sim 50\) kbps and that PGen+CFTE achieves average BD-rate improvement against VVC of DISTS \(\downarrow 35.4\%\), FVD \(\downarrow 13.0\%\), and MANIQA \(\uparrow 18.7\%\) [2502.17085].

PFVC replaces fixed-dimensional inter-frame codes with **adaptive visual tokens** at four granularities:
\[
T^{(1)}\in\mathbb{R}^{256},\quad T^{(2)}\in\mathbb{R}^{144},\quad T^{(3)}\in\mathbb{R}^{64},\quad T^{(4)}\in\mathbb{R}^{16}.
\]
The encoder selects the token prefix up to granularity level \(\ell\), and the decoder reconstructs motion features, dense flow, occlusion mask, and final frame through a GAN-based generator [2410.08485]. The paper states that PFVC strictly outperforms VVC, CFTE, and FOMM in DISTS, LPIPS, FVD, and MANIQA across the entire bitrate range, matches or slightly outperforms VVC in PSNR/SSIM below \(20\) kbps, and covers from ultra-low \(<10\) kbps up to medium \(30\text{–}50\) kbps seamlessly [2410.08485]. This suggests that hierarchical inter-frame representations address one of the central constraints identified for early GFVC systems: coarse rate control tied mainly to key-frame QP.

## 5. Standardization, interoperability, and systems aspects

GFVC has moved from isolated research codecs toward standardization through SEI carriage in standard hybrid bitstreams. The SEI-based framework proposed in 2024 defines a GFV SEI message for compact facial parameter streams such as 2D/3D keypoints, facial semantics, or compact features, inserted into a VVC bitstream and adopted as a technology under consideration in the JVET work item for Versatile Supplemental Enhancement Information, to be standardized as a new version of "ITU-T H.274 | ISO/IEC 23002-7" [2410.15105]. The message syntax includes flags for coordinate and matrix parameters, optional prediction, a precision factor, and Uniform Resource Identifiers for TranslatorNN and GenerativeNN:
```text
SEI_generative_face_video_message()
{
  coder_specify(TranslatorNN_URI, GenerativeNN_URI);
  gfv_coordinate_present_flag;
  gfv_matrix_present_flag;
  gfv_prediction_flag;
  gfv_precision_factor;
  ...
}
```
The design is backward-compatible: decoders unaware of the SEI ignore it and display the hybrid-coded base pictures, while SEI-aware decoders invoke GFV post-processing [2410.15105].

The surveys describe this as a high-level syntax for transporting facial parameters \(\{\theta\}\) and mapping codes, enabling interoperable signaling of representation type, latent dimension or point count, temporal indexing, quantization parameters, and translator flags [2506.07369][2311.02649]. In MRDAC, GFVC side information rides in the SEI payload while the conventional HEVC/VVC bitstream remains unchanged, and the SEI payload typically remains \(<100\) bytes/frame [2409.01029]. This standardization path is closely related to application-layer flexibility: user-specified animation or filtering, metaverse avatars, progressive enhancement, translator-network signaling, and model upgrades by URI changes are all explicitly identified as enabled functionalities [2410.15105].

GFVC research has also increasingly addressed **decoder complexity and deployment constraints**. The 2025 survey summarizes a low-complexity CFTE implementation based on depthwise separable convolutions and network slimming, reducing parameters from \(58.0\) M to \(5.6\) M and MACs/pixel from \(852\) to \(126\) kMAC while keeping Rate-DISTS and Rate-LPIPS virtually unchanged and still superior to VVC at low bitrate [2506.07369]. "A Lightweight Dual-Mode Optimization for Generative Face Video Coding" [2508.13547] gives a more detailed complexity-reduction study for a CFTE baseline. It replaces \(3\times3\) convolutions with depthwise separable convolutions, halves bottleneck channels in the Generation Module, and applies two-stage adaptive channel pruning using a MaskedBatchNorm2d with trainable scales \(\gamma_i\) and a learnable threshold \(\tau\). The sparsity term is
\[
\mathcal L_{\mathrm{sparsity}}=\frac{1}{N}\sum_{l=1}^L\sum_{i=1}^{C_l} m_i^{(l)},
\]
with total objective
\[
\mathcal L_{\mathrm{total}}=\mathcal L_{\mathrm{task}}+\lambda_{\mathrm{sparse}}\mathcal L_{\mathrm{sparsity}}.
\]
The resulting model reduces parameters from \(58.03\) M to \(5.58\) M and KMACs/pixel from \(852.00\) to \(94.57\), corresponding to approximately \(90.4\%\) parameter reduction and \(88.9\%\) computation saving, while the pruned CFTE-Lite model achieves BD-Rate DISTS \(-58.04\%\) and BD-Rate LPIPS \(-51.87\%\) against VVC [2508.13547].

A hardware-centric realization is provided by GRACE, which targets FPGA deployment for an animation-based generative codec decoder [2511.09272]. The system applies layer fusion and post-training static quantization, partitions lightweight tasks to the processor side and heavy operators such as Conv2D, grid-sample, upsample, and Hadamard product to programmable logic, and uses double buffering and loop unrolling for acceleration [2511.09272]. On a PYNQ-Z1 platform, the prototype reports \(1.30\) Frame/J, \(11.7\,\mu\mathrm J\) per reconstructed pixel, and energy-efficiency gains of \(24.9\times\) over CPU and \(4.1\times\) over GPU, with \(2.38\) FPS at \(256\times256\) and a \(100\) MHz PL clock [2511.09272]. These studies make clear that GFVC’s practical feasibility depends not only on coding theory but also on model compression, quantization, and hardware/software co-design.

## 6. Evaluation practice, misconceptions, and current research directions

Evaluation in GFVC has shifted decisively toward perceptual and temporal metrics. The 2025 survey reports a large-scale GFVC-compressed face video database with subjective Mean Opinion Scores based on \(33\) test sequences, \(8\) algorithms, and \(4\) QPs, evaluated under a DSCQE protocol with \(20\) participants, yielding final MOS on \(907\) sequences and \(19\) subjects after outlier processing [2506.07369]. The study finds that DISTS, LPIPS, TOPIQ, FVD, and no-reference MANIQA/FAVOR correlate best with MOS, with PLCC/SRCC approximately \(0.80\), while PSNR and SSIM correlate poorly at approximately \(0.50\) [2506.07369]. This empirical result reinforces the earlier survey observation that classic pixel-domain fidelity measures are often inadequate for generative face reconstruction [2311.02649].

A common misconception is that GFVC should be judged primarily by PSNR, in the same way as block-based hybrid codecs. The literature repeatedly qualifies this point. PFVC notes that GAN-based perceptual coding may remain below VVC in PSNR at very high rates even while outperforming in perceptual metrics [2410.08485]. The 2025 survey states that GFVC lags in PSNR/SSIM because it targets semantic-perceptual reconstruction rather than strict pixel reproduction [2506.07369]. Likewise, SGANC reports that at low BPP its PSNR is comparable to H.265/VTM while LPIPS is drastically lower and outputs are artifact-free and photo-realistic; at higher BPP GFVC leads on PSNR and MS-SSIM as well [2207.04324].

Another persistent issue is **reconstruction instability and drift**. PFVC identifies unstable reconstruction, narrow bitrate envelope, and lack of bandwidth adaptation as key limitations of prior GFVC algorithms [2410.08485]. MRDAC frames drift as the growth of reconstruction error with increasing temporal distance from reference frames and argues that multi-reference aggregation and contrastive alignment reduce drift growth rate [2409.01029]. H-DAC observes that sparse-keypoint animation can drift over long intervals and fail on non-facial motion, while the auxiliary base layer regularizes background changes and disocclusions [2207.13530]. These critiques are not contradictions of the paradigm; rather, they define the main axes along which later GFVC systems have evolved.

The application space is broader than ultra-low-bitrate face teleconferencing. IFVC emphasizes editable semantic bitstreams and interactive coding without additional manipulation processes [2302.09919]. The SEI standardization paper enumerates user-specified facial animation or filtering, metaverse avatars, progressive enhancement, chroma-key fusion, and model upgrades via URIs [2410.15105]. AVCC extends GFVC to **joint audio-video compression**. It encodes an identity code \(C_{id}\), pose code \(C_{vi}\), and audio tokens \(C_{au}\), and decodes them through an Audio-Visual Diffusion module with cross-attention between audio and visual latents [2512.15262]. The cross-modal diffusion objective denoises both modalities jointly, and the training loss adds perceptual, adversarial, feature-matching, and lip-sync terms,
\[
\mathcal{L}_{\mathrm{total}}=\lambda_{\mathrm{per}}\mathcal{L}_{\mathrm{per}}+\lambda_{\mathrm{adv}}\mathcal{L}_{\mathrm{adv}}+\lambda_{\mathrm{fea}}\mathcal{L}_{\mathrm{fea}}+\lambda_{\mathrm{sync}}L_{\mathrm{sync}}+\mathcal{L}_{\mathrm{diff}}.
\]
The paper reports that AVCC achieves uniformly better video reconstruction quality than VVC and leading GFVC methods in DISTS, LPIPS, and FID, saves up to \(96\%\) bitrate versus VVC on selected datasets, and improves audio quality at \(1.40\) kbps relative to Encodec at \(1.50\) kbps in MEL, STFT, VIS, and WER [2512.15262]. A plausible implication is that GFVC is beginning to evolve from face-only semantic video coding toward multimodal communication systems.

Current challenges remain explicit in the survey literature: computational demands and edge deployment, interpretability and robustness, quality assessment, generalization beyond celebrity-style training data, and privacy and ethics related to realistic face generation [2506.07369][2311.02649]. The same sources identify likely future directions: diffusion and transformer generators, temporal models, full-bandwidth coverage, improved GFVC-tailored metrics, hardware/software co-design, and extension beyond faces to full scenes or human bodies [2506.07369][2311.02649].

Source: https://www.emergentmind.com/topics/generative-face-video-coding-gfvc