SemanticGen: Semantic-Driven Generative Models

• SemanticGen is a class of generative models that integrate explicit semantic structure to preserve high-level properties and enhance interpretability.
• It employs methods like SPGAN and two-stage video diffusion, enabling efficient synthesis through semantic guidance and staged refinement.
• Empirical findings across domains show improved data efficiency, reduced computational costs, and enhanced control compared to traditional generative approaches.

SemanticGen refers to a class of generative models that explicitly impose or leverage semantic structure during data generation, aiming to preserve or exploit high-level properties and representations for more efficient, controllable, or data-efficient synthesis. Two primary frameworks prominently exemplify this concept: (1) Semantic Preserving Generative Adversarial Models (SPGAN), which replace the standard GAN discriminator with a semantic engine to provide semantic-level guarantees (Harel et al., 2019), and (2) SemanticGen for video generation, which employs a two-stage diffusion process operating first in a compact semantic space, then refining to pixel output (Bai et al., 23 Dec 2025). These frameworks share the underlying principle of organizing the generation pipeline around explicit semantic information, departing from conventional approaches that directly operate in high-dimensional data or VAE latent spaces.

1. Motivations and Objectives

SPGAN introduces semantics-driven generation to enforce global or domain-specific properties, such as molecular activity in chemistry or meaningful spatial distributions in geolocation tasks. Standard GAN discriminators are typically differentiable networks with little explicit semantic structure, resulting in high data requirements and weak interpretability. By contrast, SPGAN replaces the discriminator with a calibrated, non-differentiable classifier designed around a library ℱ of domain-relevant, black-box semantic functions. This enables guarantees that the generated data matches real data along chosen semantic axes and provides a well-defined, interpretable stopping condition: termination occurs when no semantic feature distinguishes generated from real samples beyond a tunable threshold (Harel et al., 2019).

SemanticGen for video generation addresses inefficiencies in text-to-video models that learn distributions over dense VAE latents via diffusion. Such models suffer from slow convergence and cubic inference cost due to the vast number of tokens required for long videos. SemanticGen's core insight is that global video semantics—such as object layout, motion, and scene structure—occupy a low-dimensional, highly compact space. Generation should begin by synthesizing semantic trajectories and only subsequently add high-frequency detail, vastly reducing computational load and enabling explicit control over global content (Bai et al., 23 Dec 2025).

2. Core Architectures and Algorithms

Semantic Preserving GAN (SPGAN)

The SPGAN model comprises:

• Generator $G_\theta$, parameterized as any neural network (FFN, LSTM, etc.), mapping source noise to output space (finite or continuous).
• Semantic engine $E$, a non-differentiable classifier outputting $E(x)=\mathbb{P}(\text{real}|x)\in[0,1]$ for $x\in\mathcal{X}$, trained with a balanced real/generated dataset and features from ℱ, then subjected to feature selection for parsimony and generalization.
• Generation loop:
  1. $G_\theta$ produces samples labeled "fake" and combined with "real" data.
  2. $E$ retrained; AUC is computed on the real/fake discrimination task.
  3. If AUC $> 1/2+\epsilon$, $E(z_i)$ acts as a reward; $G_\theta$ is updated by REINFORCE, moving toward samples $E$ judges as "real".
  4. Iteration continues until AUC $\leq 1/2+\epsilon$ or maximum epochs.

Over time, $E$ selects progressively finer semantic features from ℱ, "closing the semantic gap" between real and generated data.

SemanticGen for Video Generation

The SemanticGen video pipeline comprises two tightly coupled diffusion stages:

• Semantic Feature Extraction: For a video $V\in\mathbb{R}^{3\times F\times H\times W}$, a frozen video transformer (e.g., Qwen-2.5-VL) extracts embeddings $$z'_\text{sem}=E_\text{sem}(V)\in\mathbb{R}^{d\times F_s/2\times H/28\times W/28}$$, where $F_s\ll F$ (e.g., 1.6 fps). A lightweight MLP yields distributional parameters $(\mu,\sigma)$, and samples $$z_\text{sem}\sim \mathcal{N}(\mu(z'_\text{sem}),\,\text{diag}(\sigma^2(z'_\text{sem})))$$ in a compressed $k$-dimensional subspace.

• Stage 1: Semantic Diffusion: A diffusion model is trained on $p(z_\text{sem})$ using a DDPM-like schedule, with loss

$$\mathcal{L}_\text{semantic} = \mathbb{E}_{s_0,\,\epsilon,\,t}\|\,\epsilon-\epsilon_\theta(s_t,t)\|^2_2,$$

where $s_t$ is the noisy version of $z_\text{sem}$. The denoiser $\epsilon_\theta$ is parameterized as a spatio-temporal DiT-style U-Net.

• Stage 2: VAE Latent Diffusion: The VAE encoder produces $z_0$, and a second diffusion model learns $p(z_0\,|\,z_\text{sem})$ with in-context conditioning via concatenation. The same DDPM machinery applies; loss

$$\mathcal{L}_\text{VAE} = \mathbb{E}_{z_0,\,\epsilon,\,t}\|\,\epsilon-\epsilon_\theta(z_t,t;\,z_\text{sem})\|^2_2.$$

For long videos, full attention is restricted to semantic tokens, while latent diffusion leverages shifted-window attention to maintain scalability.

3. Theoretical Formulation and Guarantees

SPGAN Guarantees

• Generator Objective:

$$J(\theta)=\mathbb{E}_{j\sim G_\theta}[p(j)] = \sum_{j=1}^b l(j)\,p(j),$$

with gradient (REINFORCE trick): $$\nabla_\theta J(\theta)=\sum_j p(j)\nabla_\theta \log l(j)$$.

• Classifier Calibration: $E$ is trained with regularized cross-entropy, subject to early stopping and feature selection to avoid overfitting in high-dimensional semantic feature spaces. The ideal $E$ has AUC $\geq 1/2$ (properness), and if a semantic statistic $h\in H$ sharply separates distributions, AUC $>1/2+\alpha$ (separability).
• Semantic Divergence Metric:

$$d_H(D,G):= \sup_{h\in H}|\mathbb{E}_{x\sim D}[h(x)]-\mathbb{E}_{z\sim G}[h(z)]|.$$

If $\text{AUC}(E)<1/2+\alpha$, then $d_H(D,G)\leq \gamma\alpha$ for some $\gamma<1$, certifying semantic alignment of distributions up to $O(\epsilon)$. Termination at AUC $\leq 1/2+\epsilon$ provides an explicit certificate of semantic preservation.

This approach enables data efficiency (sample complexity is dictated by $|H|$, not ambient dimension), a transparent stopping rule, and transferable feature sets for downstream tasks (Harel et al., 2019).

SemanticGen Video Diffusion

• The semantic-token regime, via severe temporal and spatial downsampling, yields a low-dimensional manifold suitable for global planning and attention over long-range video structure.
• By deferring fine detail to conditional diffusion in VAE space, SemanticGen avoids the cubic scaling of standard bidirectional attention models, enabling efficient synthesis of long sequences (60 sec, 2.4K frames) with a fixed computational budget (Bai et al., 23 Dec 2025).

4. Practical Implementations and Empirical Findings

SPGAN Domain Applications

Application	Semantic Feature Library ℱ	Results & Interpretation
Cellular antennas	500K+ OpenStreetMap predicates	Uncovered feature hierarchy; error reduction in transfer tasks
Molecule generation	Molecular descriptors (MolWt, logP)	Improved validity, uniqueness at low data regimes
Trajectory extrapolation	Difference, std, length predicates	Factor-of-3 error reduction in small-sample regime

SPGAN generalizes to any domain admitting a large library ℱ of semantically meaningful features. A natural stopping rule is provided by classifier AUC, and the resulting feature set $H$ is interpretable and reusable.

SemanticGen Video Results

Short and long video synthesis was benchmarked against state-of-the-art models (Hunyuan-Video, Wan2.1-T2V-14B, etc.) using VBench and VBench-Long. SemanticGen achieves competitive or superior performance across key metrics:

Method	Subject Consistency	ImageQ	Aesthetic	Δ^M_{drift} (long)
Hunyuan-Video	91.11%	64.23%	62.60%	—
Wan2.1-T2V-14B	97.23%	66.63%	65.61%	—
Base-CT	96.17%	65.77%	63.97%	—
SemanticGen	97.79%	65.23%	64.60%	3.58%

Qualitative evaluation demonstrates improved adherence to text prompts and long-term temporal coherence, with reduced color drift on long clips. Training efficiency is significantly improved, requiring $\sim$60K updates compared to $>$200K in non-semantic baselines (Bai et al., 23 Dec 2025).

5. Efficiency, Limitations, and Future Directions

Efficiency Analysis

SemanticGen achieves:

• Short-video inference: 4 s/frame vs. 7 s/frame (full VAE model) on identical hardware.
• Long-video synthesis: 60 s clip completed in $\sim$600 s, approximately $3\times$ faster than sparse-attention baselines.
• Reduction in necessary training steps by a factor of 3–4 for quality-matched results, owing to concentrated semantic modeling.

Limitations

• Fine texture detail and high-frequency flicker (e.g., lightning) are not fully recoverable from low-fps semantic embeddings.
• The frozen semantic tokenizer (e.g., Qwen-2.5-VL) may not cover atypical video domains. Model performance is tied to properties of the chosen semantic encoder.

Future Extensions

Planned improvements include:

• Exploring alternative semantic tokenizers (VideoMAE 2, V-JEPA 2, 4DS) to optimize spatial-temporal compression.
• Jointly training the semantic encoder and MLP compressor to better retain mid-level detail.
• Hierarchical semantic modeling with multi-rate encoders to balance global planning and fine motion generation (Bai et al., 23 Dec 2025).

6. Synthesis and Broader Impact

SemanticGen frameworks systematically incorporate semantic priors—either via explicit semantic engines for distribution matching (SPGAN) or by staged semantic/latent generation in the context of video synthesis. The resulting models demonstrate improved data efficiency, interpretable and certifiable alignment with explicit semantic properties, accelerated convergence, and dramatically enhanced scalability for long-horizon synthesis. These paradigms generalize to structured data generation tasks across scientific, geographic, chemical, and creative domains where user-specified semantics are critical constraints or desiderata.

Further development hinges on richer semantic encoders, dynamic feature selection policies, and the integration of semantic trajectories across multiple abstraction layers, promising increasing controllability and tractability for high-dimensional generative modeling (Harel et al., 2019, Bai et al., 23 Dec 2025).