EndoGen: Conditional Endoscopic Video Synthesis

• EndoGen is a conditional autoregressive framework for synthesizing high-fidelity endoscopic videos with explicit control over diagnostic classes.
• It integrates an AR transformer with Spatiotemporal Grid-Frame Patterning and Semantic-Aware Token Masking to effectively capture both spatial and temporal dynamics.
• Extensive evaluations show that EndoGen outperforms previous methods in video synthesis and segmentation metrics, enhancing data augmentation for clinical applications.

EndoGen is a conditional autoregressive framework for endoscopic video generation, proposed as the first architecture explicitly designed for high-fidelity, temporally coherent video synthesis conditioned on specified medical classes (e.g., polyp, cancer, or surgical task). EndoGen addresses two central limitations of preceding work: the lack of temporal dynamics in static image generation and the absence of clinically meaningful control in unconditional video generation approaches. The framework centers on an autoregressive (AR) transformer paired with a Spatiotemporal Grid-Frame Patterning (SGP) strategy for compact joint modeling of space-time dependencies, and introduces a Semantic-Aware Token Masking (SAT) mechanism to concentrate modeling capacity on diagnostically relevant regions. Extensive quantitative and qualitative evaluation establishes gains across standard video synthesis and medical segmentation metrics, with synthetic sequences rivaling or surpassing real data for downstream training (Liu et al., 23 Jul 2025).

1. Methodological Framework

EndoGen models endoscopic video synthesis as an AR sequence modeling task in the latent token space, with explicit conditional control. Let $X_{1:T}$ denote discrete latent tokens for $T$ video frames (each with $L$ spatial tokens), and $C$ a learnable class condition embedding. The model optimizes: $$p(X_{1:T}\mid C) = \prod_{t=1}^T p\bigl(X_t\mid X_{<t}, C\bigr)$$ Training proceeds by prepending $C$ as a token and applying a transformer decoder, initialized from an ImageNet-conditional AR model, to maximize the cross-entropy: $$\mathcal{L}_{\mathrm{CE}} = -\sum_{t=1}^T \log p\bigl(X_t^\ast\mid X_{<t}^\ast, C\bigr)$$ where $X^\ast$ are ground-truth tokens derived via a frozen VQGAN encoder.

Conditional control is achieved by selecting $C$ from a pool of learned embeddings, each corresponding to a pathology class or procedural task. At inference, only $C$ is supplied; the model generates a full matching video sequence de novo, demonstrating conditional generative capability unmatched by previous unconditional or image-only methods.

2. Spatiotemporal Grid-Frame Patterning (SGP)

The SGP strategy reformulates short video clips $T$0 into a single 2D grid image $T$1 by stacking frames row-wise: $T$2. A frozen VQGAN ($T$3) encodes this grid to discrete tokens: $T$4 where $T$5 is the codebook size. Subsequent reconstruction uses the VQGAN decoder ($T$6): $T$7, which is then split back into $T$8 frames.

SGP enables simultaneous spatial and temporal context modeling within standard 2D transformer infrastructure, capitalizing on global attention mechanisms to capture dynamics without expensive 3D convolutions or complex spatiotemporal attention block arrangements.

3. Semantic-Aware Token Masking (SAT)

SAT aims to prioritize modeling diagnostically salient regions by masking out tokens corresponding to less informative (often background) regions during training. For a batch of embeddings $T$9, each sequence is partitioned into segments of length $L$0. The variance $L$1 for a segment $L$2 is computed: $L$3 The adaptive masking ratio $L$4 for each segment is defined as: $L$5 with $L$6. A binary mask $L$7 is sampled, yielding masked segment $L$8. Only unmasked tokens contribute to the loss, focusing learning on high-variance, semantically salient regions such as lesions.

4. Evaluation and Quantitative Results

The framework is evaluated on HyperKvasir (8 gastrointestinal pathologies) and SurgVisdom (3 porcine surgical tasks) using standard and domain-adapted video metrics: Fréchet Video Distance (FVD), Content-Debiased FVD (CD-FVD), FID, and LPIPS. Table 1 summarizes performance on the HyperKvasir dataset:

Method	Avg. FVD
SimDA (diffusion)	3391.8
VDM (diffusion)	2125.4
VideoGPT (AR)	1548.6
EndoGen (AR+SGP+SAT)	507.2

EndoGen also demonstrates:

• CD-FVD: 765.3 (vs prior best 851.4)
• FID: 76.56 (vs prior best 235.8)
• LPIPS: 0.528 (vs prior best 0.563)

On SurgVisdom, EndoGen reduces FVD to 1393.6, representing a ≈40% improvement over VDM (2343.3).

Qualitative inspection reveals EndoGen sequences feature crisp vessel structures, realistic mucosal textures, and consistent object shapes, with minimal temporal artifacts under moderate motion.

5. Downstream Impact: Semi-Supervised Segmentation

EndoGen-synthesized frames enhance downstream polyp segmentation when used as unlabeled data in semi-supervised settings. On HyperKvasir, a 10% labeled split combined with FixMatch or PolypMix leveraging EndoGen synthetic sequences yields segmentation performance matching—or exceeding—use of real unlabeled data (see Table 2):

Method	Dice (%)
Supervised only	69.75
FixMatch + real	70.80
FixMatch + synthetic	70.96
FixMatch + both	71.03
PolypMix + real	87.13
PolypMix + synthetic	87.84
PolypMix + both	87.92

This demonstrates the practical utility of conditionally generated sequences for data-efficient model training in medical imaging.

6. Limitations and Failure Modes

EndoGen exhibits several practical constraints:

• Resolution is bounded by GPU memory (primary experiments at $L$9, some $C$0).
• Video length is demonstrated for $C$1–$C$2 frames; longer sequences may require hierarchical/blocked AR sampling.
• Single-class token conditioning does not encode fine spatial targets (e.g., precise polyp location).
• Extreme camera motions or rare, out-of-distribution pathologies degrade temporal coherence or fidelity.

A plausible implication is that future variants might incorporate hierarchical or multi-modal conditioning to overcome these boundaries.

7. Prospects for Extension

Promising directions include:

• Extension to higher resolution and longer-horizon synthesis by sliding-window or hierarchical AR sampling.
• Multi-modal conditioning (text, segmentation maps, keyframes) to increase control granularity.
• Integration of diffusion-based refinement steps within the AR backbone, aiming for further perceptual fidelity gains.
• Real-time and interactive applications in surgical simulation environments.

EndoGen establishes a new state-of-the-art for conditional endoscopic video generation, combining principled AR sequence modeling, efficient spatiotemporal embedding, and semantic-focused training to deliver sequences of high utility for both clinical simulation and data augmentation tasks (Liu et al., 23 Jul 2025).