Surgical Imagen: Laparoscopic Diffusion Model

• Surgical Imagen is a diffusion-based text-to-image model that generates photorealistic laparoscopic images from structured instrument–verb–target triplets.
• It integrates a frozen T5 text encoder with a conditioned U-Net backbone to achieve superior semantic alignment and robust handling of imbalanced surgical data.
• The approach enhances surgical dataset augmentation by delivering high-fidelity, semantically consistent images, while noting challenges such as high GPU cost and edge-case anatomical imperfections.

Surgical Imagen is a domain-adapted diffusion-based text-to-image generative model expressly designed for surgical scenes, particularly laparoscopic procedures. It extends the Imagen diffusion framework to support high-fidelity photorealistic synthesis of surgical images directly from structured textual action triplets, enabling both data augmentation for computer vision tasks and the rapid creation of synthetic datasets under stringent clinical and ethical data constraints (Nwoye et al., 2024).

1. Dataset, Prompt Format, and Class Distribution

Surgical Imagen is trained on the CholecT50 dataset, which consists of approximately 50 laparoscopic cholecystectomy videos (~101,000 annotated frames), each annotated with a variable number of instrument–action–target triplets per frame. For each frame $X_i$, the associated set of triplets is

$$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$

Representative examples include $\langle$“grasper”, “retract”, “gallbladder”$\rangle$ and $\langle$“clipper”, “clip”, “cystic duct”$\rangle$. The marginal class distribution is highly imbalanced (e.g., “grasper” >30%, “bipolar” <1%), and the 21 most frequent triplet categories account for over 80% of all frames. This pronounced imbalance motivates instrument-centric techniques for dataset rebalancing (Nwoye et al., 2024).

2. LLM Selection and Embedding Construction

The model's text encoder is based on T5 (Text-To-Text Transfer Transformer), selected over alternatives such as SBERT due to its superior inter-class separation for short, structured inputs. Each triplet $\mathbf{t}$ is tokenized, embedded, and position-encoded as follows:

• Token embedding: $$e_i^\mathrm{token} = W_\mathrm{token} \cdot \mathrm{one\_hot}(t_i)$$
• Positional encoding based on standard sinusoidal mapping
• Layer normalization and aggregation to obtain a global embedding $e_T \in \mathbb{R}^{d_\mathrm{model}}$ ($d_\mathrm{model}=512$ for T5-Small). PCA/t-SNE analyses revealed that the resulting latent space is highly instrument-centric; embeddings are clustered more strongly by instrument than by verb or target (Nwoye et al., 2024).

3. Instrument-Based Class Balancing Strategy

To correct the instrument-dominated class skew, each sample $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$0 with instrument $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$1 receives a weight $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$2, where $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$3 is the frequency of instrument $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$4 in the training set. The normalized weights $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$5 are used to derive sampling probabilities $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$6, ensuring rare instruments are oversampled and improving convergence properties during training. This data rebalancing is implemented via categorical sampling in the mini-batch loader (Nwoye et al., 2024).

4. Surgical Imagen Architecture and Training Procedure

The core of Surgical Imagen is a diffusion model comprised of a frozen T5 text encoder and a conditioned U-Net backbone. The system operates as follows:

• The T5 encoder processes the triplet prompt into $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$7.
• A base diffusion U-Net produces a low-resolution (64$$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$864) image, conditioned on $$T_i = \{\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_1\,,\,\ldots,\,\langle\text{instrument},\,\text{verb},\,\text{target}\rangle_k\,\}.$$9 via cross-attention at multiple hierarchical stages.
• A separate, text-conditioned super-resolution Efficient U-Net module upsamples outputs to 256$\langle$0256.

The forward process is

$\langle$1

and the reverse process is

$\langle$2

with

$\langle$3

and main regression loss $\langle$4 for the diffusion model and $\langle$5 for the super-resolution branch. Overall parameter count is approximately 321.9M; training uses up to 300,000 iterations and Adafactor (base) and Adam (SR) optimizers (Nwoye et al., 2024).

5. Performance Metrics and Quantitative Benchmarks

Surgical Imagen is evaluated via:

• FID (Fréchet Inception Distance): quantifies distributional similarity between synthetic and real images; Surgical Imagen achieves 3.70 (vs. StackGAN’s 5.83).
• CLIP score: cosine similarity between vision and text embeddings; Surgical Imagen yields 26.84 ± 0.45% (real-image CLIP: 23.01 ± 0.11%). A blinded expert study (13 surgeons, 640 images; 50% synthetic, 50% real) yielded a photorealism recognition rate of real: 57.7%, synthetic: 34.6%; prompt-to-image alignment: real 72.3%, synthetic 43.6%. These results establish the photorealistic and semantic challenge of distinguishing Surgical Imagen outputs from true clinical data (Nwoye et al., 2024).

Model	FID	CLIP (%)	Real in survey (%)	Synth in survey (%)
Surgical Imagen	3.70	26.84 ± 0.45	57.7	34.6
StackGAN	5.83	-	-	-

6. Qualitative Properties and Model Limitations

Surgical Imagen produces high-fidelity images reflecting realistic lighting, textural details, and common surgical phenomena (e.g., electrocautery smoke). The model displays robust compositional capability, rendering plausible imagery even for implausible prompts (e.g., multiple simultaneous gallbladders or instruments). Some known artifacts include geometric distortion at frame borders, imperfect tool-tissue alignment, and minor texturing errors for small or rare instrument classes.

Limitations are as follows:

• Training data and prompt engineering are specific to laparoscopic cholecystectomy (CholecT50); generalization to other procedures remains untested.
• Each output requires 1000 diffusion steps, resulting in high GPU inference costs.
• Clinical safety and realistic anatomical representation in edge cases require expert review before downstream use.
• Synthesized datasets should complement, not supplant, real clinical collections. Future work proposes timed video synthesis, multi-modal conditioning (e.g., ultrasound-to-laparoscopic imagery), integration of clinician feedback, and extension to stereoscopic or VR-enabled surgical training (Nwoye et al., 2024).

7. Integration with the Broader Field of Surgical Synthetic Data Generation

Surgical Imagen situates itself among several synthetic data approaches. Compared to prior diffusion-based laparoscopic image generators (Allmendinger et al., 2023), its use of action triplet–conditioned prompts and instrument-centric balancing yields superior semantic alignment and rare-action representation. Augmentation of surgical datasets with these synthetic samples has been shown to confer improvements in tool/action classification and segmentation, highlighting the practical utility of generative models in surgical data-scarce environments (Nwoye et al., 2024, Allmendinger et al., 2023).

Other approaches using 3D Gaussian splatting (Zeng et al., 2024), unpaired image translation with consistency models (Venkatesh et al., 2024), and chroma-key compositing (Garcia-Peraza-Herrera et al., 2021) target the broader challenge of photorealistic, richly annotated synthetic data with explicit geometric, physical, or semantic consistency.

Surgical Imagen exemplifies the integration of large-scale conditional diffusion modeling, NLP-based triplet conditioning, and surgical domain-specific dataset engineering, and represents a core framework for future progress in simulation, training, and algorithm development for surgical computer vision (Nwoye et al., 2024).