MFGDiffusion: Mask-Guided Smoke Synthesis

• MFGDiffusion is a mask-guided diffusion framework that generates realistic forest-fire smoke images by inpainting with explicit mask and masked-image feature conditioning.
• It enhances a pre-trained latent diffusion model with a ResNet50 branch and Joint Cross-Attention to ensure geometric alignment and contextual consistency.
• The framework improves downstream wildfire detection by augmenting real datasets with high-quality synthetic images, boosting YOLOv8 mAP50 by up to 15–20 points.

Searching arXiv for the primary paper and closely related diffusion/inpainting context. MFGDiffusion is a mask-guided diffusion framework for synthesizing forest-fire smoke in images in order to alleviate the scarcity of smoke data for image-based wildfire detection. It is built on a pre-trained Latent Diffusion Inpainting model, specifically Stable Diffusion 2 inpainting, and augments that backbone with explicit mask-feature and masked-image-feature conditioning, a dedicated loss for boundary consistency, and a synthetic-data curation pipeline using multimodal models (Wu et al., 15 Jul 2025). In the formulation reported for the method, the framework addresses a central failure mode of existing inpainting systems for smoke synthesis: inconsistencies between synthesized smoke and the background context, especially around semi-transparent, fuzzy boundaries that are visually characteristic of early-stage smoke (Wu et al., 15 Jul 2025).

1. Problem setting and research motivation

Smoke is identified as the first visible indicator of a wildfire, and image-based smoke detection is described as a crucial method for detecting and preventing forest fires (Wu et al., 15 Jul 2025). The reported bottleneck is the scarcity of smoke image data from forest fires, which hinders supervised detector training. MFGDiffusion is positioned as a generative response to that data limitation by producing realistic and diverse synthetic smoke images that can subsequently be used for detector augmentation (Wu et al., 15 Jul 2025).

The framework is specifically designed for the inpainting setting. Rather than generating unconstrained scenes, it learns to “add smoke here” given an image, a smoke mask, and a text prompt. This design reflects the observation that smoke synthesis for downstream detection requires not merely plausible image generation, but geometric and contextual alignment between inserted smoke and the underlying forest scene (Wu et al., 15 Jul 2025).

A common misconception would be to interpret MFGDiffusion as a generic diffusion model or as a term belonging to mean field games. The supplied literature contains a distinct and unrelated use of the label in the context of controlled-diffusion mean field games (Ignazio et al., 2024). In the forest-fire literature, however, MFGDiffusion denotes the mask-guided smoke synthesis framework introduced for enhanced forest fire detection (Wu et al., 15 Jul 2025).

2. Core architecture

MFGDiffusion builds on a pre-trained Latent Diffusion Inpainting model whose core components are a VAE encoder/decoder, a U-Net denoiser $\epsilon_\theta(z_t,t,C)$, and a CLIP text encoder for prompt embedding (Wu et al., 15 Jul 2025). The architectural modification that defines the method is a ResNet50 feature-extractor branch that separately ingests the one-channel binary mask $M$ and the masked image $I \odot (1-M)$ (Wu et al., 15 Jul 2025).

The mask input is replicated to three channels before entering the ResNet50 branch. At selected U-Net blocks, indexed as layers $0,\dots,8$, the extracted ResNet50 features are projected into the U-Net intermediate dimensionality and fused by cross-attention (Wu et al., 15 Jul 2025). The reported feature tensors are

$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$

These are mapped into the U-Net feature space $\mathbb R^{B\times C_u\times H_u\times W_u}$ through a learned linear layer before fusion (Wu et al., 15 Jul 2025).

The placement of conditioning signals is heterogeneous across depth. Low-resolution layers, including layers 1, 4, and 7, receive both mask and masked-image features through a Joint Cross-Attention module. The very lowest layer, reported as layer 4, injects only the mask to capture coarse shape, whereas the very highest layers, 0 and 8, inject only the masked image to preserve background detail (Wu et al., 15 Jul 2025). This allocation suggests an explicit separation between global geometric guidance and high-frequency background preservation.

At a fused layer with input feature map $X$, the paper summarizes the fusion as

$$Z_1 = \mathrm{Softmax}\Bigl(\tfrac{XW^Q\,(F_mW_m^K)^{T}}{\sqrt{d}}\Bigr)\,(F_mW_m^V),$$

$$Z_2 = \mathrm{Softmax}\Bigl(\tfrac{XW^Q\,(F_MW_M^K)^{T}}{\sqrt{d}}\Bigr)\,(F_MW_M^V),$$

$$X_{\mathrm{next}} = X + \mathrm{MLP}\bigl(\mathrm{Concat}(Z_1,Z_2)\bigr).$$

In operational terms, the pipeline takes input image $M$0, mask $M$1, and text prompt $M$2; encodes $M$3 to a latent $M$4 through the VAE; samples noise $M$5; extracts $M$6; fuses those features into U-Net layers 0–8 by Joint Cross-Attention or single-stream cross-attention; and decodes the resulting latent through the VAE decoder to generate the inpainted image (Wu et al., 15 Jul 2025).

3. Mask-aware training objective

A central component of MFGDiffusion is the mask random difference loss, introduced to handle the fact that early-stage smoke has semi-transparent, fuzzy edges (Wu et al., 15 Jul 2025). The stated rationale is that a model trained against a hard binary mask may otherwise cling too tightly to that mask, producing overly rigid smoke boundaries. To counteract this, the method randomly erodes or dilates the mask at each training step.

The paper denotes binary dilation and erosion by $M$7 and $M$8, with kernel size $M$9. It then samples a random sequence of three morphological operations, each chosen uniformly as dilation or erosion with $I \odot (1-M)$0, producing a modified mask $I \odot (1-M)$1 (Wu et al., 15 Jul 2025). One reported form is

$I \odot (1-M)$2

with random orderings also allowed.

Training combines two losses during diffusion denoising. The standard denoising loss is an MSE over the entire latent,

$I \odot (1-M)$3

while the mask random-difference loss is an MSE restricted to the modified mask region,

$I \odot (1-M)$4

The combined objective is

$I \odot (1-M)$5

The ablation summary reports that too small a value of $I \odot (1-M)$6 makes the smoke edge too rigid, whereas too large a value causes mask control to be lost; the best value is reported as $I \odot (1-M)$7 (Wu et al., 15 Jul 2025). Within the paper’s own interpretation, the loss is therefore not merely a regularizer but a mechanism for reconciling explicit mask control with visually plausible smoke translucency.

4. Data construction and synthetic dataset pipeline

The framework depends on a multi-stage data preparation process that produces both fine-tuning triplets and a large synthetic detector-training dataset (Wu et al., 15 Jul 2025). For masks, it starts from existing smoke-detection datasets with bounding boxes and feeds those boxes into SAM, the Segment Anything Model, to auto-segment precise smoke masks $I \odot (1-M)$8 (Wu et al., 15 Jul 2025). For captions, it uses BLIP-2 with a frozen vision encoder, lightweight Q-former, and LLM to generate short descriptive captions of at most 20 tokens. This yields $I \odot (1-M)$9 triplets for approximately $0,\dots,8$0 real smoke images, which are used to fine-tune MFGDiffusion in the inpainting setting (Wu et al., 15 Jul 2025).

Synthetic dataset creation then proceeds from approximately $0,\dots,8$1 smoke-free forest backgrounds collected from live-camera archives (Wu et al., 15 Jul 2025). For each background, a large multimodal LLM such as Qwen2-VL rewrites the original caption into “the same scene with smoke,” relying on the model’s paraphrasing (Wu et al., 15 Jul 2025). Two masks are randomly sampled from the $0,\dots,8$2 pool for each background, and for each pair $0,\dots,8$3 the system generates three diffusion outputs, yielding approximately $0,\dots,8$4 synthetic images (Wu et al., 15 Jul 2025).

The paper then introduces a filtering stage based on smoke-specific perceptual criteria. Three smoke-quality metrics are defined: color fidelity, visibility, and semi-transparency (Wu et al., 15 Jul 2025). After manually annotating 100 examples, the authors fine-tune Qwen2-VL to predict each metric on a $0,\dots,8$5 scale. All $0,\dots,8$6 synthetic images are scored, and a weighted sum is computed as

$0,\dots,8$7

The top 50% by this ranking are retained, producing a final set of approximately $0,\dots,8$8 high-quality synthetic images (Wu et al., 15 Jul 2025). Automatic annotation is then obtained by converting each filtered mask to its tightest bounding box, enabling detector training without human labeling effort (Wu et al., 15 Jul 2025).

The following table summarizes the reported pipeline stages.

Stage	Method	Reported scale
Real smoke triplets	SAM for masks; BLIP-2 for captions	$0,\dots,8$9
Smoke-free backgrounds	Live-camera archives	$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$0
Synthetic generation	2 masks per background, 3 outputs per mask	$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$1
Filtering	Qwen2-VL scoring; keep top 50%	$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$2

This pipeline indicates that MFGDiffusion is not solely a model architecture. It is a full data-engineering framework coupling segmentation, caption generation, mask-guided diffusion, and multimodal quality control (Wu et al., 15 Jul 2025).

5. Experimental evaluation

The reported smoke-generation evaluation compares MFGDiffusion with SD-Inpainting, Strategy-based SD-edit, Blended Latent Diffusion, and PowerPaint (Wu et al., 15 Jul 2025). On a held-out real test set, the paper reports the following image-quality metrics.

Model	PSNR$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$3	SSIM$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$4	LPIPS$$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$5
Strategy-SD	26.11	0.7891	0.095
PowerPaint	24.14	0.761	0.243
SD-Inpainting	25.71	0.7682	0.104
Blended LDM	25.88	0.8071	0.133
MFGDiffusion	28.06	0.8763	0.078

The same evaluation also reports MSE and CLIP similarity. MFGDiffusion attains an MSE of 105.3 and CLIP similarity of 25.53, while the comparison methods report higher MSE values and comparable CLIP similarity values (Wu et al., 15 Jul 2025). In the paper’s framing, this indicates stronger reconstruction fidelity and perceptual similarity, although the CLIP similarity is not the best numerical value among the compared methods (Wu et al., 15 Jul 2025).

For downstream detection, the framework is evaluated with YOLOv6, YOLOv8, YOLOv9, YOLOv10, and YOLOv11 under train/test splits comparing real-only training against real-plus-synthetic training (Wu et al., 15 Jul 2025). The detailed example provided is YOLOv8. Training on real data only yields $$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$6, $$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$7, precision $$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$8, and recall $$F_m \in \mathbb R^{B\times C_m\times H_m\times W_m}, \qquad F_M \in \mathbb R^{B\times C_m\times H_m\times W_m}.$$9. Training on real plus synthetic data yields $\mathbb R^{B\times C_u\times H_u\times W_u}$0, $\mathbb R^{B\times C_u\times H_u\times W_u}$1, precision $\mathbb R^{B\times C_u\times H_u\times W_u}$2, and recall $\mathbb R^{B\times C_u\times H_u\times W_u}$3 (Wu et al., 15 Jul 2025). The broader summary states that all evaluated detector models gained approximately 15–20 points in mAP50 when augmented with the synthetic data (Wu et al., 15 Jul 2025).

These results are significant because they connect generation quality to a downstream operational objective. Rather than evaluating synthesis in isolation, the framework measures whether the generated data improves smoke detection under held-out real testing conditions (Wu et al., 15 Jul 2025).

6. Ablations, interpretation, and limitations

The ablation study reports that removing the Joint Cross-Attention module, or using only mask features or only masked-image features, causes visible degradation (Wu et al., 15 Jul 2025). This supports the claim that the method depends on dual conditioning: the mask provides geometric smoke placement, while the masked image preserves scene-specific context.

A second ablation compares ResNet variants and finds that ResNet50 is superior to ResNet101 and ResNet152 in stability (Wu et al., 15 Jul 2025). The summary does not provide further quantitative detail, so the interpretation remains architectural rather than statistical. A plausible implication is that the auxiliary conditioning branch benefits from moderate-capacity feature extraction rather than deeper backbones, at least in this inpainting regime.

The filtering stage is also ablated. When unfiltered synthetic data are mixed 1:1 with real data, YOLOv8 reaches $\mathbb R^{B\times C_u\times H_u\times W_u}$4; when filtered synthetic data are used, the reported value is $\mathbb R^{B\times C_u\times H_u\times W_u}$5 (Wu et al., 15 Jul 2025). This indicates that synthetic data quality, not merely synthetic data quantity, affects detector performance.

The method also has explicit scope conditions. Its masks originate from existing smoke-detection datasets with bounding boxes and are refined through SAM (Wu et al., 15 Jul 2025). Its captions are generated in a short descriptive form by BLIP-2, and its synthetic prompts are rewritten into “the same scene with smoke” by Qwen2-VL (Wu et al., 15 Jul 2025). This suggests that the framework is optimized for controlled domain transfer within forest-monitoring imagery rather than for arbitrary text-to-image smoke generation.

7. Position within the literature and practical significance

Within diffusion-based image synthesis, MFGDiffusion belongs to the branch of inpainting methods that incorporate external structural guidance. Its direct comparison set in the paper includes Stable Diffusion inpainting, strategy-based Stable Diffusion editing, Blended Latent Diffusion, and PowerPaint (Wu et al., 15 Jul 2025). The framework’s distinctive contribution is the explicit use of both mask and masked-image features as dedicated guidance signals, together with a loss that perturbs the mask boundary during training to model semi-transparent smoke edges (Wu et al., 15 Jul 2025).

Its practical significance lies in converting a detector-data scarcity problem into an overview-and-filtering problem. The full pipeline described in the paper begins with real smoke-detection data $\mathbb R^{B\times C_u\times H_u\times W_u}$6, extracts masks by SAM and captions by BLIP-2, fine-tunes MFGDiffusion with masked latent diffusion loss plus random-difference loss, generates synthetic smoke images on background forests using Qwen2-VL paraphrases, filters those images by Qwen2-VL smoke-quality scoring, derives bounding boxes from masks, and trains YOLO-family detectors on the mixed dataset (Wu et al., 15 Jul 2025). The framework therefore integrates generative modeling, multimodal annotation, multimodal evaluation, and detector training into a single workflow.

In the narrow sense, MFGDiffusion denotes a mask-guided diffusion architecture for smoke inpainting. In the broader sense suggested by the reported pipeline, it is a synthetic-data production system for forest-fire smoke detection, designed to improve realism, diversity, and downstream detector utility under conditions of limited real wildfire imagery (Wu et al., 15 Jul 2025).