SmokeBench: Dual Benchmark for Smoke Perception

• SmokeBench is a dual benchmark effort that evaluates smoke perception in two variants: one for wildfire smoke recognition/localization and one for surveillance image desmoking.
• The wildfire variant uses fixed-view imagery with a four-task evaluation suite and metrics like Accuracy and mIoU, while the surveillance variant employs paired images and restoration metrics (PSNR, SSIM, LPIPS).
• Both benchmarks reveal persistent challenges in handling early-stage smoke in detection and thick smoke in restoration, highlighting gaps in current multimodal and restoration models.

Searching arXiv for papers titled “SmokeBench” and closely related smoke benchmark works. SmokeBench is the name of two distinct benchmark efforts in computer vision and multimodal perception that were introduced in 2025. One, "SmokeBench: Evaluating Multimodal LLMs for Wildfire Smoke Detection" (Qi et al., 12 Dec 2025), evaluates whether multimodal LLMs can recognize and localize wildfire smoke in fixed-view imagery. The other, "SmokeBench: A Real-World Dataset for Surveillance Image Desmoking in Early-Stage Fire Scenes" (Jin et al., 16 Sep 2025), provides a paired benchmark for restoring smoke-degraded surveillance images recorded during the first $0\text{–}15$ minutes after ignition. The shared name reflects a common emphasis on benchmark construction for safety-critical smoke perception, but the two works target different technical problems: recognition and grounding in one case, image restoration in the other.

1. Name, scope, and disambiguation

A central point of disambiguation is that SmokeBench is not a single canonical benchmark. The 2025 wildfire benchmark studies smoke as a difficult visual target for multimodal LLMs (MLLMs), whereas the 2025 surveillance benchmark studies smoke as a visibility-degrading medium in paired image restoration. The distinction is consequential because the two benchmarks use different source data, different supervision formats, different evaluation metrics, and different notions of task difficulty (Qi et al., 12 Dec 2025, Jin et al., 16 Sep 2025).

Variant	Primary task	Core benchmark form
Wildfire SmokeBench	Smoke recognition and localization	4-task evaluation suite built from FIgLib positives and 1,000 negatives
Surveillance SmokeBench	Single-image desmoking/restoration	9,975 aligned smoke / smoke-free image pairs across 43 scenes

The wildfire benchmark treats smoke as a transparent, amorphous, low-contrast target that is often confounded with clouds, haze, or background atmosphere. The surveillance benchmark treats smoke as a dynamic, non-uniform obscurant in indoor early-stage fire scenes, and explicitly argues that smoke should not be conflated with haze. These two formulations address different failure modes: perceptual grounding failure in one case and visibility-recovery failure in the other.

2. Wildfire SmokeBench: benchmark design for MLLM evaluation

The wildfire version of SmokeBench is built from the FIgLib dataset. It uses 5,046 images with ground-truth bounding box annotations as positive samples and 1,000 non-smoke images as negative samples, with imagery coming from fixed-view cameras in Southern California. The benchmark is organized as a progressive four-task protocol: smoke classification, tile-based smoke localization, grid-based smoke localization, and smoke detection (Qi et al., 12 Dec 2025).

The task progression is designed to separate different competencies. Classification asks whether smoke is present anywhere in the image. Tile-based localization divides each image into a fixed $3 \times 4$ grid of 12 non-overlapping tiles and evaluates smoke recognition on local crops without global scene context. Grid-based localization overlays a fixed $5 \times 5$ grid with region identifiers on the full image and asks the model to output the smoke-containing regions. Detection requires direct prediction of smoke bounding boxes.

The benchmark uses Accuracy for classification and Intersection over Union with mIoU for localization and detection. The paper defines these as

$$\text{Accuracy}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\big[\hat{y}_i=y_i\big]$$

and

$\text{IoU}=\frac{|P \cap G|}{|P \cup G|}.$

A distinctive analytic component is the stratification of performance by smoke area and contrast. Smoke size is quantified as the pixel area of ground-truth bounding boxes, and images are split into five smoke-area quantile groups and five contrast quantile groups. This design allows the benchmark to test whether model success is driven primarily by plume size or by local visibility contrast.

3. Wildfire SmokeBench: empirical findings

The main empirical result is that some models can classify the presence of smoke when it occupies a large image area, but all models struggle with accurate localization, especially for early-stage smoke. On smoke classification, GPT-4o achieves the strongest overall accuracy at 0.758, followed by Gemini-2.5 Pro at 0.683. Among the open-source models reported, Idefics2 has the highest overall classification accuracy at 0.592, but its negative-sample accuracy is 0.498, which the paper interprets as a specificity failure that inflates apparent performance on positive images (Qi et al., 12 Dec 2025).

Localization performance is substantially weaker. In tile-based localization, the best reported result is 0.475 mIoU from Idefics2. In grid-based localization, the best reported result is 0.184 mIoU from Qwen2.5-VL-32B, while Qwen2.5-VL-7B scores 0.000. In direct smoke detection, the general-purpose MLLMs listed in the table—Idefics2, Qwen2.5-VL (7B), Qwen2.5-VL (32B), and InternVL3 (14B)—all score 0.000 mIoU. The strongest grounding-oriented baseline in that table is GroundingDINO at 0.245, whereas the specialized detector YOLOv8n reaches 0.773.

The benchmark’s most important analytic conclusion is that smoke area is much more influential than contrast. Performance improves markedly from “Very Small” to “Very Large” smoke, while gains across contrast bins are comparatively modest. This directly supports the benchmark’s claim that current MLLMs fail most severely in the early-stage regime that matters most for wildfire warning. A frequent misconception—that image-level smoke classification implies usable localization—is contradicted by the benchmark’s results: recognition and grounding remain sharply separated capabilities in current MLLMs.

4. Surveillance SmokeBench: a paired benchmark for early-stage fire-scene desmoking

The surveillance version of SmokeBench is a real-world surveillance image desmoking benchmark targeting the first $0\text{–}15$ minutes after ignition. Its dataset contains 9,975 precisely aligned smoke / smoke-free image pairs, partitioned into 9,875 training samples and 100 testing/evaluation samples, with evaluation reported on 4 test scenes: TestScene-1 through TestScene-4. The collection spans 43 distinct scenes generated through a four-dimensional parametric design over space scale, item density, light condition, and shooting angle (Jin et al., 16 Sep 2025).

The acquisition system consists of Scene Setup, Data Acquisition, and Data Enhancement modules. Hardware explicitly listed in the paper includes the JOOAN surveillance device, the Antari Z-1500 smoke machine, Artic Hayes smoke pellets, and a remote controller. Collection proceeds by recording a smoke-free baseline phase, remotely activating the smoke generator, and continuously recording the smoke diffusion process from sparse smoke to saturation. The first smoke-free frame is used as the ground-truth reference, while later frames are sampled as degraded images.

The benchmark’s alignment strategy is based on scene control rather than algorithmic post hoc registration. Windows and doors are sealed, lightweight objects are removed, and absolute stasis of all objects is maintained throughout recording. A three-stage data-enhancement pipeline then applies quality filter, feature augment, and label copy. The paper does not describe bounding boxes, semantic labels, or explicit smoke-severity metadata; the principal supervision is the paired smoke-free target image.

The benchmark is framed by a smoke imaging model that adapts the atmospheric-scattering form to early-stage fire scenes:

$\hat{I}(x)=\hat{J}(x)\cdot T(x)+S\cdot(1-T(x)),$

with transmission modeled by the Beer–Lambert form

$T(x)=e^{-\beta_e(x)\,d(x)}.$

The paper further states that $\beta_e=\beta_s+\beta_a\approx\beta_s$, and provides a Mie-scattering-based expression for $\beta_s(x)$. These formulations are used to motivate the claim that real smoke in fire scenes is physically and visually distinct from haze.

5. Surveillance SmokeBench: evaluation protocol and benchmarked results

The surveillance benchmark evaluates restoration quality using Y-channel PSNR, SSIM, and LPIPS. For reported experiments, all methods use 128 × 128 patches, other parameters remain at each method’s default configuration, and the hardware platform is an NVIDIA RTX 4090 GPU. The paper benchmarks FFA-Net, MPRNet, Restormer, Uformer, DehazeFormer, MB-TaylorFormer, X-Restormer, and MambaIR (Jin et al., 16 Sep 2025).

The strongest overall method is MB-TaylorFormer, which achieves 26.6283 average PSNR, 0.8570 average SSIM, and 0.1881 average LPIPS. Uformer is second-best overall in average PSNR (26.1251) and average SSIM (0.8517), while MambaIR is second-best overall in average LPIPS (0.1944). The benchmark reports substantial scene-to-scene variation, which the paper interprets as evidence that real surveillance desmoking is sensitive to smoke density, scene layout, lighting, and local occlusion structure.

Qualitative analysis is organized into thin smoke, medium-density smoke, and thick smoke cases. In thin smoke, several methods recover overall scene structure reasonably well, with DehazeFormer and Uformer noted for relatively sharp and natural results. In medium-density smoke, TaylorFormer and MambaIR are described as more robust. In thick smoke, however, the paper states that none of the evaluated methods can satisfactorily recover plausible structure and texture. Common failure modes include severe artifacts, oversmoothing, and poor structural recovery. This makes dense smoke the defining unresolved regime of the benchmark.

A common misconception addressed by the benchmark is that real smoke removal is simply a special case of dehazing. The paper rejects this equivalence on both physical and imaging grounds, arguing that smoke is combustion-generated, thermally driven, non-uniform, and often locally occlusive, whereas haze is more stable and homogeneous.

6. Position within the broader smoke-benchmark landscape

The two SmokeBench benchmarks occupy complementary positions within a broader ecosystem of smoke datasets and evaluation suites. The wildfire benchmark focuses on recognition and localization, especially under the failure modes of transparent, amorphous, and early-stage smoke. The surveillance benchmark focuses on paired restoration under real indoor early-stage fire conditions. Their coexistence shows that “smoke benchmarking” is not a single task family but a collection of technically distinct evaluation problems (Qi et al., 12 Dec 2025, Jin et al., 16 Sep 2025).

Related benchmark-oriented work further clarifies this task decomposition. "AusSmoke meets MultiNatSmoke: a fully-labelled diverse smoke segmentation dataset" introduces AusSmoke and MultiNatSmoke as smoke segmentation resources and reports improved performance and enhanced generalization across diverse geographical contexts (Li et al., 26 Apr 2026). "SmokeNet: Efficient Smoke Segmentation Leveraging Multiscale Convolutions and Multiview Attention Mechanisms" evaluates smoke segmentation on four datasets, including Quarry Smoke, and emphasizes efficiency–accuracy tradeoffs in deployment-oriented settings (Liu et al., 17 Feb 2025). "LSD3K: A Benchmark for Smoke Removal from Laparoscopic Surgery Images" defines a paired synthetic benchmark for surgical-image desmoking, showing that smoke-removal benchmarking also extends into medical imaging (Chang et al., 2024). "SmokeSeer: 3D Gaussian Splatting for Smoke Removal and Scene Reconstruction" adds a multimodal RGB-thermal, multiview benchmark setting for smoke removal and scene reconstruction under dynamic smoke (Jain et al., 22 Sep 2025).

This suggests a useful conceptual partition of the contemporary literature into at least four benchmark regimes: detection/localization, segmentation, single-image desmoking/restoration, and multiview multimodal visibility recovery. Within that partition, the two works named SmokeBench are notable because both are explicitly safety-critical, both are motivated by the inadequacy of synthetic or task-mismatched evaluation settings, and both expose persistent failure modes in current systems. The wildfire benchmark shows that MLLMs can classify smoke long before they can localize it reliably. The surveillance benchmark shows that strong restoration backbones still break down under thick smoke. Together, they establish SmokeBench as a benchmark label associated not with a single dataset, but with rigorous evaluation of difficult smoke perception tasks under operationally relevant conditions.