Two-Phase Uncertainty-Aware Shifted Windows Transformer

• The paper introduces a two-phase training regimen that decouples classification from variance learning, leading to calibrated uncertainty outputs in smoke detection.
• It utilizes a shared Swin Transformer backbone with alternating window and shifted-window self-attention to efficiently fuse appearance and motion cues.
• The framework employs optical flow-guided feature fusion and Monte Carlo sampling, achieving robust detection performance under complex visual noise.

The Two-Phase Uncertainty-Aware Shifted Windows Transformer denotes a transformer-based smoke-detection framework that combines optical flow-guided feature fusion, a shared Swin Transformer backbone, and explicit predictive-uncertainty estimation under a two-phase training schedule (Mahala et al., 20 Aug 2025). Its architectural substrate is the Swin Transformer, a hierarchical vision transformer in which computation alternates between regular window-based self-attention and shifted-window self-attention, thereby preserving local-attention efficiency while enabling cross-window information flow (Liu et al., 2021). In this sense, the designation “two-phase” combines two distinct mechanisms: the Swin backbone’s alternating W-MSA $\rightarrow$ SW-MSA computation and the detector’s Phase I $\rightarrow$ Phase II optimization regimen for accuracy learning and uncertainty calibration.

1. Conceptual definition and scope

TP-UAST was introduced for reliable smoke detection from monocular RGB video frames, a safety-critical setting in which smoke exhibits complex spatiotemporal dynamics and can resemble fog, steam, dust, and clouds. The framework is motivated by the need for not only accurate smoke/non-smoke discrimination but also calibrated confidence under illumination variability, turbulent plume motion, and environmental noise. Its pipeline fuses appearance and motion cues, then predicts a class score together with uncertainty-related quantities intended to support risk-aware decisions (Mahala et al., 20 Aug 2025).

A central point of clarification is that uncertainty awareness is not part of the original Swin Transformer. Swin is a deterministic hierarchical backbone with shifted-window self-attention and relative position bias; it does not include probabilistic attention, uncertainty estimation, or explicit aleatoric/epistemic modeling. TP-UAST should therefore be understood as a task-specific extension built on Swin rather than as a property of Swin itself (Liu et al., 2021).

Another recurrent source of ambiguity concerns the word “phase.” In TP-UAST, it refers both to the two-block alternation of regular and shifted windows inherited from Swin and to the two-phase learning regimen used to decouple detection performance from variance learning. These are separate constructs operating at different levels of the system: one architectural, one optimization-related (Mahala et al., 20 Aug 2025).

2. Backbone architecture and shifted-window computation

The backbone is a shared Swin Transformer backbone applied independently to two input branches: an RGB image and a segmented color-encoded optical-flow map. Patch embedding uses non-overlapping $4\times 4$ patches, a linear embedding dimension of 96, and reduces the input resolution by $4\times$. The encoder has four hierarchical stages with patch merging between stages, while the decoder contains five sequential decoder blocks that upsample by bilinear interpolation, concatenate skip features from the encoder, and apply two conv+ReLU layers per block to restore spatial detail (Mahala et al., 20 Aug 2025).

Stage	Blocks / heads / window	Feature dim
Stage 1	2 blocks, 4 heads, $7\times 7$	96
Stage 2	2 blocks, 4 heads, $7\times 7$	192
Stage 3	4 blocks, 8 heads, $7\times 7$	384
Stage 4	2 blocks, 8 heads, $7\times 7$	768

Within each stage, Swin computes self-attention in non-overlapping local windows of size $M\times M$, with $M=7$ by default. In a regular W-MSA block, tokens attend only within their own window. In the subsequent SW-MSA block, the feature map is cyclically shifted by $\rightarrow$0 before window partitioning, and an attention mask prevents invalid interactions between tokens that originated in different windows before shifting. This produces cross-window communication over successive blocks without reverting to global attention (Mahala et al., 20 Aug 2025).

The Swin formulation is hierarchical: patch merging concatenates each $\rightarrow$1 token group, halves spatial resolution, and increases channel width at stage transitions, producing multi-resolution features that can be interpreted at scales such as $\rightarrow$2, $\rightarrow$3, $\rightarrow$4, and $\rightarrow$5. The original Swin paper characterizes this design as having linear computational complexity with respect to image size, in contrast to global MHSA, and reports strong backbone performance across classification and dense prediction tasks, including 87.3 top-1 accuracy on ImageNet-1K, 58.7 box AP and 51.1 mask AP on COCO test-dev, and 53.5 mIoU on ADE20K val for Swin-L configurations (Liu et al., 2021).

3. Optical-flow-guided feature fusion

Before transformer inference, TP-UAST constructs an explicit motion representation using a four-color-theorem-inspired dual-phase level-set fractional-order variational model (FCDLe-FOV) for dense optical flow. This model uses an $\rightarrow$6 data term, Marchaud fractional derivatives of order $\rightarrow$7, and dual-phase level-set boundary regularization. The reported purpose is to preserve motion discontinuities and maintain boundary-aware segmentation of moving smoke structures (Mahala et al., 20 Aug 2025).

The optical flow is then color-encoded and segmented into smoke versus background by a Gaussian Mixture Model (GMM). A binary mask suppresses non-smoke motion, and the segmented motion map is obtained by multiplying that mask with the color-encoded optical-flow image. The resulting paired inputs to TP-UAST are therefore two three-channel tensors:

• Appearance branch: RGB image
• Motion branch: segmented color-encoded optical-flow map

The article’s technical description states that the two branches pass through the same backbone-decoder with shared weights. The uncertainty-aware head consumes the final feature maps from each branch in parallel. The paper does not state an explicit cross-branch concatenation mechanism; instead, it specifies that the head operates on final features from each branch and that random transformations used for Monte Carlo epistemic sampling are shared across both inputs to preserve correspondence (Mahala et al., 20 Aug 2025).

The GMM stage is described as using a $\rightarrow$8-component GMM over pixel intensities or color vectors, with $\rightarrow$9 selected empirically; the paper does not state the value of $4\times 4$0 explicitly. The resulting dataset pairs RGB frames with segmented optical-flow maps and binary masks, is released on Kaggle, and uses images resized to $4\times 4$1 for TP-UAST. The paper does not report dataset size, class balance, or train/validation/test split details, but states that experiments cover diverse scenes including forests, roads, industrial settings, and crowds under stationary and non-stationary illumination (Mahala et al., 20 Aug 2025).

4. Two-phase learning and uncertainty modeling

The prediction module is uncertainty-aware at the head level. For each branch’s final feature map, the architecture applies global average pooling, then FC with dropout, followed by three parallel FC heads producing a primary classification output $4\times 4$2, a mean output $4\times 4$3, and a variance output $4\times 4$4, with the variance constrained to be non-negative by an activation that is not explicitly specified in the paper (Mahala et al., 20 Aug 2025).

The training schedule is explicitly divided into two phases. Phase I optimizes the classification head and mean-prediction head while the variance head is frozen. Supervision uses a composite binary cross-entropy objective on both $4\times 4$5 and $4\times 4$6. The reported optimizer is Adam, with initial learning rate $4\times 4$7 and polynomial decay, and training continues until the epoch-averaged loss $4\times 4$8. Phase II freezes all weights except the variance sub-head, estimates epistemic variability by Monte Carlo sampling under random flips and rotations with a shared seed across the paired inputs, computes empirical mean and variance from the sampled predictions, and trains the variance head using an $4\times 4$9 loss against the Monte Carlo variance estimate (Mahala et al., 20 Aug 2025).

The uncertainty model decomposes predictive uncertainty into aleatoric and epistemic components:

$4\times$0

In the reported implementation, aleatoric uncertainty is learned directly through the variance head output $4\times$1, whereas epistemic uncertainty is approximated by the variance induced by stochastic test-time transformations. The paper explicitly notes that it does not use Bayesian weight posteriors, ensembles, or MC dropout; epistemic uncertainty arises from test-time augmentation rather than posterior inference over model parameters (Mahala et al., 20 Aug 2025).

At inference time, the paper reports both PP $4\times$2 and a plausibility confidence

$4\times$3

with

$4\times$4

where $4\times$5 is the logit or score draw under the Gaussian interpretation used for plausibility visualization. This construction is used to flag ambiguous cases via low-confidence tails in the plausibility distribution (Mahala et al., 20 Aug 2025).

5. Reported empirical behavior

The paper reports results at two coupled levels: the optical-flow front end and the TP-UAST detector itself. For optical flow, FCDLe-FOV is stated to outperform NFVLS, FS-FOV, NLW, HS-NE on Middlebury1, Middlebury2, and Sintel1 with respect to AAE, AEPE, and AENG. Example values given are Middlebury1: AAE=0.173, AEPE=0.877, AENG=1.875; Middlebury2: AAE=0.176, AEPE=2.330, AENG=2.461; and Sintel1: AAE=0.101, AEPE=0.393, AENG=1.079 (Mahala et al., 20 Aug 2025).

Under noise perturbations, the paper reports SSIM on color maps for Gaussian, Poisson, and salt-and-pepper noise. Example values include Smoke1: 0.87, 0.95, 0.97; Smoke4: 0.91, 0.97, 0.98; Non-Smoke1: 0.97, 0.98, 0.99; and Non-Smoke4: 0.95, 0.98, 0.99. These figures are presented as evidence that the color-encoded flow representation remains stable under multiple noise types (Mahala et al., 20 Aug 2025).

For smoke detection, the reported validation-set discriminative metrics are Accuracy 1.00, Precision 1.00, Recall 1.00, and F1 1.00, and the confusion matrix is said to show zero false positives and zero false negatives. The strongest listed competitors remain below TP-UAST: MFFNet with Accuracy 0.98, Precision 0.94; Safarov et al. with Precision 0.96, Recall 0.97; an energy-efficient model with Accuracy/Precision/Recall/F1 all 0.95; and YOLOv2 with Accuracy 0.96, Precision 0.97, Recall 0.95, F1 0.97 (Mahala et al., 20 Aug 2025).

The uncertainty outputs are evaluated by several calibration-oriented diagnostics. The paper reports a reliability diagram with 10 bins, a calibration curve nearly on the diagonal, and Expected Calibration Error (ECE) $4\times$6. The uncertainty histogram is described as sharply peaked at $4\times$7, with a tail extending to $4\times$8; larger predicted $4\times$9 correlates with larger absolute error; both classes have median $7\times 7$0, but the smoke class exhibits heavier upper quartile and outliers. The standardized score distribution shows $7\times 7$1 within $7\times 7$2 and $7\times 7$3 within $7\times 7$4, while the plausibility confidence satisfies $7\times 7$5 samples with $7\times 7$6 (Mahala et al., 20 Aug 2025).

The paper does not report explicit ablations isolating shifted versus non-shifted windows, two-phase versus single-phase training, or motion fusion versus appearance-only inputs. It nevertheless states three qualitative conclusions: shifted windows are expected, following Swin, to improve cross-window communication without global attention cost; two-phase training demonstrably improves calibration with ECE $7\times 7$7 and well-behaved $7\times 7$8 distributions; and optical-flow fusion qualitatively suppresses background motion while emphasizing smoke regions (Mahala et al., 20 Aug 2025).

6. Limitations, applications, and relation to broader transformer research

The primary application domain is reliable early smoke detection in safety-critical scenarios. The paper identifies three use-case categories: surveillance and early fire detection in public spaces, industrial safety monitoring such as factories and refineries, and autonomous environmental monitoring in forests and construction sites (Mahala et al., 20 Aug 2025).

Its limitations are correspondingly explicit. The method relies on monocular imagery and does not use depth or multispectral cues. It may fail on thin, wispy smoke under complex sky/background or low-light clutter. It also provides no reported real-time throughput, leaving embedded deployment constraints unresolved. Future work is directed toward real-time embedded implementation on limited-compute platforms and multi-class hazard detection involving fire, steam, and dust together with richer multimodal fusion (Mahala et al., 20 Aug 2025).

In relation to the broader transformer literature, TP-UAST inherits the key advantages that made Swin a general-purpose vision backbone: hierarchical feature construction, local self-attention with shifted-window cross-window interaction, and computational scaling that is substantially smaller than global MHSA. A common misconception is that the “uncertainty-aware shifted windows transformer” is simply Swin under a different name. More precisely, TP-UAST is a domain-specific extension of Swin in which the backbone remains a shifted-window transformer, while uncertainty awareness is introduced through an additional prediction head, Monte Carlo test-time augmentation, and a two-phase optimization schedule (Liu et al., 2021).