Masked CNN-Attention Autoencoder for AIRT

• The paper introduces a novel masked CNN-attention autoencoder that significantly enhances defect detection by reconstructing masked input sequences and achieving up to 30× training speedup.
• It integrates local convolutional feature extraction with multi-scale and multi-head self-attention, resulting in improved SNR, contrast, and defect localization.
• By combining reconstruction and PCA-based knowledge distillation losses, the method generalizes effectively across various materials and inspection scenarios.

The Masked CNN-Attention Autoencoder (AIRT-Masked-CAAE) is a neural architecture designed for self-supervised representation learning in active infrared thermography (AIRT) and generalized visual modeling tasks. The method combines convolutional feature extraction, multi-scale feature attention, and @@@@1@@@@ within a masked sequence autoencoding framework. By reconstructing masked regions of the input sequence, the model learns robust latent representations that enhance defect detection and visualization while significantly accelerating training and improving generalization across materials and inspection scenarios (Salah et al., 28 Dec 2025, Li et al., 2022).

1. Architectural Components

AIRT-Masked-CAAE consists of a structured dataflow with four principal stages:

1. Input Corruption (Masking + Noise): Given a raw thermal sequence $S^{(n)}\in\mathbb{R}^{N_t}$ for the $n$-th pixel, input corruption is performed as

$$\hat S^{(n)} = M\odot S^{(n)} + \varepsilon, \quad \varepsilon\sim\mathcal{N}(0,\sigma^2),$$

where $M\in\{0,1\}^{N_t}$ is a random binary mask and $\odot$ denotes element-wise multiplication.

2. CNN Head (Local Feature Extractor): The masked input is reshaped to a 2D “image” ($H\times W$) and processed through $L=3$ convolutional layers:

$$F^{(0)}\leftarrow \hat S^{(n)}, \quad F^{(\ell)} = \text{ReLU}\big(W^{(\ell)}*F^{(\ell-1)} + b^{(\ell)}\big),\quad \ell=1,2,3,$$

producing multi-scale feature maps $\{F^{(1)},F^{(2)},F^{(3)}\}$.

1. Attention Modules:
   • Multi-Level Feature Attention: Each feature map $F^{(m)}$ is re-weighted by a learned 1×1 convolution + sigmoid:

   $$\alpha_m = \sigma\big(W_m^{1\times1}*F^{(m)} + b_m\big)$$

   then fused as

   $F_{\rm att} = \sum_{m=1}^3 \alpha_m \odot F^{(m)}.$

• Multi-Head Self-Attention: $F_{\rm att}$ is flattened to $Z\in\mathbb{R}^{P\times d}$ ($P=H\cdot W$), projected to $Q,K,V$ for $H$ attention heads, and processed as

  $$\text{Attention}(Q,K,V) = \text{softmax}\bigg(\frac{QK^\top}{\sqrt{d_k}}\bigg)V,$$

  followed by a channel projection and nonlinearity.

1. Decoder and Latent Projection: The output is passed through an MLP to form a latent vector $\mathbf{z}_n\in\mathbb{R}^{32}$ and decoded by a symmetric convolutional block to reconstruct the original (unmasked) sequence.

This composition enables the model to capture both fine-grained spatial structure and long-range sequence dependencies, crucial for discriminating subtle thermal anomalies from background patterns (Salah et al., 28 Dec 2025).

2. Masked Sequence Autoencoding and Loss Functions

AIRT-Masked-CAAE is trained using a masked autoencoding regime, minimizing the following composite loss:

• Reconstruction Loss:

$$\mathcal{L}_{\rm rec} =\frac{1}{N}\sum_{i=1}^N \|\tilde S_i^{(n)} - S_i^{(n)}\|_2^2$$

• Knowledge Distillation (PCA) Loss:

For each sequence $S^{(n)}$, the PCA-projected embedding is $\mathbf{z}_n'$, and the knowledge-distillation loss penalizes angular misalignment:

$$\mathcal{L}_{\rm KD} = 1 - \frac{\langle \mathbf{z}_n, \mathbf{z}_n'\rangle}{\|\mathbf{z}_n\|_2 \|\mathbf{z}_n'\|_2}$$

• Composite Objective:

$$\mathcal{L}_{\rm total} = \mathcal{L}_{\rm rec} + \alpha\,\mathcal{L}_{\rm KD}$$

This dual-loss enforces accurate reconstruction while encouraging alignment of learned features with compact, interpretable subspaces, improving generalizability and downstream interpretability (Salah et al., 28 Dec 2025).

3. Training Process, Masking Strategy, and Efficiency

The training protocol employs aggressive masking and selective sampling:

• Masking:

In each mini-batch, a Bernoulli mask $M$ is sampled per sequence with typical masking ratio $50\%$.

• Sparse Sampling:

Only a small random subset ($B=128$) of pixel-sequences is processed per step, rather than the entire field, yielding a substantial reduction in per-epoch computational cost (~30×).

• Hyperparameters:
  • CNN head: three 3×3 layers, channel depths selected via Bayesian optimization.
  • Self-Attention: four heads ($H=4$), per-head dimension $d_k$.
  • Adam optimizer, initial learning rate $2\times10^{-5}$.

On a standard RTX 3060, masked training on $1000$ samples takes $36.7$ s versus $18.4$ min for full-set training, confirming a ~30× speedup (Salah et al., 28 Dec 2025).

4. Quantitative Performance and Ablation

AIRT-Masked-CAAE exhibits substantial performance improvements over both traditional and contemporary autoencoder-based methods, as substantiated by multiple evaluation metrics:

Material	Contrast (Raw→AIRT-Masked-CAAE)	SNR (Raw→AIRT-Masked-CAAE)	U-Net Defect IoU (Test)
CFRP	0.221 → 0.706 (+0.485)	22.83 dB → 45.26 dB (+22.4 dB)	0.759 → 0.836
PLA	0.296 → 0.641 (+0.345)	22.83 dB → 43.86 dB (+21.0 dB)	—
PVC	0.324 → 0.791 (+0.467)	27.05 dB → 49.88 dB (+22.8 dB)	—

Key ablation findings:

• Masking regime (vs. unmasked): improves robustness, boosts SNR and contrast, and increases IoU (PVC: 0.791 → 0.841 val, 0.803 → 0.836 test).
• Multi-level feature attention and self-attention each contribute ≈10–15 dB SNR improvement, attributed to deeper multi-scale and temporal context integration (Salah et al., 28 Dec 2025).

5. Comparison with Architecture-Agnostic Masked Image Modeling

The AIRT-Masked-CAAE paradigm generalizes the architectural principles of Architecture-Agnostic Masked Image Modeling (A²MIM) (Li et al., 2022) to the AIRT sequence domain. Both share these foundational details:

• Masking at Intermediate Feature Levels: Rather than input-level [MASK] tokens, learnable mask embeddings are injected at deep feature stages (e.g., conv4_x in ResNet-50).
• Joint CNN-Attention Decoder: Flattened masked features, with position embedding, are passed through multitier transformer blocks, supporting both spatial and channelwise context aggregation.
• Loss Functions: Spatial MSE and frequency-weighted MSE (focal-frequency) losses enforce both pixelwise accuracy and spectral fidelity.
• Ablations: Optimal performance at $60\%$ mask ratio; shallower decoders (L_dec=2) provide near-optimal transfer with substantially faster inference (Li et al., 2022).

This alignment demonstrates that the advantages of masked autoencoding for efficient and generalizable representation learning are robust across both visual (image) and thermographic (sequence) modalities.

6. Applications and Generalization

AIRT-Masked-CAAE excels in active infrared thermography for non-destructive testing (NDT), specifically for aerospace components made from PVC, CFRP, and PLA. The method:

• Enhances defect contrast and SNR, yielding clear visibility of sub-surface flaws under varying inspection and depth conditions.
• Learns generalizable representations, as evidenced by downstream U-Net segmentation, where test IoU improves from baseline 0.759 (raw) to 0.836 (AIRT-Masked-CAAE).
• Substantially reduces computational resources, enabling rapid model development and deployment in high-throughput NDT pipelines (Salah et al., 28 Dec 2025).

A plausible implication is that this framework, owing to its architecture-agnostic design, can be ported to broader domains including imaging, medical diagnostics, and generic sequence modeling where robust and efficient representation learning from partially observed data is critical (Li et al., 2022).

7. Significance and Limitations

AIRT-Masked-CAAE demonstrates that coupling convolutional local pattern extraction, multi-scale attention-based feature fusion, and masked autoencoding leads to:

• Lean models with compact latent codes (size 32),
• Substantial improvements in defect localization and quantitative imaging metrics,
• Convergence speedups ($\sim 30\times$).

While masking and attention blocks are shown to substantially uplift SNR and defect contrast, the specific contributions of each architectural element are not always independently quantified. This suggests that further ablations—isolating the self-attention, multi-level fusion, and masking regimes—could further clarify attributable gains.

In summary, the Masked CNN-Attention Autoencoder establishes an efficient, generalizable pattern for masked self-supervised learning across structured sequence and image data, with proven empirical gains in both defect analysis and computational efficiency (Salah et al., 28 Dec 2025, Li et al., 2022).