Interventional Multi-Scale Encoder (IMSE)

• IMSE is a module in the CausalFSFG framework that applies sample-level causal interventions to counteract confounding effects in few-shot categorization.
• It employs a multi-scale feature extraction strategy with learned weighting and front-door adjustment to dynamically emphasize discriminative features.
• Empirical evaluations on architectures like Conv-4 and ResNet-12 show significant accuracy improvements in tasks such as CUB, Dogs, and Cars.

The Interventional Multi-Scale Encoder (IMSE) is a core module within the CausalFSFG architecture for few-shot fine-grained visual categorization (FS-FGVC) that operationalizes sample-level causal interventions to mitigate confounding effects inherent to few-shot episode construction. IMSE systematically assigns learned weights to multi-scale feature maps, enabling the network to dynamically emphasize discriminative features from different convolutional depths for each input episode. This approach is grounded in a structural causal model (SCM) and leverages front-door adjustment to estimate the interventional distribution $P(Y\,|\,do(X))$, distinguishing itself from conventional strategies that are limited to observational correlations $P(Y\,|\,X)$. IMSE integrates multi-scale attentional mechanisms and learned weighting schemes to enhance robustness and generalization in FS-FGVC settings by breaking the spurious correlations between support selection and class predictions (Yang et al., 25 Dec 2025).

1. Structural Causal Model Motivation and Front-Door Formulation

Within the CausalFSFG method, IMSE is motivated by the SCM that models fine-grained visual categorization as a process wherein both the support/query sample selection ($O$) and the intrinsic fine-grained nature of the data ($I$) act as unobserved confounders, thereby introducing spurious correlations between the observed input image $X$ and the label $Y$. The SCM is represented as:

• $D \to I \to Y$
• $D \to O \to X \to M \to Y$
• $X \leftarrow C \rightarrow Y$, where $C = \{O, I\}$

The primary inferential goal is to recover the interventional distribution $P(Y\,|\,do(X))$ rather than the conventionally observed $P(Y\,|\,X)$. Using the front-door criterion [Pearl, 2016], the estimation is decomposed as:

$$P(Y\,|\,do(X)) = \sum_m P(M\,|\,do(X))\,P(Y\,|\,do(M)) = \sum_m P(M\,|\,X) \sum_x P(Y\,|\,M,X) P(X)$$

IMSE realizes the first term $P(M\,|\,X)$, serving as a sample-level intervention on the feature extraction process, thereby facilitating identification of true causal relationships between inputs and subcategories in the presence of confounding (Yang et al., 25 Dec 2025).

2. Multi-Scale Feature Extraction and Weighted Aggregation

IMSE is architected to exploit feature maps at multiple scales, rather than extracting discriminative cues solely from the deepest convolutional layer. For a standard Conv-4 backbone, the feature maps are:

• $M_1$: $64{\times}20{\times}20$
• $M_2$: $64{\times}10{\times}10$
• $M_3$, $M_4$: $64{\times}5{\times}5$

For ResNet-12:

• $M_i$ analogous, $640$ channels in the final block.

Each $M_i$ undergoes a dimension alignment via a $1{\times}1$ convolution yielding $[\mathbf{F}_i, \mathbf{I}_i]$, where $$\mathbf{F}_i \in \mathbb{R}^{\Gamma \times H_i \times W_i}$$ is the principal feature tensor and $$\mathbf{I}_i \in \mathbb{R}^{1 \times H_i \times W_i}$$ serves as the "interventional token" for scale $i$. After concatenating and flattening spatial dimensions, a single-layer Transformer self-attention is applied to $[\mathbf{F}_i; \mathbf{I}_i]$, resulting in updated feature and token representations per scale.

Each $\mathbf{I}_i$ is pooled to obtain scalar $\alpha_i$, then normalized with a Softmax:

$$\alpha' = \operatorname{Softmax}([\alpha_1, \alpha_2, \alpha_3, \alpha_4])$$

Each scale's features are reweighted: $\mathbf{F}_i' = \alpha_i' \cdot \mathbf{F}_i$. The four feature tensors are hierarchically fused with a "reversed" Feature Pyramid Network (FPN) schema:

$$F_I = F_4' + \operatorname{MaxPool}(F_3' + \operatorname{MaxPool}(F_2' + \operatorname{MaxPool}(F_1')))$$

yielding a unified intervened feature map $F_I \in \mathbb{R}^{\Gamma \times 5 \times 5}$ (Yang et al., 25 Dec 2025).

3. Sample-Level Causal Intervention Mechanism

IMSE operationalizes sample-level intervention by enabling episode-adaptive, learned attention over scale-specific features. The scalar weights $\alpha'_i$ correspond to the estimated discriminativeness of each scale under the current support/query sample selection, thus realizing $P(M\,|\,do(X))$ as an explicit, learned conditional distribution. The architecture avoids rigid, pre-set preferences (such as defaulting to deepest features), instead allowing the model to shift adaptive focus across scales to counteract sampling-induced biases.

The interventional tokens $\mathbf{I}_i$ are directly trained (via backpropagation from the cross-entropy loss) to encode the discriminative relevance of each scale, conditioned on both individual samples and the support set context. This procedure is designed to disrupt spurious support-to-label correlations induced by random few-shot sampling, addressing confounding introduced via $C = \{O, I\}$ in the SCM.

4. Mathematical Implementation and Optimization

• Dimension Alignment: $[\mathbf{F}_i, \mathbf{I}_i] = \mathrm{DA}(M_i)$, where $\mathrm{DA}$ denotes a $1 \times 1$ convolution to $\Gamma + 1$ channels.
• Attention Update: $$[\mathbf{F}_i, \mathbf{I}_i] \gets \mathrm{Softmax}\left(\frac{Q_i K_i^T}{\sqrt{\Gamma}} \right)V_i$$, with $Q_i, K_i, V_i$ as learned projections.
• Scale Importance: $\alpha_i = \operatorname{Pool}(\mathbf{I}_i)$, then $\alpha'$ via Softmax.
• Weighted Fusion: $\mathbf{F}_i' = \alpha_i' \otimes \mathbf{F}_i$; $F_I$ merged as above.
• Loss: Classification uses cross-entropy:

$$p_{ij} = \exp(-d_{ij}) / \sum_{n=1}^N \exp(-d_{in}), \quad d_{ij} = \|q_i^j - V_j\|_2$$

$\mathcal{L} = -\sum_i \log p_{i, y_i}$

• No separate regularization or reconstruction losses are applied to IMSE, as its learning is governed by the end-to-end objective; however, its architectural placement guarantees that its outputs are involved in estimating $P(Y|do(X))$ as per the SCM.

5. System Design and Practical Considerations

Backbone Variants:

• Conv-4: $[\mathrm{M}_1, ..., \mathrm{M}_4]$ sizes $$[64{\times}20{\times}20,\,64{\times}10{\times}10,\,64{\times}5{\times}5,\,64{\times}5{\times}5]$$
• ResNet-12: [640 channels $\ldots$]

Embedding Dimension: $\Gamma = 128$ for Conv-4, $256$ for ResNet-12.

Meta-Training Protocols:

• Conv-4: 30-way 5-shot, also 5-way 5-shot.
• ResNet-12: 15-way 5-shot, also 5-way 5-shot.
• Input: $84 \times 84$. Data augmentation: random crop, flip, color jitter (train); center crop (test).

Optimization: Stochastic gradient descent with Nesterov momentum $0.9$, weight decay $3\,\times\,10^{-4}$, initial learning rate $0.1$, decayed by $1/20$ at epochs 400 and 600, for a total of 800 epochs.

Computational Footprint: Conv-4 with IMSE is approximately $168$K parameters and $66$ GFLOPs (compared to $150$K/$60$ GFLOPs for Bi-FRN).

6. Empirical Evaluation and Ablative Insights

Ablation studies on CUB-200-2011 and ResNet-12 backbones demonstrate the efficacy of IMSE:

Condition	1-shot Accuracy	1-shot Δ	5-shot Accuracy	5-shot Δ
Baseline	64.82%	-	85.74%	-
+IMSE only	77.13%	+12.31	88.24%	+2.50
+IMFR only	73.51%	+8.69	88.75%	+3.01
IMSE + IMFR (full)	81.94%	+17.12	93.33%	+7.59

Analogous improvements are reported with ResNet-12 (full model: 1-shot 87.05%, +6.03; 5-shot 95.26%, +3.33). These results indicate that IMSE alone provides substantial bias reduction and generalization improvement over baselines and, when combined with the interventional masked feature reconstruction (IMFR), achieves state-of-the-art results on CUB, Dogs, and Cars datasets (Yang et al., 25 Dec 2025).

7. Context, Implications, and Related Work

IMSE addresses a critical limitation in previous FS-FGVC pipelines: the failure to account for confounding induced by support/query sampling and the fine-grained nature of dataset structure. By leveraging causal inference, specifically the front-door adjustment within an SCM, IMSE transitions the field from observational post-hoc feature enrichment to explicit estimation of $P(Y|do(X))$. A plausible implication is that such interventional encoding mechanisms could generalize to other domains with selection biases or unobserved confounders, as well as inform principled approaches to meta-learning beyond vision. This work aligns with the trend of integrating causal modeling in representation learning and meta-learning, specifically referencing the principles established by Pearl [Pearl, 2016].

Reference:

CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective (Yang et al., 25 Dec 2025)