---
title: Few-shot Multispectral Object Detection
url: https://www.emergentmind.com/topics/few-shot-multispectral-object-detection-fsmod
type: topic
---

# Few-shot Multispectral Object Detection

Few-shot multispectral object detection (FSMOD) is the problem of detecting object categories from paired visible-spectrum and thermal/infrared images when only a handful of labeled examples are available for the target classes. In the RGB + IR setting emphasized by recent work, a detector receives aligned or co-registered visible and thermal images and must learn to recognize base classes with abundant annotations and novel classes with only \(K\) examples per class. The task combines few-shot generalization with cross-modal fusion, so the detector must exploit the complementary strengths of visible and thermal imagery without overfitting to one modality or to the few available examples. Recent formulations characterize FSMOD as a distinct regime rather than a straightforward extension of standard few-shot object detection, because the model must simultaneously address a modality gap, spatial misalignment, severe label scarcity, and strong illumination- and environment-dependent appearance changes [2509.20905].

## 1. Problem formulation and task structure

FSMOD assumes paired multispectral inputs, typically RGB and infrared or thermal images, together with a class split
\[
C = C_{\text{base}} \cup C_{\text{novel}}, \qquad C_{\text{base}} \cap C_{\text{novel}} = \emptyset .
\]
Base classes have abundant annotations, whereas novel classes are available only through \(K\)-shot support examples. In the setting described for FSMODNet, the detector is trained first on \(C_{\text{base}}\) and then adapted to both base and novel classes using only \(K\) labeled examples per novel class under an episodic protocol [2509.20905].

The distinctive feature of FSMOD is that the support and query data are multispectral. Visible images provide rich appearance and texture, while thermal images remain informative under low light, fog, occlusion, or other adverse conditions. This complementarity is beneficial but does not remove the few-shot constraint. A class may be easy to detect in IR in one scene but harder in RGB, or vice versa, so the detector must learn how to exploit complementary cues without assuming that either modality is uniformly dominant [2509.20905].

A broader multimodal perspective appears in multi-modal few-shot object detection, where support images are combined with semantic information from a pretrained language model. In that setting, the goal is to detect novel objects by exploiting both few-shot visual examples and semantic information from a pretrained language model, yielding a true multi-modal FSOD formulation rather than purely visual FSOD or purely zero-shot object detection [2204.07841]. This line of work is not itself multispectral, but it is relevant because it formalizes how few-shot detection can be conditioned on more than one information source.

## 2. Central difficulties in few-shot multispectral detection

The core challenges identified for FSMOD are intertwined rather than independent. The first is the modality gap between RGB and IR feature distributions. Even when the images are aligned or co-registered, the two modalities differ strongly in appearance and statistics, so fusion cannot be reduced to simple feature addition without risking information loss or spurious correspondences [2509.20905].

The second challenge is misalignment across modalities. RGB and thermal images are often not perfectly aligned, and they frequently emphasize different object evidence. A detector must therefore align features spatially across modalities while avoiding the assumption of exact pixel-level correspondence. This requirement becomes more acute in the few-shot regime because robust alignment and fusion are difficult to learn from extremely limited supervision [2509.20905].

The third challenge is low-data instability. Few-shot learning makes robust fusion difficult to learn, and monomodal models can occasionally outperform fusion in very low-shot configurations, especially when one modality is already highly informative. On FLIR, for example, the paper notes that IR can be highly informative, and this can make learning a robust fused representation under very sparse supervision particularly difficult. This directly counters a common misconception that adding modalities is automatically beneficial in the lowest-shot regime; the reported evidence indicates that multimodal fusion becomes clearly beneficial as shot count increases, rather than uniformly across all shot settings [2509.20905].

A related challenge identified in VLM-based FSMOD is the unlabeled object problem. In few-shot settings, many objects in the image are not annotated, and standard training may wrongly treat them as background, creating false-negative supervision. This point extends the discussion of scarcity beyond labeled support size to the structure of the supervision itself [2512.15971].

## 3. FSMODNet: architecture and learning pipeline

FSMODNet is a modular few-shot detector built around a prototype-based meta-learning pipeline and a cross-modality fusion block. Its overall structure follows a two-branch design in which both support and query images are encoded separately for RGB and IR using shared modality-specific backbones. The default backbone is ResNet-50 for both visible and infrared branches, with six-layer encoder and decoder stacks. Features from each modality are processed by a multi-stage fusion module, passed to a meta-learning support-query mechanism, and finally sent to a DINO-DETR detection head [2509.20905].

The architecture is explicitly episodic. During training, a support set provides class prototypes from a small number of annotated examples, and a query image is detected conditioned on those prototypes. This makes the method compatible with few-shot evaluation while preserving a detection pipeline that can adapt to novel classes. On top of the fused multispectral representation, support annotations are turned into class prototypes by applying RoIAlign to each ground-truth support box and average pooling the resulting features, yielding a class prototype matrix \(S \in \mathbb{R}^{C \times D}\), where \(C\) is the number of support classes [2509.20905].

The support-query aggregation module is derived from Meta-DETR. Query features \(F_q \in \mathbb{R}^{HW \times D}\), support prototypes \(S\), and task encodings \(T \in \mathbb{R}^{C \times D}\) are projected into a shared space and combined using attention:
\[
A = \text{Softmax}\!\left( \frac{(F_q W)(S W)^\top}{\sqrt{d}} \right).
\]
From this, the model computes
\[
Q_F = A \odot \sigma(S), \qquad Q_E = A T,
\]
and then forms the aggregated class-aware representation
\[
F_{\text{CAM}} = \text{FFN}(Q_F + Q_E).
\]
This is described as a class-agnostic-to-class-specific transformation: the support prototype filters the query features, while the task encoding injects episode-level class structure. The final detection stage uses DINO-DETR to produce boxes and class predictions. In addition, the model employs a cosine-similarity-based cross-entropy loss, following Meta R-CNN, to supervise prototype classification and encourage alignment between prototype embeddings and query features [2509.20905].

Training is organized into two stages. In the base meta-learning stage, the model is trained on \(C_{\text{base}}\) with abundant data to learn transferable representations. In the few-shot fine-tuning stage, it adapts to both base and novel classes using only \(K\) labeled examples per novel class. To reduce catastrophic forgetting, base-class samples are included during fine-tuning under the same episodic regime. At inference time, class prototypes can be precomputed as average prototypes to reduce variance and then used for detection on each RGB/IR image pair [2509.20905].

## 4. Cross-modality feature integration in FSMODNet

The central technical contribution of FSMODNet is the cross-modality feature integration block, which is designed to fuse visible and thermal information more carefully than naïve concatenation or addition. Fusion proceeds in three stages: intra-spectrum refinement via Neighborhood Attention, bidirectional Cross-Deformable Attention, and final channel fusion by concatenation and pointwise convolution [2509.20905].

In the first stage, each modality undergoes local contextual refinement independently. For a feature map \(F_{RGB} \in \mathbb{R}^{D \times H \times W}\), the module computes
\[
Q = W_Q * F_{RGB}, \quad K = W_K * F_{RGB}, \quad V = W_V * F_{RGB},
\]
and restricts attention to a local neighborhood \(\mathcal{N}_{(i,j)}\). The attention score and normalized weights are
\[
\alpha_{(i,j),(u,v)} = \frac{Q_{i,j} \cdot K_{u,v}}{\sqrt{D}},
\]
\[
A_{i,j,u,v} = \frac{\exp(\alpha_{(i,j),(u,v)})}{\sum\limits_{(u',v') \in \mathcal{N}_{(i,j)}} \exp(\alpha_{(i,j),(u',v')})},
\]
and the refined feature at \((i,j)\) is
\[
F'_{RGB,i,j} = \sum_{(u,v) \in \mathcal{N}_{(i,j)}} A_{i,j,u,v} \cdot V_{u,v}.
\]
The same local attention is applied independently to RGB and IR, yielding \(F'_{RGB}\) and \(F'_{IR}\). The stated function of this step is to denoise and contextualize each modality before cross-modal fusion [2509.20905].

The second stage is bidirectional Cross-Deformable Attention. Queries from one modality attend to keys and values sampled from the other modality using predicted offsets. For one direction, the offsets are produced by a lightweight network \(\theta_{\text{offset}}\) using depth-wise convolution, normalization, and GELU, followed by a \(1 \times 1\) convolution:
\[
\Delta p = s \cdot \tanh(\theta_{\text{offset}}(u_p)),
\]
where \(u_p = W_u F'_{IR}\). Deformed samples are then obtained by bilinear interpolation,
\[
\tilde{x} = \phi(F'_{IR,\,p+\Delta p}), \quad \tilde{k} = W_k \cdot \tilde{x}, \quad \tilde{v} = W_v \cdot \tilde{x},
\]
and the query is computed from the other modality:
\[
q = W_q \cdot F'_{RGB}.
\]
Attention is then computed in scaled dot-product form,
\[
A = \operatorname{Softmax}\!\left(\exp(q \cdot k / \sqrt{D})\right),
\]
followed by the feature update
\[
F''_{RGB} = F'_{RGB} + \text{ConvFFN}(F'_{RGB} + A \cdot v).
\]
An analogous update is applied in the reverse direction, with IR querying RGB. This design is intended to solve two problems at once: spatial alignment across modalities and adaptive focus on informative cross-spectral regions rather than exact correspondence [2509.20905].

The final fused map is
\[
F_q = \text{Conv}_{1 \times 1}(\text{cat}(F''_{IR}, F''_{RGB})).
\]
The paper’s broader claim is that few-shot multispectral detection should not rely on naïve fusion; it requires local refinement within each modality, deformable cross-modal alignment, and prototype-conditioned detection. The ablations reported for concatenation, addition, and the prior CMI module support the claim that the CDA-based fusion is best overall, especially at higher shot counts [2509.20905].

## 5. Datasets, protocols, and empirical behavior

FSMODNet is evaluated on two public datasets: FLIR and M\(^3\)FD. FLIR contains aligned RGB-thermal pairs with three classes, and the authors define three base/novel splits such as \((\text{bicycle}, \text{car} / \text{people})\), evaluating novel-class detection. M\(^3\)FD contains co-registered RGB-thermal pairs under day, night, overcast, and challenging conditions, with six classes; three splits are created by selecting the least frequent classes as novel. The main few-shot settings are 5-shot, 10-shot, and 30-shot, and the metric is nAP50 on novel classes [2509.20905].

The baselines include DAMSDet+TFA and CAFF-DINO+TFA, described as baselines built by combining state-of-the-art multispectral detectors with TFA, together with monospectral Meta-DETR baselines for RGB and IR separately. On FLIR, FSMODNet with RGB+IR reaches 30.67/36.56/49.70 nAP50 on Split 1 for 5/10/30 shots and beats DAMSDet+TFA by up to 11.09 points at 5-shot in Split 1. On M\(^3\)FD, RGB+IR achieves 29.43/40.11/47.32 nAP50 on Split 1 and 37.02/47.22/52.18 on Split 2, with gains over DAMSDet+TFA reaching up to 6.74 points at 5-shot [2509.20905].

These results are accompanied by two important qualifications. First, monomodal models can occasionally outperform fusion in very low-shot configurations, especially when one modality, such as IR on FLIR, is already highly informative. Second, as shot count increases, multimodal fusion becomes clearly beneficial. The empirical picture is therefore not that multispectral fusion is uniformly superior, but that its advantages become more stable as the amount of support supervision increases [2509.20905].

The same datasets are used in a later VLM-based study, which explicitly adopts the FSMODNet few-shot benchmark protocol. That work reports support sets with \(k \in \{5, 10\}\) annotated instances per class, evaluates on the full test set, and reports extended 30-shot results in the supplement. It further specifies FLIR as an aligned RGB-thermal dataset with 4,129 training pairs and 1,013 test pairs, and M3FD as containing 4,200 co-registered RGB-thermal pairs, with the split from DAMSDet adopted for fairness [2512.15971].

## 6. Relation to multimodal FSOD and VLM-based FSMOD

Although FSMOD is specifically concerned with visible and thermal modalities, it sits within a broader family of few-shot detection methods that combine multiple sources of information. MM-FSOD is a multi-modal few-shot object detection framework that combines few-shot visual examples with class semantic information while avoiding the requirement of fine-tuning to add novel classes and avoiding the requirement of human-provided class names for novel classes. Its core design combines metric-based meta-learning for few-shot visual recognition and prompt-based learning for language-based semantic recognition [2204.07841].

The detector in MM-FSOD is built on a siamese Faster R-CNN-style framework, and its central module is Multi-modal Prototype Generation. The visual prototype is
\[
p_i^v = \frac{1}{K}\sum_{j=1}^{K} F^v(I_i^j),
\]
while the semantic branch uses a pretrained language model \(F^t\) to encode soft prompts,
\[
\bar{p}_i^S = F^t([s_i^S]), \qquad \bar{p}_i^T = F^t([s_i^T, E^t(t_i)]).
\]
These are fused with the visual prototype by projection and elementwise addition,
\[
p_i^S = \mathbf{FC}(\bar{p}_i^S) + p_i^v, \qquad
p_i^T = \mathbf{FC}(\bar{p}_i^T) + p_i^v.
\]
A student-teacher prompt generator and a distillation loss
\[
L_{KD} = \frac{1}{N}\sum_i \lVert \bar{p}_i^S - \bar{p}_i^T \rVert_2
\]
allow the student model to operate without class-name priors at test time [2204.07841]. The paper is not specifically about multispectral object detection, but it explicitly states that its meta-learning + prompt-generation + prototype-fusion design is transferable to a multispectral FSMOD setting.

A later development makes that connection explicit by exploring VLMs for FSMOD. “From Words to Wavelengths” adapts Grounding DINO and YOLO-World to multispectral inputs, producing MS-GDINO and MS-YOLOW, and proposes a mechanism to integrate text, visual, and thermal modalities. The architecture uses separate RGB and IR visual encoders, a shared text prompt, and simple visual and textual fusion:
\[
\mathbf{F}_{\mathrm{fused}} = \mathcal{F}'_{\mathrm{RGB}} + \mathcal{F}'_{\mathrm{IR}}, \qquad
\mathbf{T}_{\mathrm{fused}} = \mathrm{Concat}(T'_{\mathrm{RGB}}, T'_{\mathrm{IR}}).
\]
The paper argues that semantic priors learned by large-scale VLMs effectively transfer to unseen spectral modalities and reports that VLM-based detectors not only excel in few-shot regimes but also achieve competitive or superior results under fully supervised settings [2512.15971].

This creates a notable contrast within the FSMOD literature. FSMODNet emphasizes careful local refinement, deformable cross-modal alignment, and prototype-conditioned detection, whereas the VLM-based study argues that simple fusion plus semantic supervision is enough to make VLMs strong FSMOD learners. A plausible implication is that the field contains at least two complementary design philosophies: one centered on explicit multispectral alignment and episodic few-shot adaptation, and another centered on semantic grounding from large-scale pretraining.

## 7. Limitations, misconceptions, and emerging directions

One recurring misconception is that multispectral few-shot detection is simply standard few-shot detection with an additional channel. The reported methods do not support that view. FSMODNet identifies the modality gap, cross-modal misalignment, low-data fusion instability, and illumination-dependent appearance change as core problems, while VLM-based FSMOD adds the unlabeled object problem and the semantic gap across spectral domains [2509.20905].

A second misconception is that more elaborate fusion always yields better few-shot performance. FSMODNet reports that explicit alignment and adaptive cross-spectral sampling are more effective than simple feature merging in its ablations, particularly at higher shot counts. By contrast, the VLM-based study argues that heavy, specialized fusion architectures are not always necessary when semantic priors are strong and shows strong performance with elementwise visual addition and text-feature concatenation [2512.15971]. These findings are not logically incompatible; they indicate that the relative importance of fusion complexity and semantic priors depends on the detector family, the pretraining regime, and the supervision level.

Efficiency is an additional limitation. FSMODNet is reported to be slower and heavier than the TFA-based baselines, largely because episodic conditioning means the detector processes only the supported classes per pass, so detecting many classes requires multiple passes through the DETR head. The authors note that if the target classes are few, single-pass inference is practical [2509.20905]. The VLM-based comparison offers a different trade-off: MS-YOLOW is described as the better choice for real-time applications, whereas MS-GDINO gives the best accuracy but is slower [2512.15971].

The current literature suggests several converging directions. Prototype-based episodic learning remains central in FSMODNet; prompt- and language-conditioned prototype generation broadens the available supervision in MM-FSOD; and VLM-based multispectral detection indicates that textual semantic priors can be effective even when transferred from RGB image-text pretraining to RGB-IR detection. A plausible implication is that future FSMOD systems may combine these strands: multispectral alignment mechanisms, prototype-conditioned adaptation, semantic prompts, and explicit handling of sparse or missing annotations.

Source: https://www.emergentmind.com/topics/few-shot-multispectral-object-detection-fsmod