FSMODNet: Multispectral Detection & Modulated Aggregation

• FSMODNet is a dual-purpose term representing both a few-shot multispectral object detector that fuses RGB and IR features and a frequency-modulated interface for edge learning.
• The detection framework enhances accuracy through a three-stage modality fusion using neighborhood and cross-deformable attention paired with prototype-based meta-learning.
• The communications construct leverages frequency modulation and type-based multiple access to achieve constant-envelope, low-PAPR transmissions for robust federated learning.

FSMODNet denotes two distinct constructs in recent arXiv literature. In "FSMODNet: A Closer Look at Few-Shot Detection in Multispectral Data," it is explicitly defined as a Few-Shot Multispectral Object Detection Network for visible (RGB) and thermal/infrared (IR) detection with limited annotation (Nkegoum et al., 25 Sep 2025). In "Frequency Modulation for Task-Oriented Communications and Multiple Access," the same label is associated, in the supplied terminology, with a frequency-modulation-based air interface for task-oriented communications in federated edge learning rather than with a separate deep neural network architecture (Martinez-Gost et al., 2024). The shared label therefore spans two technically unrelated lines of work: few-shot multispectral perception and frequency-modulated over-the-air aggregation.

1. Terminological scope and problem domains

In the multispectral detection literature, FSMODNet addresses few-shot multispectral object detection (FSMOD), a setting that combines few-shot object detection (FSOD) and multispectral object detection (MOD) (Nkegoum et al., 25 Sep 2025). The task is to detect novel categories with only $K$ labeled examples per class while exploiting both RGB and thermal/IR imagery. The training set is partitioned into a base set of classes $C_{\text{base}}$ with many labeled instances and a novel set $C_{\text{novel}}$ with only $K$ annotated examples per class, with $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$. Training is episodic, with a support set and a query image in each episode.

The motivation given for this formulation is that existing multispectral detectors work well only when large annotated datasets are available, whereas FSOD methods usually focus on RGB only (Nkegoum et al., 25 Sep 2025). RGB contributes appearance, texture, and shape, while IR contributes heat signatures and can be more reliable under poor visibility, low light, occlusion, fog, and cluttered scenes. This motivates adaptive cross-modality integration rather than naive concatenation.

In the communications literature, FSMODNet refers to a waveform and access design for Federated Edge Learning (FEEL) under task-oriented communications (Martinez-Gost et al., 2024). In this setting, $K$ edge devices collaboratively train a shared model. Rather than transmitting local updates for bit-exact reconstruction, the devices quantize model parameters or gradients, map them to tones, and transmit them simultaneously so that the receiver estimates a histogram over transmitted types. The server then extracts an aggregation statistic such as the mean. This formulation is grounded in over-the-air computing (AirComp), which exploits the superposition property of the wireless multiple-access channel.

The two usages share only the acronymic surface form. One is a detector for low-data multispectral recognition; the other is a frequency-modulated, type-based aggregation mechanism for wireless learning. A plausible implication is that the term should always be disambiguated by citation or by expansion on first use.

2. Few-shot multispectral object detection framework

The 2025 FSMODNet framework has three major parts: modality-specific feature extraction, cross-modality fusion, and prototype-based few-shot detection + DINO head (Nkegoum et al., 25 Sep 2025). Each modality has its own encoder for support and query images, and the features are fused before detection. The overall pipeline extracts RGB and IR features separately, refines each modality with intra-modal attention, exchanges information across modalities with cross-deformable attention, merges them into a unified query representation, aggregates support prototypes, and uses a DINO-DETR-based detector for final prediction.

The principal technical contribution is the three-stage cross-modality integration mechanism. Let

$$F_{IR}, F_{RGB} \in \mathbb{R}^{D \times H \times W}.$$

In Stage 1, each modality is refined with Neighborhood Attention. For RGB,

$$Q = W_Q * F_{RGB}, \quad K = W_K * F_{RGB}, \quad V = W_V * F_{RGB},$$

and for a spatial location $(i,j)$, attention is computed over a local neighborhood $\mathcal{N}_{(i,j)}$. The normalized attention and output feature are given by

$C_{\text{base}}$0

$C_{\text{base}}$1

The same process is applied to IR, producing $C_{\text{base}}$2 and $C_{\text{base}}$3 (Nkegoum et al., 25 Sep 2025).

In Stage 2, bidirectional Cross-Deformable Attention performs cross-modal exchange and alignment. A grid of reference points

$C_{\text{base}}$4

is defined on one modality. Offsets are predicted by

$C_{\text{base}}$5

where $C_{\text{base}}$6 is a projected feature at $C_{\text{base}}$7, $C_{\text{base}}$8 controls offset magnitude, and $C_{\text{base}}$9 uses depth-wise convolution, normalization, GELU, and a $C_{\text{novel}}$0 convolution. Deformed sampling is performed by bilinear interpolation,

$C_{\text{novel}}$1

followed by key and value projections

$C_{\text{novel}}$2

The query from the opposite modality is

$C_{\text{novel}}$3

with attention

$C_{\text{novel}}$4

The updated RGB feature is

$C_{\text{novel}}$5

and the same process is applied in reverse to update IR (Nkegoum et al., 25 Sep 2025).

In Stage 3, the updated modality-specific features are concatenated,

$C_{\text{novel}}$6

and a $C_{\text{novel}}$7 convolution produces the fused query representation

$C_{\text{novel}}$8

This design is explicitly intended to go beyond simple concatenation or addition. The stated rationale is that Cross-Deformable Attention learns where to sample across modalities, handles imperfect alignment, lets each modality query the other adaptively, and focuses on informative regions in the counterpart modality (Nkegoum et al., 25 Sep 2025). This suggests that the framework treats alignment as a learned correspondence problem rather than as a fixed registration assumption.

3. Prototype aggregation, detection head, and optimization protocol

FSMODNet builds on Meta-DETR style prototype-based meta-learning (Nkegoum et al., 25 Sep 2025). For each support box, the framework applies RoIAlign, average pools the region, and obtains one class prototype. Let

$C_{\text{novel}}$9

be the support prototypes, and

$K$0

the fixed task encodings. The fused query features are flattened as

$K$1

Attention between query and support prototypes is computed as

$K$2

The prototype-conditioned quantities are

$K$3

and the aggregated prototype feature is

$K$4

The interpretation provided in the source is that $K$5 filters query features using class-specific support information, $K$6 injects task-level encodings, and the FFN converts class-specific support into a class-agnostic prototype embedding (Nkegoum et al., 25 Sep 2025). After fusion and prototype aggregation, the model uses a DINO-DETR detection head to produce final bounding boxes and class predictions.

The training objective combines detector losses from DINO-DETR, prototype classification supervision, and episodic meta-learning over base and novel classes (Nkegoum et al., 25 Sep 2025). The paper also uses a cosine similarity–based cross-entropy loss inspired by Meta R-CNN / closer-to-center style supervision to classify prototypes and align support/query embeddings. The training protocol has two stages: Base meta-learning on $K$7 to learn transferable representations, followed by Few-shot fine-tuning on novel classes with only $K$8 examples per class while including base samples to reduce catastrophic forgetting. During inference, class prototypes can be precomputed by averaging class-specific support prototypes, which is reported to reduce variance and speed inference.

The explicit emphasis on reproducibility is part of the framework’s stated motivation. The authors position FSMODNet as a reproducible, strong baseline for FSMOD and note earlier limitations in the area, including no strong monospectral baselines, training and fine-tuning on different datasets, limited shot settings, and lack of code and public splits (Nkegoum et al., 25 Sep 2025).

4. Datasets, baselines, and empirical behavior

FSMODNet is evaluated on two public multispectral datasets (Nkegoum et al., 25 Sep 2025). FLIR contains 4129 train / 1013 test aligned RGB-thermal pairs, with 3 classes: people, car, bicycle, and three base/novel splits. M$K$9FD contains 4200 co-registered RGB-thermal pairs, spans Day, Night, Overcast, Challenge, and has 6 classes: people, car, truck, bus, motorcycle, lamp, with three novel/base splits based on rare classes.

The experimental protocol uses ResNet-50 for both RGB and IR branches, 6-layer encoder and decoder, AdamW, learning rate $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$0, weight decay $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$1, batch size 16 during base training and 2 during fine-tuning, 50 epochs in the base stage, and 100 epochs in fine-tuning (Nkegoum et al., 25 Sep 2025). Evaluations are conducted at 5-shot, 10-shot, 30-shot, with nAP50 as the metric, and 10 support sets are generated during fine-tuning to assess robustness.

Because FSMOD baselines were scarce, the comparisons combine DAMSDet + TFA, CAFF-DINO + TFA, and Meta-DETR on RGB only and IR only (Nkegoum et al., 25 Sep 2025). Fusion strategies include Concat, Add, CMI, and CDA, where CDA denotes the proposed Cross-Deformable Attention.

On FLIR, FSMODNet RGB+IR achieves the following nAP50 values (Nkegoum et al., 25 Sep 2025):

Shot setting	Split 1	Split 2	Split 3
5-shot	30.67	27.05	32.04
10-shot	36.56	29.76	37.20
30-shot	49.70	42.47	46.18

On M$C_{\text{base}} \cap C_{\text{novel}} = \emptyset$2FD, FSMODNet RGB+IR achieves (Nkegoum et al., 25 Sep 2025):

Shot setting	Split 1	Split 2	Split 3
5-shot	29.43	37.02	25.14
10-shot	40.11	47.22	42.71
30-shot	47.32	52.18	48.77

The paper states that the framework consistently outperforms the baselines, especially in low-data regimes (Nkegoum et al., 25 Sep 2025). It also reports several notable observations. Multimodal fusion is usually best, especially as shot count increases. In some very low-shot cases, monomodal IR can be competitive or even better on FLIR, which the paper attributes to optimization difficulty when data are extremely limited. The fused model nevertheless tends to dominate as more support examples become available.

The ablations further specify the contribution of the architecture. For backbone choice on FLIR Split 1, the reported results are ConvNeXt-S: 22.34 / 30.59 / 41.83, Swin-T: 23.66 / 33.42 / 44.50, DINOv2-S: 25.30 / 33.21 / 44.67, and ResNet-50: 30.67 / 36.56 / 49.07 across 5-shot, 10-shot, and 30-shot (Nkegoum et al., 25 Sep 2025). For fusion on FLIR Split 1, the reported results are Concat: 29.62 / 33.38 / 45.40, Add: 22.82 / 32.67 / 44.95, CMI: 31.52 / 34.61 / 46.04, and CDA: 30.67 / 36.56 / 49.07 (Nkegoum et al., 25 Sep 2025). The stated takeaway is that simple addition is weakest, concatenation is strong but limited, CMI is competitive, and Cross-Deformable Attention provides the best overall results, especially at 30-shot.

Qualitatively, the model is reported to detect small and distant pedestrians better than baselines and to produce fewer false positives (Nkegoum et al., 25 Sep 2025). Runtime figures are 362 GFLOPs, 86M parameters, and 5.94 FPS, with the added cost attributed to episodic conditioning and repeated decoding per class set.

5. Frequency-modulated over-the-air aggregation usage

In the 2024 communications paper, the construct labeled FSMODNet in the supplied material is not a detector and not a separate neural architecture; it is the paper’s frequency-modulation-based air interface for task-oriented communications in federated edge learning (Martinez-Gost et al., 2024). The central claim is that waveform and access design should be optimized for the learning task rather than for symbol-by-symbol recovery.

The method combines frequency modulation / MFSK with type-based multiple access (TBMA) (Martinez-Gost et al., 2024). Each local model parameter or gradient component is quantized and mapped according to

$C_{\text{base}} \cap C_{\text{novel}} = \emptyset$3

where $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$4 is the $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$5-th parameter of the $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$6-th local model, $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$7 is an $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$8-level uniform quantizer, and $C_{\text{base}} \cap C_{\text{novel}} = \emptyset$9 maps the quantized value to an integer index. The symbol is then transmitted through an $K$0-ary frequency-shift-keying-like waveform,

$K$1

for $K$2. The supplied description notes a formatting issue in the paper text, but the intended point is that each parameter value selects a single tone and the tone index encodes the magnitude or type of the parameter.

For parameter $K$3, the receiver observes

$K$4

with additive white Gaussian noise. Because the tones are orthogonal, a matched filter bank yields, for each frequency bin, a noisy count of how many users selected that tone. The receiver therefore estimates a noisy histogram of parameter types and can compute an aggregate such as the mean (Martinez-Gost et al., 2024). The paper explicitly positions TBMA between classical AirComp and orthogonal multiple access: when multiple users choose the same type, their signals add constructively as in AirComp; when different types map to orthogonal tones, they are distinguishable as in OMA, but in a distributed and semantic way.

The practical federated-learning loop implied by the paper proceeds as follows: each device computes a local update; each parameter is quantized; the quantized value is mapped to a tone index; the device transmits the corresponding waveform; the server receives the sum; matched filtering estimates a histogram over tones; the server computes the desired aggregate; the aggregated update forms the new global model; and the global model is broadcast back to devices (Martinez-Gost et al., 2024). The paper further emphasizes that TBMA supports aggregation of functions beyond the mean, including harmonic/geometric means, maximum, minimum, and variance.

6. Comparative significance, assumptions, and limitations

The significance of the multispectral FSMODNet lies in its integration of modality-specific encoders, Neighborhood Attention, bidirectional Cross-Deformable Attention, prototype aggregation, and DINO-DETR detection within a single few-shot framework (Nkegoum et al., 25 Sep 2025). Its reported role is to provide a strong and reproducible baseline for few-shot multispectral perception. The most prominent limitations documented in the source are computational: 362 GFLOPs, 86M parameters, and 5.94 FPS. The paper states that this overhead arises from episodic conditioning and repeated decoding per class set, while maintaining that the model remains useful when the target category set is small.

The significance of the communications usage lies in reframing the wireless link as part of the learning algorithm (Martinez-Gost et al., 2024). Instead of reconstructing each user’s message, the receiver extracts the learning-relevant aggregate from the superposed waveform. The paper argues that this is better aligned with FEEL objectives, where the server needs an aggregate such as the average model parameter or gradient rather than perfect decoding of each individual transmission.

A major practical point in that work is peak-to-average power ratio (PAPR) (Martinez-Gost et al., 2024). The paper states that MFSK/FM has constant envelope, so its PAPR is 0 dB, while the compared linear analog scheme, DSB-based AirComp, has 14 dB PAPR in the reported experiment. The argument is that high PAPR stresses power amplifiers, can force operation outside the linear regime, increases sensitivity to distortion and interference, and is difficult for battery-powered edge devices. The same source also notes that linear analog modulation requires prior knowledge of transmitted data to set power appropriately, which it characterizes as unrealistic in a distributed setting.

The reported FEEL experiments use $K$5 devices, AWGN channel, standard parameter averaging, MNIST classification, and a CNN model, with coherent demodulation assumed, with fading already compensated (Martinez-Gost et al., 2024). The comparisons include TBMA with MFSK, linear analog AirComp with DSB modulation, and centralized training as an upper bound. The key findings are that MFSK/TBMA maintains performance almost unchanged down to about $K$6 dB SNR, while DSB degrades much earlier; below $K$7 dB, noise corrupts the mean estimate enough to trap learning in a local minimum or prevent convergence. The paper also notes that accurate histogram recovery requires $K$8 in its setup and shows results for $K$9.

Several assumptions and open issues are explicit. In the communications setting, coherent demodulation is required, quantization is mandatory, histogram recovery depends on the relation between user count and number of tones, and the bandwidth vs. convergence tradeoff is left for future work (Martinez-Gost et al., 2024). Future work is stated to include averaging techniques, robust learning, and the effect of fading channels and bandwidth on convergence speed. In the detection setting, the paper’s own observations indicate that multimodal fusion is not uniformly superior in the most extreme low-shot cases, where monomodal IR can occasionally be competitive on FLIR (Nkegoum et al., 25 Sep 2025).

Taken together, the two usages of FSMODNet illustrate a nomenclatural collision rather than a unified research program. One denotes a multispectral few-shot detector centered on adaptive RGB-IR fusion and prototype-conditioned DINO-DETR inference (Nkegoum et al., 25 Sep 2025). The other denotes a frequency-modulated, type-based over-the-air aggregation mechanism for federated learning, with emphasis on constant-envelope signaling, histogram recovery, and low-PAPR transmission (Martinez-Gost et al., 2024).