Tiny-BioMoE: Compact Biosignal Embedding Model

• Tiny-BioMoE is a lightweight pretrained biosignal embedding model that converts visualized physiological signals into 192-dimensional representations for pain assessment.
• It employs a dual-encoder architecture combining spectral self-attention and spatial mixing with a gated fusion mechanism to selectively aggregate features.
• Pretrained on millions of biosignal images, the model shows improved multimodal pain recognition performance while maintaining efficiency with 7.34M parameters and 3.04 GFLOPs per image.

Searching arXiv for Tiny-BioMoE and closely related context. Tiny-BioMoE is a lightweight pretrained embedding model for biosignal analysis introduced for automatic pain assessment in the context of the Second Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4PAIN). It is a compact Mixture-of-Experts–inspired vision model that operates on visual representations of physiological time series rather than directly on raw one-dimensional signals, and it is designed to extract a 192-dimensional embedding from biosignal images with 7.34 million parameters and 3.04 GFLOPs per $224\times224$ image (Gkikas et al., 29 Jul 2025). The model was pretrained on 4.39 million biosignal image representations derived from large EEG, EMG, ECG, and PPG corpora, and then fine-tuned for downstream pain recognition using electrodermal activity, blood volume pulse, respiration, peripheral oxygen saturation, and multimodal combinations (Gkikas et al., 29 Jul 2025).

1. Conceptual positioning

Tiny-BioMoE was proposed as a compact pretrained embedding model for biosignal analysis, with the specific downstream target of automatic pain recognition from physiological measurements (Gkikas et al., 29 Jul 2025). The motivating premise is that pain assessment benefits from objective physiological sensing, that multimodal physiological data can improve robustness, and that automatic systems may support continuous monitoring and clinical decision-making (Gkikas et al., 29 Jul 2025).

The model is “Mixture-of-Experts–inspired” rather than a conventional sparse routed MoE. Its expert-like behavior arises from a dual-encoder design whose outputs are fused through a lightweight gating mechanism, producing per-channel modulation before concatenation into a shared embedding space (Gkikas et al., 29 Jul 2025). This suggests that Tiny-BioMoE should be understood less as a large-scale conditional-computation MoE in the language-model sense and more as a compact gated multi-branch encoder for representation learning on biosignal imagery.

A potential source of confusion is the coexistence of unrelated “BioMoE”-like nomenclature in recent arXiv literature. For example, CodonMoE addresses DNA LLMs for mRNA analyses through codon-level mixture-of-experts adapters (Du et al., 6 Aug 2025). Tiny-BioMoE, by contrast, concerns biosignal image embeddings for pain recognition and uses a vision architecture with dual encoders and fusion gating rather than codon-level genomic adaptation (Gkikas et al., 29 Jul 2025).

2. Architecture

Tiny-BioMoE comprises two lightweight vision-transformer–style encoders, denoted Encoder-1 and Encoder-2, whose outputs are fused into a 192-dimensional embedding (Gkikas et al., 29 Jul 2025). The architecture is explicitly organized around complementary processing paths.

Encoder-1: Spectral+Self-Attention

Encoder-1 receives an image $I\in\mathbb R^{3\times224\times224}$, partitions it into $n=(224/16)^2=196$ non-overlapping $16\times16$ patches, and linearly projects each patch to a token of dimension $d=768$, yielding $X\in\mathbb R^{n\times d}$ (Gkikas et al., 29 Jul 2025). Positional encodings are added, after which each token undergoes a two-dimensional FFT: $X=\mathscr F[x]\in\mathbb C^{h\times w\times d}.$ A learnable complex filter $K\in\mathbb C^{h\times w\times d}$ modulates the spectrum,

$\tilde X=K\odot X,$

followed by inverse FFT: $x\leftarrow\mathscr F^{-1}[\tilde X].$ Channel refinement is then performed with a depthwise-convolutional MLP,

$I\in\mathbb R^{3\times224\times224}$0

Self-attention uses a single head per stage: $I\in\mathbb R^{3\times224\times224}$1 The encoder uses four hierarchical stages, reducing spatial resolution by a factor of $I\in\mathbb R^{3\times224\times224}$2 at each step, with embedding widths $I\in\mathbb R^{3\times224\times224}$3 (Gkikas et al., 29 Jul 2025).

This branch combines spectral-domain modulation, local channel mixing, and tokenwise self-attention. A plausible implication is that it is intended to capture frequency-structured regularities that are salient in spectrogram-like or wavelet-like biosignal visualizations.

Encoder-2: Spatial-Mixer + Waterfall-Attention

Encoder-2 processes overlapping $I\in\mathbb R^{3\times224\times224}$4 patches and projects them to token dimensions $I\in\mathbb R^{3\times224\times224}$5 across three stages (Gkikas et al., 29 Jul 2025). Each stage begins and ends with a Spatial-Mixer consisting of depthwise convolution per channel,

$I\in\mathbb R^{3\times224\times224}$6

followed by a feed-forward network,

$I\in\mathbb R^{3\times224\times224}$7

Each stage also contains a single Waterfall-Attention block that splits the embedding $I\in\mathbb R^{3\times224\times224}$8 into $I\in\mathbb R^{3\times224\times224}$9 segments: $n=(224/16)^2=196$0

$n=(224/16)^2=196$1

Residual wiring enriches each head according to

$n=(224/16)^2=196$2

The three stages employ $n=(224/16)^2=196$3 attention heads and embedding widths $n=(224/16)^2=196$4 (Gkikas et al., 29 Jul 2025).

Relative to Encoder-1, this path emphasizes spatial mixing and staged attention with residual inter-head enrichment. This suggests architectural complementarity between a more explicitly spectral branch and a more explicitly spatial branch.

Fusion and embedding head

After LayerNorm, the input is fed to both encoders to produce embeddings $n=(224/16)^2=196$5, which are individually normalized to $n=(224/16)^2=196$6 (Gkikas et al., 29 Jul 2025). A lightweight gating network

$n=(224/16)^2=196$7

produces per-channel gates $n=(224/16)^2=196$8. These gates modulate the branch outputs: $n=(224/16)^2=196$9 The modulated embeddings are concatenated,

$16\times16$0

and a final LayerNorm yields the output embedding (Gkikas et al., 29 Jul 2025).

The gating mechanism is central to the “MoE-inspired” characterization. Rather than routing tokens to disjoint experts, Tiny-BioMoE performs channelwise reweighting of two encoder outputs before fusion. For biosignal representations that differ substantially in spectral and morphological content, this provides a compact adaptive fusion mechanism.

3. Pretraining corpus and biosignal visualization

Tiny-BioMoE was pretrained in a supervised multitask setting on 4.39 million training samples from three large biosignal datasets: EEG-BST-SZ with 1.20 million EEG samples, Silent-EMG with 1.03 million EMG samples, and ECG HBC with 2.16 million ECG/PPG samples (Gkikas et al., 29 Jul 2025). The pretraining input domain was not raw waveform tensors but image representations generated from filtered one-dimensional biosignals.

The preprocessing described in the paper includes filtering specific modalities such as EDA and SpO$16\times16$1 with a $16\times16$2–$16\times16$3 Hz bandpass, BVP with a $16\times16$4–$16\times16$5 Hz bandpass, and respiration with a $16\times16$6–$16\times16$7 Hz bandpass, followed by low-pass filtering at $16\times16$8 Hz (Gkikas et al., 29 Jul 2025). Six visual representations were then generated for each sample:

Representation	Description
Spectrogram-Angle	phase angle of STFT magnitudes
Spectrogram-Phase	unwrapped phase
Spectrogram-PSD	power spectral density
Recurrence plot	phase-space recurrences
Scalogram	continuous wavelet transform
Waveform diagram	raw amplitude vs. time

The paper notes that the exact STFT window and overlap are not specified, but a typical short-time Fourier transform was used, for example with window length $16\times16$9 and hop size $d=768$0: $d=768$1 Because this STFT description is explicitly framed as typical rather than dataset-specific, it should not be treated as a definitive implementation detail beyond the statement provided (Gkikas et al., 29 Jul 2025).

The pretraining design indicates a representation-learning strategy in which a shared visual encoder is exposed to multiple biosignal domains and multiple transforms of each signal. A plausible implication is that the model’s embedding space is expected to capture cross-representation invariants rather than features tied to a single handcrafted view.

4. Optimization and regularization

Pretraining was formulated as multi-task supervised learning over three datasets comprising 14 sub-tasks, using an adaptive weighted loss

$d=768$2

where $d=768$3 is the cross-entropy of task $d=768$4 and $d=768$5 are trainable log-weights (Gkikas et al., 29 Jul 2025). This objective dynamically balances task contributions during optimization.

The reported training configuration uses 200 epochs, AdamW with base learning rate $d=768$6, batch size 32, and a cosine warmup-cooldown learning-rate schedule (Gkikas et al., 29 Jul 2025). Data augmentation includes AugMix, TrivialAugment, random center crops, conditional Gaussian blur, and two levels of Cutout with $d=768$7 blocks (Gkikas et al., 29 Jul 2025). Dropout probability is linearly annealed according to

$d=768$8

and label smoothing in the cross-entropy is also linearly scheduled (Gkikas et al., 29 Jul 2025).

These training choices place Tiny-BioMoE within a regularized supervised pretraining paradigm rather than a self-supervised contrastive or masked-modelling regime. The paper explicitly identifies self-supervised objectives as future work rather than part of the present method (Gkikas et al., 29 Jul 2025).

5. Downstream pain-recognition pipeline

For the AI4PAIN challenge, Tiny-BioMoE was applied to four physiological modalities: electrodermal activity, blood volume pulse, respiration, and peripheral oxygen saturation (Gkikas et al., 29 Jul 2025). Each modality was converted into one or more of the six visual representations described above. For each resulting image, the pretrained Tiny-BioMoE produces an embedding $d=768$9, and the model may be fine-tuned end-to-end on the pain-recognition task (Gkikas et al., 29 Jul 2025).

The downstream procedure consists of three main steps. First, an embedding is computed with the pretrained model. Second, multiple visual representations can optionally be fused at the embedding level using either element-wise addition or concatenation. Third, a linear classification head,

$X\in\mathbb R^{n\times d}$0

is attached for three pain levels: No Pain, Low Pain, and High Pain (Gkikas et al., 29 Jul 2025). Batch size, learning rate, and augmentation schedules mirror pretraining, and either only the classification head or the full model can be fine-tuned (Gkikas et al., 29 Jul 2025).

The fusion protocol is important because the paper reports that selective fusion of top representations per modality can outperform naïve aggregation of all available representations (Gkikas et al., 29 Jul 2025). This design point is methodologically significant: Tiny-BioMoE functions both as a single-image encoder and as a common embedding backbone for late fusion across transforms and physiological channels.

6. Quantitative performance

The empirical evaluation reports validation and test results across single-modality, representation-fusion, and multimodal settings (Gkikas et al., 29 Jul 2025).

Impact of pretraining

Across all representations and modalities, average macro-averaged validation accuracy improved from 52.13% for scratch-trained models to 53.96% for pretrained models (Gkikas et al., 29 Jul 2025). SpO$X\in\mathbb R^{n\times d}$1 exhibited the largest gain, increasing from 55.59% to 62.73% (Gkikas et al., 29 Jul 2025).

Best single-modality and single-representation results

Modality	Best representation	Scratch	Pretrained
BVP	Scalogram	66.53%	67.13%
EDA	Scalogram	73.41%	71.15%
Respiration	Scalogram	71.91%	73.53%
SpO$X\in\mathbb R^{n\times d}$2	Waveform	54.87%	70.49%

These results show that pretraining is not uniformly beneficial for every modality-representation pair. In particular, EDA with Scalogram decreased slightly from 73.41% to 71.15% (Gkikas et al., 29 Jul 2025). This is relevant for avoiding the misconception that larger-scale pretraining necessarily improves every downstream slice of the evaluation.

Fusion of representations within modality

The paper reports several representation-fusion experiments. For BVP, concatenating all six representations yielded 58.87%, which is below the best single representation at 67.13% (Gkikas et al., 29 Jul 2025). For EDA, combining Scalogram and Recurrence improved performance to 76.85% with addition and 77.88% with concatenation, above the 71.15% best pretrained single-representation score reported for EDA (Gkikas et al., 29 Jul 2025). For respiration, concatenating Scalogram and PSD gave 69.10%, below the best single representation at 73.53% (Gkikas et al., 29 Jul 2025). For SpO$X\in\mathbb R^{n\times d}$3, concatenating Scalogram and Recurrence reached 74.54%, exceeding the best single representation at 70.49% (Gkikas et al., 29 Jul 2025).

These outcomes support the paper’s claim that selective fusion outperforms naïve “all-in” fusion (Gkikas et al., 29 Jul 2025). The effect is evidently modality-dependent.

Multimodal fusion

The best-performing multimodal setting selected the following modality-specific representations: BVP with Scalogram; EDA with Scalogram + Waveform; Respiration with Scalogram; and SpO$X\in\mathbb R^{n\times d}$4 with Scalogram + Waveform (Gkikas et al., 29 Jul 2025). Under this selection, fusion by addition achieved 81.02% validation accuracy with 81.72% precision and 81.36% F1, while fusion by concatenation achieved 82.41% validation accuracy with 84.25% precision and 82.43% F1 (Gkikas et al., 29 Jul 2025).

On the AI4PAIN test set using physiological modalities only, Tiny-BioMoE with EDA+BVP+Resp+SpO$X\in\mathbb R^{n\times d}$5 obtained 54.89% accuracy, described as competitive with prior transformer and ensemble approaches, including 55.17% with EDA only and 42.24% with respiration only (Gkikas et al., 29 Jul 2025).

The validation–test gap suggests caution in interpreting the downstream gains as established clinical robustness. The paper itself attributes a limitation to evaluation on controlled laboratory data and notes that real-world generalization remains to be validated (Gkikas et al., 29 Jul 2025).

7. Efficiency, limitations, and reproducibility

Tiny-BioMoE is reported to have 7.34 million parameters and to require 3.04 GFLOPs per $X\in\mathbb R^{n\times d}$6 image (Gkikas et al., 29 Jul 2025). The paper characterizes this footprint as orders of magnitude smaller than conventional vision transformers and therefore suitable for resource-constrained deployment (Gkikas et al., 29 Jul 2025). In the context of biosignal analysis, where portable or embedded inference can be operationally relevant, this compactness is a defining feature of the model.

The principal strengths identified in the paper are efficient representation learning, improved feature extraction from large-scale biosignal imagery, especially for weaker-signal modalities such as SpO$X\in\mathbb R^{n\times d}$7, the advantage of selective representation fusion, and the strong gains obtained through multimodal fusion (Gkikas et al., 29 Jul 2025). The principal stated limitation is that the reported evaluation was conducted on controlled laboratory data, leaving real-world generalization unresolved (Gkikas et al., 29 Jul 2025). Future directions proposed in the paper include self-supervised objectives, additional biosignal types such as ECG and EEG, and dynamic gating strategies (Gkikas et al., 29 Jul 2025).

Reproducibility is supported by public release of model code, pretrained weights, data-processing scripts, and training scripts at the GitHub repository linked in the paper (Gkikas et al., 29 Jul 2025). The repository is described as including instructions for environment setup with Python $X\in\mathbb R^{n\times d}$8 and PyTorch, conversions to the six visual formats, and example training and fine-tuning configurations (Gkikas et al., 29 Jul 2025).

Taken together, Tiny-BioMoE occupies a specific place in the emerging literature on learned biosignal representations: a compact, supervised-pretrained, image-based physiological embedding model with gated dual-encoder fusion, evaluated primarily on automatic pain recognition. Its empirical profile indicates that the model is most compelling when used as a pretrained feature extractor within selective intra-modality and multimodal fusion pipelines rather than as a universal replacement for modality-specific engineering (Gkikas et al., 29 Jul 2025).