---
title: 'Tiny-BioMoE: Compact Biosignal Embedding Model'
url: https://www.emergentmind.com/topics/tiny-biomoe
type: topic
---

# Tiny-BioMoE: Compact Biosignal Embedding Model

Searching arXiv for Tiny-BioMoE and closely related context.
Tiny-BioMoE is a lightweight pretrained embedding model for biosignal analysis introduced for automatic pain assessment in the context of the Second Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4PAIN). It is a compact Mixture-of-Experts–inspired vision model that operates on visual representations of physiological time series rather than directly on raw one-dimensional signals, and it is designed to extract a 192-dimensional embedding from biosignal images with 7.34 million parameters and 3.04 GFLOPs per \(224\times224\) image [2507.21875]. The model was pretrained on 4.39 million biosignal image representations derived from large EEG, EMG, ECG, and PPG corpora, and then fine-tuned for downstream pain recognition using electrodermal activity, blood volume pulse, respiration, peripheral oxygen saturation, and multimodal combinations [2507.21875].

## 1. Conceptual positioning

Tiny-BioMoE was proposed as a compact pretrained embedding model for biosignal analysis, with the specific downstream target of automatic pain recognition from physiological measurements [2507.21875]. The motivating premise is that pain assessment benefits from objective physiological sensing, that multimodal physiological data can improve robustness, and that automatic systems may support continuous monitoring and clinical decision-making [2507.21875].

The model is “Mixture-of-Experts–inspired” rather than a conventional sparse routed MoE. Its expert-like behavior arises from a dual-encoder design whose outputs are fused through a lightweight gating mechanism, producing per-channel modulation before concatenation into a shared embedding space [2507.21875]. This suggests that Tiny-BioMoE should be understood less as a large-scale conditional-computation MoE in the language-model sense and more as a compact gated multi-branch encoder for representation learning on biosignal imagery.

A potential source of confusion is the coexistence of unrelated “BioMoE”-like nomenclature in recent arXiv literature. For example, CodonMoE addresses DNA language models for mRNA analyses through codon-level mixture-of-experts adapters [2508.04739]. Tiny-BioMoE, by contrast, concerns biosignal image embeddings for pain recognition and uses a vision architecture with dual encoders and fusion gating rather than codon-level genomic adaptation [2507.21875].

## 2. Architecture

Tiny-BioMoE comprises two lightweight vision-transformer–style encoders, denoted Encoder-1 and Encoder-2, whose outputs are fused into a 192-dimensional embedding [2507.21875]. The architecture is explicitly organized around complementary processing paths.

### Encoder-1: Spectral+Self-Attention

Encoder-1 receives an image \(I\in\mathbb R^{3\times224\times224}\), partitions it into \(n=(224/16)^2=196\) non-overlapping \(16\times16\) patches, and linearly projects each patch to a token of dimension \(d=768\), yielding \(X\in\mathbb R^{n\times d}\) [2507.21875]. Positional encodings are added, after which each token undergoes a two-dimensional FFT:
\[
X=\mathscr F[x]\in\mathbb C^{h\times w\times d}.
\]
A learnable complex filter \(K\in\mathbb C^{h\times w\times d}\) modulates the spectrum,
\[
\tilde X=K\odot X,
\]
followed by inverse FFT:
\[
x\leftarrow\mathscr F^{-1}[\tilde X].
\]
Channel refinement is then performed with a depthwise-convolutional MLP,
\[
f(x)=W_2\;\mathrm{GELU}(\mathrm{DWConv}(W_1x+b_1))+b_2.
\]
Self-attention uses a single head per stage:
\[
\mathrm{Attn}(Q,K,V)=\mathrm{softmax}(QK^\top/\sqrt d)\,V,\quad Q=XW_q,\;K=XW_k,\;V=XW_v.
\]
The encoder uses four hierarchical stages, reducing spatial resolution by a factor of \(2\) at each step, with embedding widths \([64,\,128,\,320,\,96]\) [2507.21875].

This branch combines spectral-domain modulation, local channel mixing, and tokenwise self-attention. A plausible implication is that it is intended to capture frequency-structured regularities that are salient in spectrogram-like or wavelet-like biosignal visualizations.

### Encoder-2: Spatial-Mixer + Waterfall-Attention

Encoder-2 processes overlapping \(16\times16\) patches and projects them to token dimensions \(d\in\{192,288,96\}\) across three stages [2507.21875]. Each stage begins and ends with a Spatial-Mixer consisting of depthwise convolution per channel,
\[
Y_c=K_c* T_c+ b_c,\quad Z_c=\mathrm{BN}(Y_c),
\]
followed by a feed-forward network,
\[
\Phi^F(Z_c)=W_2\,\mathrm{ReLU}(W_1Z_c+b_1)+b_2.
\]

Each stage also contains a single Waterfall-Attention block that splits the embedding \(X_i=[X_{i1},\dots,X_{ih}]\) into \(h\) segments:
\[
\widetilde X_{ij}=\mathrm{Attn}(X_{ij}W^Q_{ij},X_{ij}W^K_{ij},X_{ij}W^V_{ij}),
\]
\[
\widetilde X_{i+1}=\mathrm{Concat}_j[\widetilde X_{ij}]\,W^P_i.
\]
Residual wiring enriches each head according to
\[
X'_{ij}=X_{ij}+\widetilde X_{i,j-1}.
\]
The three stages employ \(\{3,3,4\}\) attention heads and embedding widths \(\{192,288,96\}\) [2507.21875].

Relative to Encoder-1, this path emphasizes spatial mixing and staged attention with residual inter-head enrichment. This suggests architectural complementarity between a more explicitly spectral branch and a more explicitly spatial branch.

### Fusion and embedding head

After LayerNorm, the input is fed to both encoders to produce embeddings \(z_1,z_2\in\mathbb R^{96}\), which are individually normalized to \(\hat z_1,\hat z_2\) [2507.21875]. A lightweight gating network
\[
g(x)=\mathrm{HardTanh}(\mathrm{ELU}(W x))
\]
produces per-channel gates \(\alpha_1,\alpha_2\in[0,1]^{96}\). These gates modulate the branch outputs:
\[
z'_k=\alpha_k\odot\hat z_k,\quad k=1,2.
\]
The modulated embeddings are concatenated,
\[
z_\mathrm{cat}=[\,z'_1\|\;z'_2]\in\mathbb R^{192},
\]
and a final LayerNorm yields the output embedding [2507.21875].

The gating mechanism is central to the “MoE-inspired” characterization. Rather than routing tokens to disjoint experts, Tiny-BioMoE performs channelwise reweighting of two encoder outputs before fusion. For biosignal representations that differ substantially in spectral and morphological content, this provides a compact adaptive fusion mechanism.

## 3. Pretraining corpus and biosignal visualization

Tiny-BioMoE was pretrained in a supervised multitask setting on 4.39 million training samples from three large biosignal datasets: EEG-BST-SZ with 1.20 million EEG samples, Silent-EMG with 1.03 million EMG samples, and ECG HBC with 2.16 million ECG/PPG samples [2507.21875]. The pretraining input domain was not raw waveform tensors but image representations generated from filtered one-dimensional biosignals.

The preprocessing described in the paper includes filtering specific modalities such as EDA and SpO\(_2\) with a \(0.04\)–\(1.7\) Hz bandpass, BVP with a \(0.05\)–\(0.5\) Hz bandpass, and respiration with a \(0.04\)–\(1.7\) Hz bandpass, followed by low-pass filtering at \(5\) Hz [2507.21875]. Six visual representations were then generated for each sample:

| Representation | Description |
|---|---|
| Spectrogram-Angle | phase angle of STFT magnitudes |
| Spectrogram-Phase | unwrapped phase |
| Spectrogram-PSD | power spectral density |
| Recurrence plot | phase-space recurrences |
| Scalogram | continuous wavelet transform |
| Waveform diagram | raw amplitude vs. time |

The paper notes that the exact STFT window and overlap are not specified, but a typical short-time Fourier transform was used, for example with window length \(N\) and hop size \(N/2\):
\[
\mathrm{STFT}\{s\}(f,t)=\sum_n s[n]\,w[n-t]\,e^{-j2\pi fn/N}.
\]
Because this STFT description is explicitly framed as typical rather than dataset-specific, it should not be treated as a definitive implementation detail beyond the statement provided [2507.21875].

The pretraining design indicates a representation-learning strategy in which a shared visual encoder is exposed to multiple biosignal domains and multiple transforms of each signal. A plausible implication is that the model’s embedding space is expected to capture cross-representation invariants rather than features tied to a single handcrafted view.

## 4. Optimization and regularization

Pretraining was formulated as multi-task supervised learning over three datasets comprising 14 sub-tasks, using an adaptive weighted loss
\[
L_\text{total}=\sum_{i=1}^3\Bigl(e^{w_i}L_{S_i}+w_i\Bigr),
\]
where \(L_{S_i}\) is the cross-entropy of task \(i\) and \(w_i\) are trainable log-weights [2507.21875]. This objective dynamically balances task contributions during optimization.

The reported training configuration uses 200 epochs, AdamW with base learning rate \(1\times10^{-4}\), batch size 32, and a cosine warmup-cooldown learning-rate schedule [2507.21875]. Data augmentation includes AugMix, TrivialAugment, random center crops, conditional Gaussian blur, and two levels of Cutout with \(32\times32\) blocks [2507.21875]. Dropout probability is linearly annealed according to
\[
p(t)=p_\text{start}+\tfrac tT\,(p_\text{end}-p_\text{start}),\quad t\in[0,T],
\]
and label smoothing in the cross-entropy is also linearly scheduled [2507.21875].

These training choices place Tiny-BioMoE within a regularized supervised pretraining paradigm rather than a self-supervised contrastive or masked-modelling regime. The paper explicitly identifies self-supervised objectives as future work rather than part of the present method [2507.21875].

## 5. Downstream pain-recognition pipeline

For the AI4PAIN challenge, Tiny-BioMoE was applied to four physiological modalities: electrodermal activity, blood volume pulse, respiration, and peripheral oxygen saturation [2507.21875]. Each modality was converted into one or more of the six visual representations described above. For each resulting image, the pretrained Tiny-BioMoE produces an embedding \(z\in\mathbb R^{192}\), and the model may be fine-tuned end-to-end on the pain-recognition task [2507.21875].

The downstream procedure consists of three main steps. First, an embedding is computed with the pretrained model. Second, multiple visual representations can optionally be fused at the embedding level using either element-wise addition or concatenation. Third, a linear classification head,
\[
\mathrm{Softmax}(W_\mathrm{cls}\,z+b),
\]
is attached for three pain levels: No Pain, Low Pain, and High Pain [2507.21875]. Batch size, learning rate, and augmentation schedules mirror pretraining, and either only the classification head or the full model can be fine-tuned [2507.21875].

The fusion protocol is important because the paper reports that selective fusion of top representations per modality can outperform naïve aggregation of all available representations [2507.21875]. This design point is methodologically significant: Tiny-BioMoE functions both as a single-image encoder and as a common embedding backbone for late fusion across transforms and physiological channels.

## 6. Quantitative performance

The empirical evaluation reports validation and test results across single-modality, representation-fusion, and multimodal settings [2507.21875].

### Impact of pretraining

Across all representations and modalities, average macro-averaged validation accuracy improved from 52.13% for scratch-trained models to 53.96% for pretrained models [2507.21875]. SpO\(_2\) exhibited the largest gain, increasing from 55.59% to 62.73% [2507.21875].

### Best single-modality and single-representation results

| Modality | Best representation | Scratch | Pretrained |
|---|---|---:|---:|
| BVP | Scalogram | 66.53% | 67.13% |
| EDA | Scalogram | 73.41% | 71.15% |
| Respiration | Scalogram | 71.91% | 73.53% |
| SpO\(_2\) | Waveform | 54.87% | 70.49% |

These results show that pretraining is not uniformly beneficial for every modality-representation pair. In particular, EDA with Scalogram decreased slightly from 73.41% to 71.15% [2507.21875]. This is relevant for avoiding the misconception that larger-scale pretraining necessarily improves every downstream slice of the evaluation.

### Fusion of representations within modality

The paper reports several representation-fusion experiments. For BVP, concatenating all six representations yielded 58.87%, which is below the best single representation at 67.13% [2507.21875]. For EDA, combining Scalogram and Recurrence improved performance to 76.85% with addition and 77.88% with concatenation, above the 71.15% best pretrained single-representation score reported for EDA [2507.21875]. For respiration, concatenating Scalogram and PSD gave 69.10%, below the best single representation at 73.53% [2507.21875]. For SpO\(_2\), concatenating Scalogram and Recurrence reached 74.54%, exceeding the best single representation at 70.49% [2507.21875].

These outcomes support the paper’s claim that selective fusion outperforms naïve “all-in” fusion [2507.21875]. The effect is evidently modality-dependent.

### Multimodal fusion

The best-performing multimodal setting selected the following modality-specific representations: BVP with Scalogram; EDA with Scalogram + Waveform; Respiration with Scalogram; and SpO\(_2\) with Scalogram + Waveform [2507.21875]. Under this selection, fusion by addition achieved 81.02% validation accuracy with 81.72% precision and 81.36% F1, while fusion by concatenation achieved 82.41% validation accuracy with 84.25% precision and 82.43% F1 [2507.21875].

On the AI4PAIN test set using physiological modalities only, Tiny-BioMoE with EDA+BVP+Resp+SpO\(_2\) obtained 54.89% accuracy, described as competitive with prior transformer and ensemble approaches, including 55.17% with EDA only and 42.24% with respiration only [2507.21875].

The validation–test gap suggests caution in interpreting the downstream gains as established clinical robustness. The paper itself attributes a limitation to evaluation on controlled laboratory data and notes that real-world generalization remains to be validated [2507.21875].

## 7. Efficiency, limitations, and reproducibility

Tiny-BioMoE is reported to have 7.34 million parameters and to require 3.04 GFLOPs per \(224\times224\) image [2507.21875]. The paper characterizes this footprint as orders of magnitude smaller than conventional vision transformers and therefore suitable for resource-constrained deployment [2507.21875]. In the context of biosignal analysis, where portable or embedded inference can be operationally relevant, this compactness is a defining feature of the model.

The principal strengths identified in the paper are efficient representation learning, improved feature extraction from large-scale biosignal imagery, especially for weaker-signal modalities such as SpO\(_2\), the advantage of selective representation fusion, and the strong gains obtained through multimodal fusion [2507.21875]. The principal stated limitation is that the reported evaluation was conducted on controlled laboratory data, leaving real-world generalization unresolved [2507.21875]. Future directions proposed in the paper include self-supervised objectives, additional biosignal types such as ECG and EEG, and dynamic gating strategies [2507.21875].

Reproducibility is supported by public release of model code, pretrained weights, data-processing scripts, and training scripts at the GitHub repository linked in the paper [2507.21875]. The repository is described as including instructions for environment setup with Python \(\ge 3.8\) and PyTorch, conversions to the six visual formats, and example training and fine-tuning configurations [2507.21875].

Taken together, Tiny-BioMoE occupies a specific place in the emerging literature on learned biosignal representations: a compact, supervised-pretrained, image-based physiological embedding model with gated dual-encoder fusion, evaluated primarily on automatic pain recognition. Its empirical profile indicates that the model is most compelling when used as a pretrained feature extractor within selective intra-modality and multimodal fusion pipelines rather than as a universal replacement for modality-specific engineering [2507.21875].

Source: https://www.emergentmind.com/topics/tiny-biomoe