---
title: 'CircFormerMoE: Plant circRNA Prediction'
url: https://www.emergentmind.com/topics/circformermoe
type: topic
---

# CircFormerMoE: Plant circRNA Prediction

CircFormerMoE is an end-to-end deep learning framework for predicting plant circular RNAs directly from genomic DNA sequence rather than from RNA-seq alignment pipelines. It was introduced to address a specific bottleneck in plant circRNA annotation: existing methods depend heavily on RNA experimental evidence, require costly alignment and filtering, and cannot directly scan raw DNA for candidate circRNAs, even though plant circRNA splice sites often do not follow the canonical GT-AG motifs common in human splicing and plant circRNAs are likely under-annotated [2507.08542]. The framework reformulates circRNA discovery as two linked sequence modeling tasks—splicing site detection (SSD) and splicing site pairing (SSP)—and combines a shared transformer-based backbone, local convolutional feature extraction, linear-attention global modeling, and species-specific mixture-of-experts heads to support large-scale plant genome analysis [2507.08542].

## 1. Biological and computational setting

CircFormerMoE is motivated by the observation that circRNA biogenesis is a long-range sequence problem. A circRNA is defined by a back-splicing event, so prediction requires both locating candidate splice sites and determining which sites form a valid circular pair. Prior circRNA identification methods were described as primarily relying on high-throughput RNA sequencing data combined with alignment-based algorithms that detect back-splicing signals; these methods cannot predict circRNAs directly from genomic DNA sequences, incur high computational cost because of alignment and filtering, and are inefficient for large-scale or genome-wide prediction in plants [2507.08542].

The plant setting sharpens these difficulties. Plant circRNA splice sites often lack the canonical GT-AG motif seen in human mRNA splicing, and the paper states that no efficient deep learning model with strong generalization capability currently exists for this setting [2507.08542]. Existing deep learning methods for circRNAs are characterized as operating mainly on short fixed-length sequence classification windows, which is not practical for genome-wide plant annotation because plant chromosomes are large and relevant signals can be separated by very long distances [2507.08542].

CircFormerMoE addresses this by operating in an explicitly genome-oriented manner. Instead of attempting whole-chromosome inference in a single step, it identifies splice sites positionally across long windows and then classifies candidate site pairs as circular or non-circular. This suggests that the framework is designed not merely as a sequence classifier, but as a structured prediction system for plant genome annotation [2507.08542].

## 2. Problem decomposition and dataset construction

The framework is divided into SSD and SSP. In SSD, genomic sequences are scanned and each nucleotide position is labeled as splice-site or non-splice-site. In SSP, the model takes local context around two candidate splice sites, concatenates the two windows with a separator token, and predicts whether the pair forms a circRNA [2507.08542]. This decomposition is presented as the mechanism that makes inference tractable on plant genomes spanning tens of millions to billions of bases, because only relevant subsequences need to be examined rather than entire chromosomes [2507.08542].

The authors constructed separate datasets for SSD and SSP from PlantCircRNA annotations and EnsemblPlants genome FASTA files. The ten evaluated species are Brachypodium distachyon, Brassica juncea, Brassica napus, Cajanus cajan, Glycine max, Gossypium raimondii, Lactuca sativa, Medicago truncatula, Pisum sativum, and Theobroma cacao [2507.08542].

For SSD, the training instances are 5001-bp windows centered on annotated splice sites, with 2500 bases on each side and missing values padded with `N`. Each sequence carries a label list of splice-site positions, for example \([6, 2500, 2566]\), and during training every nucleotide position receives a binary label: \(1\) for circRNA splice-site positions and \(0\) otherwise. Negative samples are drawn from non-splicing regions [2507.08542].

For SSP, the construction uses 1001-bp windows around each splice site, with 500 bp upstream and 500 bp downstream. Positive samples are formed by concatenating paired splice-site sequences from the same circRNA, while negative samples are produced by concatenating unpaired splice-site sequences. A spacer token `MMMMM` is inserted in the middle to mark the junction. The final SSP dataset contains 1,394,981 samples with an approximate positive:negative ratio of 1:10 [2507.08542].

The reported circRNA length distribution across the ten species is strongly skewed toward short-range pairings. Short distance \((<1000\) bases\()\) accounts for 82% (106,626 samples), medium distance (1000–5000 bp) for 6.8% (8,788), long distance (5000–10000 bp) for 3.1% (4,001), and ultra-long distance \((>10000\) bases\()\) for 8.1% (10,563) [2507.08542]. This distribution helps explain why a system must capture both local motifs and long-range dependencies.

## 3. Architectural organization

CircFormerMoE consists of four main components: an input encoder, a local encoder, a global encoder, and species-specific MoE heads [2507.08542]. The input encoder one-hot encodes DNA into five channels for SSD \((A, C, G, T, N)\) and six channels for SSP \((A, C, G, T, N, M)\), then maps the input into a higher-dimensional latent representation [2507.08542].

The local encoder uses multiple 1D convolutional residual blocks to learn short-range sequence patterns. The global encoder uses attention blocks to capture long-range dependencies. After shared encoding, the model produces a hidden feature map \(H \in \mathbb{R}^{L \times D}\), where \(L\) is sequence length and \(D\) is hidden dimension [2507.08542]. Task-specific prediction is then defined as

$$
f = \begin{cases}
f^{SSD}_{s^*}(H), & \text{for Task 1 (SSD)} \\
f^{SSP}_{s^*}(\text{Pool}(H)), & \text{for Task 2 (SSP)}
\end{cases}
$$

where \(s^*\) denotes the known species identity [2507.08542]. For SSD, each species-specific head \(f^{SSD}_{s^*}\) is implemented as a \(1 \times 1\) convolution so that sequence length is preserved and a score is output for every base. For SSP, the model first performs global average pooling over \(H\) and then applies a fully connected species-specific head \(f^{SSP}_{s^*}\) to produce a single classification score [2507.08542].

A central design choice is the meaning of “mixture-of-experts.” In CircFormerMoE, the MoE mechanism is not a soft-routed MoE with learned gating probabilities over multiple experts; it is a hard routing scheme in which each sample is sent to the expert corresponding to its known species identity [2507.08542]. The backbone is shared across species, while specialization is confined to species-specific heads. This allows the backbone to learn common circRNA syntax across plants while the expert heads adapt to divergent splicing patterns [2507.08542].

A useful clarification arises from comparison with other MoE-transformer systems. In MoEformer for automatic modulation recognition, by contrast, expert fusion is dense and input-dependent, using Softmax gating over multiple temporal-scale experts [2606.09085]. CircFormerMoE therefore uses “mixture-of-experts” in a species-conditioned specialization sense rather than in the adaptive multi-expert routing sense common in other transformer literature. This distinction helps avoid a common misconception that all MoE models employ learned probabilistic routing.

## 4. Long-range modeling, training strategy, and metrics

The global encoder adopts lightweight multi-head self-attention based on linear attention, specifically Performer-style attention with random feature maps and the FAVOR+ algorithm [2507.08542]. Standard self-attention is written as

$$
\text{Attention}(Q, K, V) = \text{softmax}(QK^T)V
$$

and is replaced by the kernelized approximation

$$
\text{Attention}(Q,K,V)=\phi(Q)\left(\phi(K)^TV\right),
$$

where \(\phi\) is a random feature map [2507.08542]. The paper gives

$$
\phi(x) = \exp(w^Tx - \mathbf{1}\times \mathbf{1}),
$$

with \(w \sim \mathcal{N}(0, I)\), and \(m\) random features controlling approximation quality [2507.08542]. This reduces the scaling from quadratic to linear in sequence length and is presented as necessary because SSD windows are length 5001.

Training explicitly addresses heavy class imbalance. For SSD, the positive class weight is set to 285 for the first 10 epochs. For SSP, the positive class weight is 10 for the first 5 epochs. Uniform sampling across species is used in each epoch so that each batch contains an equal number of samples from each species, mitigating inter-species imbalance [2507.08542]. The model is pretrained across all species, initially on SSD, for 500 epochs, and is compared with a baseline CircFormer that has the same architecture but without species-specific MoE heads. After pretraining, the model is fine-tuned separately on each species for both SSD and SSP [2507.08542].

The SSD evaluation metrics are average precision (AP), average recall (AR), average F1-score, and top-k accuracy. Top-k accuracy is defined as

$$
\text{Top-k} = \frac{\text{top } k \text{ predicted pos } \cap \text{ true splice pos}}{k},
$$

where \(k\) is the number of true splice-site positions in the test set and predicted peaks are obtained by applying a peak detection algorithm to the output probability map [2507.08542]. For SSP, the reported metrics are balanced accuracy, precision, recall, and F1-score, with balanced accuracy defined as

$$
\text{Balanced Accuracy} = \frac{1}{2}\left(\frac{TP}{TP+FN} + \frac{TN}{TN+FP}\right).
$$

These metric choices reflect the rarity of positive labels in both subtasks [2507.08542].

## 5. Empirical results and cross-species behavior

Across all ten species, pretrained CircFormerMoE outperformed the baseline CircFormer on SSD, with a 4.1% higher mean average recall and 2.3% higher mean average precision [2507.08542]. In the main averaged SSD comparison, the reported metrics are as follows:

| Model | AP / AR / F1 | Top-k acc |
|---|---:|---:|
| CircFormer | 0.976 / 0.966 / 0.972 | 0.896 |
| CircFormerMoE | 0.992 / 0.992 / 0.992 | 0.919 |
| SpliceAI | 0.917 / 0.910 / 0.913 | 0.855 |

CircFormerMoE therefore improves over SpliceAI by 7.5% in AP, 8.2% in AR, 7.9% in F1, and 6.4% in top-k accuracy, while the non-MoE CircFormer also outperforms SpliceAI [2507.08542]. The paper attributes the gain from CircFormer to CircFormerMoE to the species-specific heads, which capture divergent plant splicing patterns more effectively than a single shared head.

Per-species SSD plots show precision and recall above 99% for all species, with top-k accuracy above 90% for nine of the ten species. Pisum sativum is the main exception, with somewhat lower performance, which the authors suggest may reflect noise or label inaccuracy in that dataset [2507.08542].

For SSP, the model is initialized from the SSD-pretrained backbone and then trained as a classifier over concatenated splice-site pairs. The reported overall SSP performance is accuracy 97.35%, precision 96.68%, recall 92.27%, and F1-score 94.41% [2507.08542]. Lactuca sativa achieves perfect scores of 100% in accuracy, precision, recall, and F1, while Glycine max, Pisum sativum, and Cajanus cajan also reach near-perfect performance. Brassica juncea and Theobroma cacao show somewhat lower recall and F1, likely due to noisier annotations or more variable sequence patterns [2507.08542].

A central empirical claim is that the combination of a shared backbone and species-specific experts generalizes well across plant species. Training on all species together with uniform batch sampling is said to help the model learn robust cross-species features, and the MoE approach is reported to perform better than training separate models per species because the shared backbone captures general circRNA syntax and the expert heads specialize in species-specific variation [2507.08542]. This suggests that CircFormerMoE occupies an intermediate point between fully pooled multi-species training and entirely species-isolated models.

## 6. Interpretability, discovery use, and limitations

The paper includes gradient-based saliency analysis to identify sequence patterns used for prediction. For an input DNA sequence \(x \in \mathbb{R}^{L \times C}\), the saliency score at each position is computed by summing absolute input gradients across channels:

$$
\sum_{j=1}^{C}\left|\frac{\partial y}{\partial x_{i,j}}\right|.
$$

The species-level average saliency profile is then

$$
S = \frac{1}{N}\sum_{k=1}^{N}S^{(k)}.
$$

These analyses show that regions immediately surrounding the splice site are most important, although multiple peaks can appear across the \(\pm 2500\) bp context [2507.08542]. To analyze local sequence preference, the authors build saliency-weighted sequence logos in a \(\pm 50\) bp window around splice sites. The reported logos indicate that continuous stretches of adenine and thymine are particularly important, and polyA/polyT stretches of length at least 5 bases within a 100-base window around splice sites occur in 43.7% of real sequences, compared with about 17.4% in random sequences [2507.08542].

This interpretability analysis is used to argue that the model has learned biologically meaningful local sequence patterns, especially AT-rich regions near plant circRNA junctions, rather than only memorizing labels [2507.08542]. The species-wise logos also reveal both conserved and species-specific importance patterns, which is consistent with the architectural decision to combine a shared backbone with species-specific heads.

Beyond benchmark performance, CircFormerMoE is presented as a discovery tool. After training, it was used to identify substantially more circular RNA splice sites than previously reported, and the abstract states that it is capable of discovering previously unannotated circRNAs [2507.08542]. The authors also claim practical speed: on a consumer GPU such as an NVIDIA RTX 4080, inference can exceed one million bases per second, supporting large-scale plant genome annotation [2507.08542].

The limitations noted in the discussion are comparatively specific. SSP still has some speed bottlenecks, and the implementation could benefit from broader and larger datasets [2507.08542]. The authors also observe that human splice-site signals may differ in a way that makes individual splice-site detection harder in humans than in plants, suggesting that circRNA biogenesis signals may depend on joint context rather than individual site features alone [2507.08542]. A plausible implication is that the CircFormerMoE design is particularly well matched to plant genomic circRNA prediction, while cross-domain transfer to other taxa may require rethinking both data construction and inductive bias.

## 7. Position within plant circRNA modeling

CircFormerMoE’s main contribution is the integration of three ideas into a single genome-sequence-based plant circRNA framework: decomposition into SSD and SSP, a lightweight transformer-style backbone with linear attention, and species-specific MoE heads [2507.08542]. Taken together, these components are intended to process long genomic sequences, capture both local motifs and long-range dependencies, and specialize predictions to the sequence idiosyncrasies of individual plant species.

Its empirical comparison against CircFormer without MoE heads isolates the effect of species-specific specialization, while comparison against SpliceAI situates the model relative to a widely used splice prediction baseline originally developed for human sequences [2507.08542]. The resulting picture is that CircFormerMoE is not simply a generic transformer applied to DNA, but a task-structured and taxon-aware model designed for plant circRNA annotation from raw genomic sequence.

In methodological terms, CircFormerMoE also clarifies a broader point about MoE usage in biological sequence modeling. The model does not employ learned soft gating over experts, but instead uses hard routing by known species identity [2507.08542]. This makes its MoE layer function as a species-conditional specialization mechanism. In contrast, other transformer-based MoE systems such as MoEformer use dense, input-dependent routing to combine multiple experts for different temporal resolutions [2606.09085]. CircFormerMoE therefore represents a biologically indexed variant of expert specialization, with the expert assignment determined externally by species labels rather than internally by a learned gate.

Overall, CircFormerMoE is positioned as a fast and accurate computational method for large-scale circRNA discovery in plants, expanding annotation beyond RNA-seq-based pipelines while providing interpretable evidence about sequence contexts that may govern plant circRNA biogenesis [2507.08542].

Source: https://www.emergentmind.com/topics/circformermoe