CircFormerMoE: Plant circRNA Prediction

• The paper introduces CircFormerMoE, a framework that decomposes circRNA prediction into splice site detection (SSD) and splicing site pairing (SSP) tasks.
• It integrates a transformer-based backbone with species-specific MoE heads to capture both local motifs and long-range dependencies in plant genomes.
• Evaluated across ten plant species, the method outperforms baselines by achieving high precision and recall, enabling efficient, genome-wide circRNA annotation.

CircFormerMoE is an end-to-end deep learning framework for predicting plant circular RNAs directly from genomic DNA sequence rather than from RNA-seq alignment pipelines. It was introduced to address a specific bottleneck in plant circRNA annotation: existing methods depend heavily on RNA experimental evidence, require costly alignment and filtering, and cannot directly scan raw DNA for candidate circRNAs, even though plant circRNA splice sites often do not follow the canonical GT-AG motifs common in human splicing and plant circRNAs are likely under-annotated (Jiang, 11 Jul 2025). The framework reformulates circRNA discovery as two linked sequence modeling tasks—splicing site detection (SSD) and splicing site pairing (SSP)—and combines a shared transformer-based backbone, local convolutional feature extraction, linear-attention global modeling, and species-specific mixture-of-experts heads to support large-scale plant genome analysis (Jiang, 11 Jul 2025).

1. Biological and computational setting

CircFormerMoE is motivated by the observation that circRNA biogenesis is a long-range sequence problem. A circRNA is defined by a back-splicing event, so prediction requires both locating candidate splice sites and determining which sites form a valid circular pair. Prior circRNA identification methods were described as primarily relying on high-throughput RNA sequencing data combined with alignment-based algorithms that detect back-splicing signals; these methods cannot predict circRNAs directly from genomic DNA sequences, incur high computational cost because of alignment and filtering, and are inefficient for large-scale or genome-wide prediction in plants (Jiang, 11 Jul 2025).

The plant setting sharpens these difficulties. Plant circRNA splice sites often lack the canonical GT-AG motif seen in human mRNA splicing, and the paper states that no efficient deep learning model with strong generalization capability currently exists for this setting (Jiang, 11 Jul 2025). Existing deep learning methods for circRNAs are characterized as operating mainly on short fixed-length sequence classification windows, which is not practical for genome-wide plant annotation because plant chromosomes are large and relevant signals can be separated by very long distances (Jiang, 11 Jul 2025).

CircFormerMoE addresses this by operating in an explicitly genome-oriented manner. Instead of attempting whole-chromosome inference in a single step, it identifies splice sites positionally across long windows and then classifies candidate site pairs as circular or non-circular. This suggests that the framework is designed not merely as a sequence classifier, but as a structured prediction system for plant genome annotation (Jiang, 11 Jul 2025).

2. Problem decomposition and dataset construction

The framework is divided into SSD and SSP. In SSD, genomic sequences are scanned and each nucleotide position is labeled as splice-site or non-splice-site. In SSP, the model takes local context around two candidate splice sites, concatenates the two windows with a separator token, and predicts whether the pair forms a circRNA (Jiang, 11 Jul 2025). This decomposition is presented as the mechanism that makes inference tractable on plant genomes spanning tens of millions to billions of bases, because only relevant subsequences need to be examined rather than entire chromosomes (Jiang, 11 Jul 2025).

The authors constructed separate datasets for SSD and SSP from PlantCircRNA annotations and EnsemblPlants genome FASTA files. The ten evaluated species are Brachypodium distachyon, Brassica juncea, Brassica napus, Cajanus cajan, Glycine max, Gossypium raimondii, Lactuca sativa, Medicago truncatula, Pisum sativum, and Theobroma cacao (Jiang, 11 Jul 2025).

For SSD, the training instances are 5001-bp windows centered on annotated splice sites, with 2500 bases on each side and missing values padded with N. Each sequence carries a label list of splice-site positions, for example $[6, 2500, 2566]$, and during training every nucleotide position receives a binary label: $1$ for circRNA splice-site positions and $0$ otherwise. Negative samples are drawn from non-splicing regions (Jiang, 11 Jul 2025).

For SSP, the construction uses 1001-bp windows around each splice site, with 500 bp upstream and 500 bp downstream. Positive samples are formed by concatenating paired splice-site sequences from the same circRNA, while negative samples are produced by concatenating unpaired splice-site sequences. A spacer token MMMMM is inserted in the middle to mark the junction. The final SSP dataset contains 1,394,981 samples with an approximate positive:negative ratio of 1:10 (Jiang, 11 Jul 2025).

The reported circRNA length distribution across the ten species is strongly skewed toward short-range pairings. Short distance $(<1000$ bases$)$ accounts for 82% (106,626 samples), medium distance (1000–5000 bp) for 6.8% (8,788), long distance (5000–10000 bp) for 3.1% (4,001), and ultra-long distance $(>10000$ bases$)$ for 8.1% (10,563) (Jiang, 11 Jul 2025). This distribution helps explain why a system must capture both local motifs and long-range dependencies.

3. Architectural organization

CircFormerMoE consists of four main components: an input encoder, a local encoder, a global encoder, and species-specific MoE heads (Jiang, 11 Jul 2025). The input encoder one-hot encodes DNA into five channels for SSD $(A, C, G, T, N)$ and six channels for SSP $(A, C, G, T, N, M)$, then maps the input into a higher-dimensional latent representation (Jiang, 11 Jul 2025).

The local encoder uses multiple 1D convolutional residual blocks to learn short-range sequence patterns. The global encoder uses attention blocks to capture long-range dependencies. After shared encoding, the model produces a hidden feature map $H \in \mathbb{R}^{L \times D}$, where $1$0 is sequence length and $1$1 is hidden dimension (Jiang, 11 Jul 2025). Task-specific prediction is then defined as

$1$2

where $1$3 denotes the known species identity (Jiang, 11 Jul 2025). For SSD, each species-specific head $1$4 is implemented as a $1$5 convolution so that sequence length is preserved and a score is output for every base. For SSP, the model first performs global average pooling over $1$6 and then applies a fully connected species-specific head $1$7 to produce a single classification score (Jiang, 11 Jul 2025).

A central design choice is the meaning of “mixture-of-experts.” In CircFormerMoE, the MoE mechanism is not a soft-routed MoE with learned gating probabilities over multiple experts; it is a hard routing scheme in which each sample is sent to the expert corresponding to its known species identity (Jiang, 11 Jul 2025). The backbone is shared across species, while specialization is confined to species-specific heads. This allows the backbone to learn common circRNA syntax across plants while the expert heads adapt to divergent splicing patterns (Jiang, 11 Jul 2025).

A useful clarification arises from comparison with other MoE-transformer systems. In MoEformer for automatic modulation recognition, by contrast, expert fusion is dense and input-dependent, using Softmax gating over multiple temporal-scale experts (Wang et al., 8 Jun 2026). CircFormerMoE therefore uses “mixture-of-experts” in a species-conditioned specialization sense rather than in the adaptive multi-expert routing sense common in other transformer literature. This distinction helps avoid a common misconception that all MoE models employ learned probabilistic routing.

4. Long-range modeling, training strategy, and metrics

The global encoder adopts lightweight multi-head self-attention based on linear attention, specifically Performer-style attention with random feature maps and the FAVOR+ algorithm (Jiang, 11 Jul 2025). Standard self-attention is written as

$1$8

and is replaced by the kernelized approximation

$1$9

where $0$0 is a random feature map (Jiang, 11 Jul 2025). The paper gives

$0$1

with $0$2, and $0$3 random features controlling approximation quality (Jiang, 11 Jul 2025). This reduces the scaling from quadratic to linear in sequence length and is presented as necessary because SSD windows are length 5001.

Training explicitly addresses heavy class imbalance. For SSD, the positive class weight is set to 285 for the first 10 epochs. For SSP, the positive class weight is 10 for the first 5 epochs. Uniform sampling across species is used in each epoch so that each batch contains an equal number of samples from each species, mitigating inter-species imbalance (Jiang, 11 Jul 2025). The model is pretrained across all species, initially on SSD, for 500 epochs, and is compared with a baseline CircFormer that has the same architecture but without species-specific MoE heads. After pretraining, the model is fine-tuned separately on each species for both SSD and SSP (Jiang, 11 Jul 2025).

The SSD evaluation metrics are average precision (AP), average recall (AR), average F1-score, and top-k accuracy. Top-k accuracy is defined as

$0$4

where $0$5 is the number of true splice-site positions in the test set and predicted peaks are obtained by applying a peak detection algorithm to the output probability map (Jiang, 11 Jul 2025). For SSP, the reported metrics are balanced accuracy, precision, recall, and F1-score, with balanced accuracy defined as

$0$6

These metric choices reflect the rarity of positive labels in both subtasks (Jiang, 11 Jul 2025).

5. Empirical results and cross-species behavior

Across all ten species, pretrained CircFormerMoE outperformed the baseline CircFormer on SSD, with a 4.1% higher mean average recall and 2.3% higher mean average precision (Jiang, 11 Jul 2025). In the main averaged SSD comparison, the reported metrics are as follows:

Model	AP / AR / F1	Top-k acc
CircFormer	0.976 / 0.966 / 0.972	0.896
CircFormerMoE	0.992 / 0.992 / 0.992	0.919
SpliceAI	0.917 / 0.910 / 0.913	0.855

CircFormerMoE therefore improves over SpliceAI by 7.5% in AP, 8.2% in AR, 7.9% in F1, and 6.4% in top-k accuracy, while the non-MoE CircFormer also outperforms SpliceAI (Jiang, 11 Jul 2025). The paper attributes the gain from CircFormer to CircFormerMoE to the species-specific heads, which capture divergent plant splicing patterns more effectively than a single shared head.

Per-species SSD plots show precision and recall above 99% for all species, with top-k accuracy above 90% for nine of the ten species. Pisum sativum is the main exception, with somewhat lower performance, which the authors suggest may reflect noise or label inaccuracy in that dataset (Jiang, 11 Jul 2025).

For SSP, the model is initialized from the SSD-pretrained backbone and then trained as a classifier over concatenated splice-site pairs. The reported overall SSP performance is accuracy 97.35%, precision 96.68%, recall 92.27%, and F1-score 94.41% (Jiang, 11 Jul 2025). Lactuca sativa achieves perfect scores of 100% in accuracy, precision, recall, and F1, while Glycine max, Pisum sativum, and Cajanus cajan also reach near-perfect performance. Brassica juncea and Theobroma cacao show somewhat lower recall and F1, likely due to noisier annotations or more variable sequence patterns (Jiang, 11 Jul 2025).

A central empirical claim is that the combination of a shared backbone and species-specific experts generalizes well across plant species. Training on all species together with uniform batch sampling is said to help the model learn robust cross-species features, and the MoE approach is reported to perform better than training separate models per species because the shared backbone captures general circRNA syntax and the expert heads specialize in species-specific variation (Jiang, 11 Jul 2025). This suggests that CircFormerMoE occupies an intermediate point between fully pooled multi-species training and entirely species-isolated models.

6. Interpretability, discovery use, and limitations

The paper includes gradient-based saliency analysis to identify sequence patterns used for prediction. For an input DNA sequence $0$7, the saliency score at each position is computed by summing absolute input gradients across channels:

$0$8

The species-level average saliency profile is then

$0$9

These analyses show that regions immediately surrounding the splice site are most important, although multiple peaks can appear across the $(<1000$0 bp context (Jiang, 11 Jul 2025). To analyze local sequence preference, the authors build saliency-weighted sequence logos in a $(<1000$1 bp window around splice sites. The reported logos indicate that continuous stretches of adenine and thymine are particularly important, and polyA/polyT stretches of length at least 5 bases within a 100-base window around splice sites occur in 43.7% of real sequences, compared with about 17.4% in random sequences (Jiang, 11 Jul 2025).

This interpretability analysis is used to argue that the model has learned biologically meaningful local sequence patterns, especially AT-rich regions near plant circRNA junctions, rather than only memorizing labels (Jiang, 11 Jul 2025). The species-wise logos also reveal both conserved and species-specific importance patterns, which is consistent with the architectural decision to combine a shared backbone with species-specific heads.

Beyond benchmark performance, CircFormerMoE is presented as a discovery tool. After training, it was used to identify substantially more circular RNA splice sites than previously reported, and the abstract states that it is capable of discovering previously unannotated circRNAs (Jiang, 11 Jul 2025). The authors also claim practical speed: on a consumer GPU such as an NVIDIA RTX 4080, inference can exceed one million bases per second, supporting large-scale plant genome annotation (Jiang, 11 Jul 2025).

The limitations noted in the discussion are comparatively specific. SSP still has some speed bottlenecks, and the implementation could benefit from broader and larger datasets (Jiang, 11 Jul 2025). The authors also observe that human splice-site signals may differ in a way that makes individual splice-site detection harder in humans than in plants, suggesting that circRNA biogenesis signals may depend on joint context rather than individual site features alone (Jiang, 11 Jul 2025). A plausible implication is that the CircFormerMoE design is particularly well matched to plant genomic circRNA prediction, while cross-domain transfer to other taxa may require rethinking both data construction and inductive bias.

7. Position within plant circRNA modeling

CircFormerMoE’s main contribution is the integration of three ideas into a single genome-sequence-based plant circRNA framework: decomposition into SSD and SSP, a lightweight transformer-style backbone with linear attention, and species-specific MoE heads (Jiang, 11 Jul 2025). Taken together, these components are intended to process long genomic sequences, capture both local motifs and long-range dependencies, and specialize predictions to the sequence idiosyncrasies of individual plant species.

Its empirical comparison against CircFormer without MoE heads isolates the effect of species-specific specialization, while comparison against SpliceAI situates the model relative to a widely used splice prediction baseline originally developed for human sequences (Jiang, 11 Jul 2025). The resulting picture is that CircFormerMoE is not simply a generic transformer applied to DNA, but a task-structured and taxon-aware model designed for plant circRNA annotation from raw genomic sequence.

In methodological terms, CircFormerMoE also clarifies a broader point about MoE usage in biological sequence modeling. The model does not employ learned soft gating over experts, but instead uses hard routing by known species identity (Jiang, 11 Jul 2025). This makes its MoE layer function as a species-conditional specialization mechanism. In contrast, other transformer-based MoE systems such as MoEformer use dense, input-dependent routing to combine multiple experts for different temporal resolutions (Wang et al., 8 Jun 2026). CircFormerMoE therefore represents a biologically indexed variant of expert specialization, with the expert assignment determined externally by species labels rather than internally by a learned gate.

Overall, CircFormerMoE is positioned as a fast and accurate computational method for large-scale circRNA discovery in plants, expanding annotation beyond RNA-seq-based pipelines while providing interpretable evidence about sequence contexts that may govern plant circRNA biogenesis (Jiang, 11 Jul 2025).