scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation

Abstract: Simultaneous measurement of multiple omics modalities in single cells enables researchers to gain a more comprehensive understanding of cellular states and regulatory mechanisms. However, due to high experimental costs, significant noise, and incomplete modality coverage, a variety of computational methods for modality translation have emerged in recent years. Despite the development of translation models, there is still a lack of systematic benchmark evaluation in terms of datasets, evaluation metrics, and influencing factors. To address this, we present scTranslation, a comprehensive benchmark for single-cell multi-omics modality translation tasks. It includes diverse translation datasets, integrates state-of-the-art models, and provides a comprehensive evaluation metrics. In addition, we assess model performance under different scenarios, such as feature selection, feature quality, and few-shot settings. These factors significantly affect model performance but have rarely been systematically studied before. Leveraging this benchmark, we conduct a large-scale study of current methods, report many insightful findings that open up new possibilities for future development. The benchmark is open-sourced to facilitate future research. The code is anonymously released at https://github.com/Bunnybeibei/scTranslation.

• The paper establishes an extensible benchmark for rigorously evaluating deep learning models in single-cell multi-omics modality translation.
• It employs multi-faceted metrics, including clustering, regression, and distribution similarity, to reveal trade-offs between biological resolution and quantitative fidelity.
• Empirical findings highlight the impact of model architecture and feature quality, guiding future improvements in imputation and network inference for multi-omics data.

scTranslation: A Task-Oriented Benchmark for Single-Cell Multi-Omics Modality Translation

Introduction

Single-cell technologies have accelerated the characterization of regulatory programs through direct measurement of DNA accessibility (ATAC-seq), gene expression (RNA-seq), and protein abundance (Protein-seq) in individual cells, forming the empirical basis for regulatory network modeling (Figure 1). However, high costs, data sparsity, and protocol complexity limit routine experimental multiplexing, making computational cross-modality translation a critical approach for inferring regulatory or phenotypic layers from partially observed data. Despite rapid advances in deep learning–based translators, the absence of a systematic benchmark impedes robust algorithmic comparison. "scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation" (2606.03906) establishes a unified and extensible evaluation framework, enabling stratified assessment of state-of-the-art translation models across diverse biological axes, metrics, and noise regimes.

Figure 1: Schematic representation of the central dogma and the major single-cell modalities, highlighting ATAC-seq, RNA-seq, and Protein-seq.

Task Definition and Challenges

Cross-modality translation is formally posed as the mapping of a cell-by-feature matrix from source to target omics (e.g., RNA$\rightarrow$ATAC, ATAC$\rightarrow$RNA, RNA$\leftrightarrow$Protein) using paired measurements, seeking to preserve both biological specificity and quantitative fidelity (Figure 2). The dominant challenges addressed include:

• Dataset diversity and generalization: Most models are validated on limited, technology-restricted cohorts, undermining generalization claims.
• Multi-faceted evaluation: Single-metric reporting (e.g., mean squared error) fails to capture critical aspects such as cell-type resolution, local structure, or global batch effects.
• Robustness to biological/technical uncertainty: Data quality, feature sparsity, and sample scarcity are not adequately interrogated by extant benchmarks.

  Figure 2: Formalization of each omic layer as a cell-by-feature matrix, with the translation task requiring accurate bidirectional imputation.

Datasets and Benchmark Structure

The authors curate eight public datasets covering a broad spectrum of organisms (mouse, human), organs (brain, PBMC, bone marrow), protocols (10x Multiome, SNARE-seq, sci-CAR, CITE-seq, scCAT-seq), and multi-omics layers (ATAC, RNA, Protein). The selection follows the "6M" criteria (technique, species, organ, modality, scale, developmental stage) to enable rigorous cross-context evaluation.

scTranslation organizes benchmarking in a modular pipeline (Figure 3):

• Dataset ingestion and standardization (including joint samples and feature preprocessing)
• Task definition (directional translation: ATAC$\leftrightarrow$RNA, RNA$\leftrightarrow$Protein)
• Model evaluation under a controlled protocol (stratified cross-validation; fixed hyperparameters)
• Metric reporting across three axes: clustering (NMI, ARI, AMI, HOM), regression (PCC, MSE), and distributional similarity (MMD, LISI)

  Figure 3: Overview of the scTranslation benchmark’s modular structure, from datasets through evaluation metrics.

Model Landscape and Methodological Spectrum

Baseline models are classified according to underlying generative approaches:

• Autoencoders (AE): BABEL, scPair rely on encoder-decoder architectures with joint latent spaces and adversarial alignment to support bidirectional translation.
• Variational Autoencoders (VAE): JAMIE and scButterfly employ probabilistic latent representations and explicit latent-space alignment, with JAMIE supporting missing modality imputation and scButterfly leveraging dual-aligned, semi-supervised VAE frameworks.
• Distribution-based Models: multiDGD (Gaussian mixture modeling), scDiffusion-X (diffusion process + cross-attention with optional label conditioning). These prioritize global data distribution matching.

Comprehensive Metrics for Biological and Quantitative Fidelity

The benchmark implements three metric groups:

• Clustering: NMI, ARI, AMI, HOM—probes conservation of cell identity and biological structure post-translation.
• Regression: PCC, MSE—quantifies the fidelity of feature-level prediction and reconstructive accuracy.
• Distribution: MMD, LISI—assesses global and local distributional concordance, critical for downstream integration and batch correction.

Empirical Results and Influencing Factors

Supervised experiments demonstrate several key findings (Figure 4):

• No single universal optimum: Models excel on different metric subsets and datasets, with VAE-based models (scButterfly, JAMIE) consistently achieving higher NMI/ARI (clustering consistency) and lowest MSE in ATAC$\rightarrow$RNA tasks. However, AE models (BABEL, scPair) often yield superior PCC, indicating strong trend capturing but sometimes high amplitude error.
• Feature selection: Expansion from 500 to 3,000 highly variable genes enhances clustering and regression performance, but performance plateaus and sometimes declines beyond this point due to noise accretion.
• Feature quality: Increasing feature sparsity (simulating dropout) degrades clustering and regression metrics, but, paradoxically, improves global distribution metrics (MMD), as predictions drift toward mean-averaged solutions favoring population-level alignment.
• Few-shot learning: All models show degradation with extreme data subsampling, but diffusion-based methods (scDiffusion-X) demonstrate resistance, attributed to implicit data manifold modeling via denoising diffusion.
• Directionality and protein modality: RNA$\leftrightarrow$Protein translation is more volatile, with high variance in MSE due to sensitivity to normalization and task support; VAE-based models offer better bidirectional robustness than AEs.

  Figure 4: Performance trends for six models as a function of feature quality, selection, and few-shot training on the PBMC dataset.

Practical and Theoretical Implications

scTranslation exposes intrinsic trade-offs between biological resolution and quantitative or global concordance, substantiating the claim that current approaches have not converged on a uniformly optimal regime. The results suggest the importance of robust architecture selection in routine biological applications such as multi-omic imputation, legacy dataset augmentation, and high-confidence network inference. Notably, translation performance is highly contingent on both input feature quality and the task’s data regime, indicating the need for adaptive, context-aware models in the future.

In the context of foundation models for biomedicine and generative AI, these findings underscore the necessity for large-scale, context-diverse multi-omic integration during pre-training and for evaluating model robustness under experimental noise, feature corruption, and data starvation. The limitations of current metrics (e.g., MSE overemphasizing amplitude, LISI missing rare population fidelity) call for future research into biologically grounded, composite evaluation frameworks.

Conclusion

scTranslation delivers a rigorous and extensible benchmarking system for single-cell multi-omics modality translation. By integrating task diversity, comprehensive metrics, and systematic robustness studies, it provides the necessary infrastructure for reproducible comparison, illuminates decisive factors affecting model performance, and outlines the major gaps remaining in neural-based cross-modality inference. The benchmark is well-positioned to catalyze the development of robust, generalizable, and biologically interpretable translation models in high-dimensional omic data analysis.

---

Reference:

"scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation" (2606.03906).