PerturbCellRL: Verifier-Guided Reinforcement Learning for Single-Cell Perturbation Prediction

Abstract: Single-cell perturbation models can reduce costly wet-lab screening by predicting how cells respond transcriptionally to interventions. While recent generative models improve population-level prediction, individual generated cells are not explicitly checked for biological consistency. We introduce PerturbCellRL, a reinforcement learning (RL) framework that post-trains a pretrained single-cell transcriptomic generator using a suite of cell-level verifiers as rewards. These verifiers define four rewards: Pearson top-k similarity, RMSE top-k proximity, DE Spearman, and Pathway activity. The Pathway activity verifier rewards cells whose pathway responses match known perturbation biology. We evaluate PerturbCellRL on multiple genetic and chemical perturbation benchmarks. Across these benchmarks, PerturbCellRL improves over the pretrained flow-matching generator on reward-aligned evaluation metrics and a held-out evaluation metric. Moreover, PerturbCellRL remains competitive with state-of-the-art methods on population-level metrics. Together, these results frame trustworthy single-cell prediction as verifier-guided generative alignment, moving beyond matching expression distributions toward predictions whose single-cell perturbation effects are explicitly checked for biological consistency.

• The paper introduces a verifier-guided reinforcement learning framework that integrates biologically grounded reward functions to improve single-cell perturbation prediction.
• It employs a suite of verifiers, including Pearson Top-k, RMSE, DE Spearman, and pathway activity, to ensure cell-level biological consistency.
• Empirical results show that PerturbCellRL outperforms baseline models while preserving population-level accuracy and enhancing test-time scalability.

Verifier-Guided Reinforcement Learning for Single-Cell Perturbation Prediction: The PerturbCellRL Framework

Motivation and Problem Statement

Single-cell transcriptomics enables precise characterization of cellular state changes following genetic or chemical perturbations. A central computational biology task involves predicting these transcriptional responses in silico, aiming to alleviate the cost and throughput limitations of wet-lab screening. While deep generative models, particularly flow-matching models such as scDFM, have advanced the modeling of conditional single-cell response distributions, they primarily optimize for population-level concordance. This neglects cell-centric measures of biological consistency, leading to plausible macroscopic distributions despite biologically implausible single-cell profiles.

This paper identifies and addresses two fundamental limitations in prior work:

1. Distributional-Only Training: Existing models primarily optimize population statistics (e.g., pseudobulk means), which fail to enforce biological realism at the level of individual cells.
2. Absence of Explicit Cell-Level Biological Verification: There is no mechanism to ensure that generated cell-level profiles respond to perturbations in a manner consistent with known pathway activity or gene regulatory logic.

Proposed Method: The PerturbCellRL Framework

PerturbCellRL introduces a framework for verifier-guided reinforcement learning (RL) atop pretrained flow-matching single-cell perturbation generators. The central innovation is to define a suite of cell-level, potentially non-differentiable, biological reward functions—referred to as verifiers—and utilize them as optimization signals for RL-based post-training. The framework integrates reward-driven RL objectives with a KL-divergence constraint regularizing against the pretrained generative dynamics, reducing the propensity for reward hacking and excessive drift.

Verifier Suite

The four verifiers encapsulate diverse aspects of biological and transcriptional integrity:

1. Pearson Top-$k$ Similarity: Measures alignment in the direction of expression changes between generated cells and real target cells under the same perturbation.
2. RMSE Top-$k$ Proximity: Assesses local manifold closeness to target-cell distributions, encouraging heterogeneity and avoiding mode collapse.
3. DE Spearman: Evaluates correlation in ranked log fold changes of differentially expressed genes, quantifying concordance of perturbation response at the gene ranking level.
4. Pathway Activity Reward: Leverages curated pathway signatures (e.g., PROGENy) and manually or empirically derived pathway annotations to reward cells whose pathway response (directionality and confidence) matches the known effect of a perturbation. Notably, this is a reference-free verifier applicable at test time.

RL Algorithmic Backbone

PerturbCellRL adopts the DiffusionNFT [zheng2025diffusionnft] RL algorithm, operating on the forward process of the pretrained flow-matching model. Generated candidate rollouts are scored by the verifier suite. Contrastive updates are applied to move the conditional generative policy toward higher reward regions, with KL regularization ensuring similarity to the original diffusion (scDFM) policy. This approach efficiently leverages non-differentiable, biologically rich objectives and circumvents complications associated with intractable sample likelihoods in modern generative flows.

Inference-Time Test-Time Scaling

The reference-free pathway verifier allows for test-time scaling (best-of-$N$ selection), where, at inference, multiple candidate cells are generated for a given perturbation and control state, pathwise evaluated, and the candidate with maximal pathway reward is returned. This boosts practical output alignment even in the absence of ground-truth target data, and its performance continues to improve monotonically with the number of candidates sampled.

Empirical Validation

Comprehensive benchmarks are conducted on highly curated datasets, including the Norman and ComboSciPlex panels, featuring both single and combinatorial perturbations. Evaluation protocols employ both additive and double-gene holdout splits to assess generalization.

Single-Cell-Level Improvements

PerturbCellRL consistently outperforms the scDFM base generator in all four verifier-aligned rewards:

• Increases are observed in Pearson top-$k$, RMSE top-$k$, DE Spearman, and pathway activity, indicating more biologically plausible single-cell outputs.
• Gains on single-cell Discrimination Score (DS), a held-out metric not used for RL optimization, suggest that RL improvements are not limited to metric gaming but generalize to unseen biological consistency measures.

Population-Level Metrics

Despite explicit optimization at the single-cell level, PerturbCellRL maintains state-of-the-art population-level accuracy:

• In most cases, it matches or surpasses scDFM and other baselines (GEARS, CPA, STATE, CellFlow) in MAE, Pearson $\Delta$, MMD, and Energy Distance.
• Critically, no degradation is observed in held-out metrics, such as DS, MMD, or Energy, underscoring the stability of the RL alignment process and the efficacy of KL regularization.

Test-Time Scaling

The test-time use of the pathway verifier for best-of-$N$ generation demonstrates strong, monotonic improvements in pathway-aligned metrics with increasing sample count per inference. Notably, the population-level pathway reward can be substantially increased without additional model retraining, though excessive $N$ can pull the distribution away from population mean metrics (a trade-off that can be managed depending on the application focus).

Visualization and Case Studies

UMAP projections substantiate that post-trained models more faithfully reproduce the local structure of true perturbed populations. Predictions from PerturbCellRL exhibit tighter overlap with real target manifolds compared to the base model, even in challenging combinatorial perturbation cases.

Theoretical and Practical Implications

This work reconceptualizes model alignment in the context of biological generative modeling, framing trustworthy prediction as a problem of verifier-guided generative alignment. By introducing transparent, biologically meaningful reward functions, the approach enables rapid model prototyping, modular reward engineering, and systematic evaluation grounded in scientific constraints. The reference-free property of the pathway verifier critically lowers barriers to deployment in real-world inference, where ground-truth perturbation data for novel interventions is inherently unavailable.

A pivotal insight is that reinforcement learning–based post-training, anchored to verifiers and regularized via policy KL, meaningfully improves the actionable plausibility of single-cell predictions without sacrificing macroscopic distributional fidelity.

Limitations and Future Directions

A current limitation is the dependence of some verifiers (especially pathway activity) on the breadth and accuracy of external annotations, limiting coverage for less-characterized perturbations and combinatorial settings. Additionally, the pathway reward is presently evaluated only for single-gene interventions due to annotation bottlenecks. Broader, community-wide annotation efforts and automated literature mining could mitigate this.

Future directions include development of additional reference-free verifiers—such as cell-type classifiers or domain-specific integrity checks, systematic generalization across species and cell types, and prospective experimental validation of high-scoring predictions. This opens avenues for RL-driven, semi-supervised discovery in virtual cell modeling, drug prioritization, and genotype-phenotype mapping.

Conclusion

PerturbCellRL constitutes a rigorously designed, verifier-guided RL framework for single-cell perturbation prediction, addressing the critical gap of biological consistency at the level of individual cells. The integration of biologically motivated reward functions into generative post-training yields outputs with both enhanced plausibility and competitive distributional accuracy. As the field advances toward comprehensive virtual cell simulations, such approaches will be central in aligning powerful generative models with both the realities of cellular biology and the practical imperatives of biomedical research.

---

Reference: "PerturbCellRL: Verifier-Guided Reinforcement Learning for Single-Cell Perturbation Prediction" (2606.27752)