Prost-RL: RL for Prostate Cancer Detection

• Prost-RL is a reinforcement learning framework that reframes prostate cancer detection as a spatially aware, policy-driven inference problem using high-resolution micro-ultrasound.
• It integrates a lightweight adaptation of ProstNFound+ with an explicit spatial policy and noise-robust weak supervision to improve both localization and classification.
• The framework employs Adaptive Policy Optimization to refine attention maps, yielding enhanced AUROC and sensitivity metrics compared to previous methods.

Searching arXiv for Prost-RL and closely related prostate RL papers to ground the article. Prost-RL is a reinforcement learning framework for automated prostate cancer detection in high-resolution micro-ultrasound ($\mu$US) that reframes diagnosis as a spatially aware, policy-driven inference problem: the model learns where to look before decoding, rather than relying on implicit attention over the whole image (Abootorabi et al., 29 Jun 2026). It is built on ProstNFound+, which adapts the MedSAM medical foundation model to $\mu$US with lightweight adapters and a clinical prompt encoder, and it introduces an explicit spatial attention policy, a noise-robust weakly supervised objective for heatmap learning, and an RL fine-tuning stage called Adaptive Policy Optimization (APO) (Abootorabi et al., 29 Jun 2026). In the reported study, Prost-RL was evaluated on 6,607 biopsy cores from 693 patients across five clinical sites and achieved $79.0\pm3.5$ AUROC with $64.6\pm6.3\%$ sensitivity at 80% specificity for core-level detection, together with $79.3\pm5.8$ AUROC for clinically significant prostate cancer classification (Abootorabi et al., 29 Jun 2026).

1. Clinical setting and weak-supervision regime

Prost-RL addresses a setting in which $\mu$US is promising for prostate cancer detection because of its very high spatial resolution, yet the visual signal is subtle, heterogeneous, and heavily confounded by speckle noise and artifacts (Abootorabi et al., 29 Jun 2026). Suspicious patterns can be faint and variable across patients, and interpretation remains highly dependent on clinical experience, producing substantial inter-observer variability. The learning problem is further constrained by the form of supervision that is available in routine biopsy workflows: histopathology is provided at the biopsy-core level rather than as pixel-level lesion annotations, and positive cores may contain only a small fraction of tumor, making naïve dense supervision inappropriate (Abootorabi et al., 29 Jun 2026).

The cohort used to evaluate Prost-RL comprises 6,607 biopsy cores from 693 patients collected prospectively across five clinical sites between 2013 and 2016 under ClinicalTrials.gov trial NCT02079025 (Abootorabi et al., 29 Jun 2026). Images were acquired using the ExactVu micro-ultrasound system, and for each core the frame immediately before needle firing was extracted as a sagittal-plane B-mode image with depth 28 mm and width 46.06 mm (Abootorabi et al., 29 Jun 2026). Each patient typically contributed 10–12 cores. The dataset also includes patient age, PSA, PSA density, anatomical core location encoded as 2D base–apex and medial–lateral coordinates, histopathology in terms of ISUP Grade Group and proportional cancer involvement, together with needle trace masks and whole-gland binary masks (Abootorabi et al., 29 Jun 2026).

The label structure is strongly imbalanced. Positive cores are those with GG $\ge 1$; among these, the paper lists 480 clinically insignificant cases (isPCa, GG $=1$–$2$) and 400 clinically significant cases (csPCa, GG $\ge 3$), for 880 positives total, versus 5,727 benign cores, meaning 86% of the dataset is benign (Abootorabi et al., 29 Jun 2026). This regime motivates a method that can exploit coarse supervision while avoiding the instability of dense prediction targets that would incorrectly assign the same positive label everywhere inside the biopsy region.

2. Architecture and spatial policy

Prost-RL starts from the ProstNFound+ backbone. Given an input image $\mu$0, the encoder produces spatial feature maps

$\mu$1

Instead of passing these features directly to the decoder, Prost-RL inserts a policy network $\mu$2 that predicts a spatial attention distribution over encoder locations (Abootorabi et al., 29 Jun 2026). The policy also receives clinical metadata $\mu$3, which includes age, PSA, PSA density, and anatomical core position. These metadata are embedded into a channel-wise gate

$\mu$4

which modulates the policy-processed visual features through elementwise multiplication (Abootorabi et al., 29 Jun 2026).

A convolutional policy head then produces spatial logits over the feature map. These logits are masked outside the prostate region, so the policy is constrained to attend only to anatomically plausible regions, and then normalized to produce attention weights $\mu$5 (Abootorabi et al., 29 Jun 2026). The description indicates that the policy computes processed features, multiplies them by the spatial attention map, and injects the result back into the backbone representation through residual modulation: $\mu$6 where $\mu$7 is a bias-free $\mu$8 projection followed by GELU (Abootorabi et al., 29 Jun 2026). This residual design preserves the original encoder features $\mu$9 while using the learned policy as a soft spatial prompt.

The modulated embedding $79.0\pm3.5$0 is shared by two downstream heads. A heatmap decoder predicts dense cancer likelihood over the image, and an image-level classifier predicts clinically significant prostate cancer (Abootorabi et al., 29 Jun 2026). The same attention map $79.0\pm3.5$1 therefore supports both localization and core-level risk classification. In the authors’ formulation, interpretability is not merely a by-product of internal attention, but an explicit model output: the policy exposes where it is looking before final decoding (Abootorabi et al., 29 Jun 2026).

3. Weakly supervised objective and localization losses

Weak supervision enters through the heatmap pathway. Ground truth provides only a core-level cancer involvement proportion $79.0\pm3.5$2, and this supervision applies only within the valid spatial region $79.0\pm3.5$3, defined as the intersection of the needle trace and the prostate mask (Abootorabi et al., 29 Jun 2026). ProstNFound+ had already relaxed supervision by assigning the proportion $79.0\pm3.5$4 within this region, but Prost-RL argues that standard cross-entropy still over-trusts noisy labels. For a predicted probability $79.0\pm3.5$5, the cross-entropy gradient is

$79.0\pm3.5$6

which diverges if the model confidently disagrees with a noisy target (Abootorabi et al., 29 Jun 2026).

To address this, Prost-RL uses Symmetric Cross-Entropy (SCE): $79.0\pm3.5$7 with reverse cross-entropy

$79.0\pm3.5$8

and

$79.0\pm3.5$9

The reverse term has bounded gradient

$64.6\pm6.3\%$0

which is independent of the prediction itself (Abootorabi et al., 29 Jun 2026). In experiments, the paper sets $64.6\pm6.3\%$1.

Because proportion-based supervision alone can be satisfied by diffuse predictions, Prost-RL adds entropy minimization over pixels in the valid region: $64.6\pm6.3\%$2 where $64.6\pm6.3\%$3 is the predicted pixel probability and $64.6\pm6.3\%$4 is the binary entropy (Abootorabi et al., 29 Jun 2026). Minimizing this term penalizes uncertain mid-range probabilities and encourages sharp, decisive localization. The paper describes this as “negative-entropy regularization” in the sense that entropy is minimized to sharpen localization (Abootorabi et al., 29 Jun 2026).

The total supervised objective is

$64.6\pm6.3\%$5

where $64.6\pm6.3\%$6 is a balanced cross-entropy on the image/core-level classification head (Abootorabi et al., 29 Jun 2026). This supervised stage provides the warm-up before reinforcement learning.

4. Adaptive Policy Optimization

APO is introduced to refine the spatial policy beyond what supervised weak-label learning can provide. The paper motivates this on two grounds: core-level diagnosis is inherently ranking-like, and the labels provide no direct spatial supervision for which locations correspond to the lesion (Abootorabi et al., 29 Jun 2026). Rather than optimizing only for label fidelity, APO encourages the policy to produce attention patterns that improve malignant-versus-benign ranking.

The reward is defined from in-batch pairwise ranking. Let $64.6\pm6.3\%$7 and $64.6\pm6.3\%$8 denote positive and negative samples in a mini-batch, and let $64.6\pm6.3\%$9 be the predicted cancer probability for sample $79.3\pm5.8$0. Then the per-sample reward is

$79.3\pm5.8$1

This reward lies in $79.3\pm5.8$2 and directly measures local ranking quality within the batch (Abootorabi et al., 29 Jun 2026). The paper additionally states that clinically significant cancers receive a multiplicative bonus $79.3\pm5.8$3, explicitly prioritizing high-risk tumors, although the exact formula for applying this bonus is not separately written out (Abootorabi et al., 29 Jun 2026).

The policy is almost deterministic, so repeated rollouts for the same image would otherwise give identical attention maps and zero reward variance. Prost-RL therefore injects Gaussian noise into the attention logits before the softmax: $79.3\pm5.8$4 where $79.3\pm5.8$5 are the attention logits and $79.3\pm5.8$6 controls exploration (Abootorabi et al., 29 Jun 2026). This produces $79.3\pm5.8$7 stochastic spatial attention maps per image, each corresponding to a slightly different “where to look” configuration. In the reported implementation, RL fine-tuning uses $79.3\pm5.8$8 rollouts per sample with exploration noise scale $79.3\pm5.8$9 (Abootorabi et al., 29 Jun 2026).

The optimization stage borrows from Group Relative Policy Optimization (GRPO)-style ideas, but the paper argues that standard GRPO is poorly suited to the 86% benign class imbalance and the wide variation in case difficulty. APO therefore incorporates Domain-Robust Policy Optimization (DRPO), which performs decoupled reward normalization and hierarchical clustering of samples based first on clinical domain (benign versus cancer) and then empirical difficulty estimated from rollout reward vectors using K-means (Abootorabi et al., 29 Jun 2026). Hierarchical temperature scaling is then applied to the advantages so that gradients from rare cancer cases and hard borderline lesions are upweighted. The exact DRPO loss formula is not printed in the provided paper text, but the role of APO is clear: it is a GRPO-like policy optimization stage augmented with DRPO-style domain- and difficulty-aware normalization (Abootorabi et al., 29 Jun 2026).

During RL fine-tuning, the model jointly optimizes the DRPO policy loss alongside the supervised objectives rather than replacing them (Abootorabi et al., 29 Jun 2026). This preserves heatmap quality and classification stability while nudging the policy toward better spatial attention patterns.

5. Dataset, protocol, and empirical performance

The evaluation protocol follows ProstNFound and uses patient-level five-fold cross-validation with center-stratified folds (Abootorabi et al., 29 Jun 2026). To reduce label ambiguity, benign cores from patients with any malignant core are excluded from training. There is no filtering by involvement threshold and no benign undersampling (Abootorabi et al., 29 Jun 2026). Images are resized to $\mu$0, while masks are resized to $\mu$1 to match attention-map resolution. Core-level scores are computed as the mean heatmap activation within the needle–prostate intersection, whereas csPCa image-level scores come from the classification head (Abootorabi et al., 29 Jun 2026).

The baselines are MedSAM-UNETR, MedSAM-FT, MicroSegNet, ProstNFound, and ProstNFound+, with ProstNFound+ as the strongest baseline and the basis of the Prost-RL backbone (Abootorabi et al., 29 Jun 2026). Supervised training runs for 35 epochs with batch size 8, AdamW, learning rate $\mu$2, encoder learning rate $\mu$3, cosine annealing, and weight decay $\mu$4 (Abootorabi et al., 29 Jun 2026). RL fine-tuning then runs for another 35 epochs with batch size 16. Model selection uses validation AUC on high-involvement ($\mu$5) cores (Abootorabi et al., 29 Jun 2026).

On the main heatmap-decoder core-level detection task, Prost-RL achieves $\mu$6 AUROC and $\mu$7 sensitivity at 80% specificity (Abootorabi et al., 29 Jun 2026). The corresponding ProstNFound+ result is $\mu$8 AUROC and $\mu$9 sensitivity, so the headline gains over the strongest baseline are $\ge 1$0 AUROC and $\ge 1$1 sensitivity points at 80% specificity (Abootorabi et al., 29 Jun 2026). For high-involvement cores, Prost-RL reaches $\ge 1$2 AUROC and $\ge 1$3 sensitivity at 80% specificity; ProstNFound+ reports $\ge 1$4 AUROC and $\ge 1$5 sensitivity (Abootorabi et al., 29 Jun 2026). For csPCa detection via the heatmap decoder, Prost-RL reports $\ge 1$6 AUROC (Abootorabi et al., 29 Jun 2026).

The auxiliary image-level classification head also improves. On all cores, ProstNFound+ achieves $\ge 1$7 AUROC and $\ge 1$8 sensitivity at 80% specificity, while Prost-RL improves to $\ge 1$9 AUROC and $=1$0 sensitivity (Abootorabi et al., 29 Jun 2026). For csPCa versus non-csPCa classification, ProstNFound+ obtains $=1$1 AUROC, $=1$2 sensitivity at 60% specificity, and $=1$3 sensitivity at 80% specificity, whereas Prost-RL improves these to $=1$4 AUROC, $=1$5, and $=1$6, respectively (Abootorabi et al., 29 Jun 2026).

The paper does not provide FLOPs, GPU memory, or wall-clock training time. A plausible implication is that the added RL phase and four rollouts per sample increase training cost relative to supervised-only methods, although the policy itself is described as lightweight (Abootorabi et al., 29 Jun 2026).

6. Ablations, interpretability, and position within prostate reinforcement learning

The ablation results reinforce the proposed decomposition. A supervised-only Prost-RL variant, meaning the attention policy and noise-robust losses without APO, already outperforms ProstNFound and ProstNFound+ across all involvement thresholds (Abootorabi et al., 29 Jun 2026). The full Prost-RL model improves further, and the gap over the supervised-only version is larger at lower cancer involvement thresholds, which the authors interpret as evidence that APO is particularly helping with low-involvement, hard-to-localize cases rather than merely inflating confidence (Abootorabi et al., 29 Jun 2026).

Prompt ablations show that even with no clinical prompts, Prost-RL still outperforms ProstNFound+, suggesting that the gains are not merely due to metadata; adding more prompts improves performance monotonically, indicating complementary value from age, PSA, PSA density, and positional information (Abootorabi et al., 29 Jun 2026). Policy-architecture ablation shows that transformer and convolutional variants perform comparably, supporting the claim that the improvement comes more from the RL training strategy than from a specific policy network form (Abootorabi et al., 29 Jun 2026). Loss-function ablation shows that SCE plus entropy regularization outperforms both the needle-region cross-entropy used in ProstNFound and the MIL-style proportion BCE used in ProstNFound+, confirming that noise robustness matters in this weak-label regime (Abootorabi et al., 29 Jun 2026).

Interpretability is a major claimed advantage. Qualitatively, Prost-RL attention maps and resulting heatmaps are described as sharper and more biopsy-aligned than ProstNFound+ (Abootorabi et al., 29 Jun 2026). For malignant cases, Prost-RL tends to focus on compact regions that correspond to biopsy-confirmed cancer areas, whereas ProstNFound+ often produces diffuse activations; for benign cases, Prost-RL suppresses activation in the needle region more effectively (Abootorabi et al., 29 Jun 2026). The paper explicitly states that the learned policy highlights biopsy-aligned regions and provides “transparent, spatially grounded evidence” alongside risk predictions (Abootorabi et al., 29 Jun 2026).

The limits of the evidence are also clear. The paper does not provide exact segmentation-overlap scores between attention maps and biopsy regions, a formal site-wise breakdown of quantitative performance, or explicit robustness numbers across centers (Abootorabi et al., 29 Jun 2026). It does not include a numerical failure-mode analysis, and extension to 3D volumes is identified as an important future direction (Abootorabi et al., 29 Jun 2026). A plausible implication is that the method’s localization fidelity remains constrained by the granularity of core-level labels, even though APO and the heatmap losses substantially sharpen the spatial output.

Within prostate-focused reinforcement learning, Prost-RL belongs to a distinct lineage. It is not a TRUS boundary-tracing method such as "Boundary-RL: Reinforcement Learning for Weakly-Supervised Prostate Segmentation in TRUS Images" (Yi et al., 2023), nor a crop-based weakly supervised lesion localizer for bi-parametric MRI (Pocius et al., 2024). It is also separate from RL formulations for intervention planning, including template-guided biopsy needle placement (Gayo et al., 2022), automated cryoablation planning (Simangan et al., 5 Sep 2025), and HDR prostate brachytherapy pre-planning (Wang et al., 11 Jun 2025). Prost-RL is specifically an RL-guided, weakly supervised $=1$7US cancer-detection system in which the policy operates as an explicit spatial attention mechanism rather than as a direct mask predictor or a treatment planner (Abootorabi et al., 29 Jun 2026).