TractOracle-RL: RL for Anatomical Tractography

• TractOracle-RL is a reinforcement learning tractography framework that uses a transformer-based anatomical oracle to score and guide streamline propagation.
• It integrates local diffusion data with global anatomical assessment to dynamically reward and terminate streamlines, addressing false positive generation.
• Empirical evaluations across multiple datasets show robust performance improvements over traditional methods by coupling generation with learned online filtering.

Searching arXiv for TractOracle-RL and closely related tractography papers. I’ll look up the cited arXiv papers to ground the article in current records. TractOracle-RL is a reinforcement-learning-based tractography framework in which streamline propagation is guided not only by local diffusion information but also by a learned anatomical oracle that scores the plausibility of the partially reconstructed streamline. In its original formulation, the oracle is a transformer-based streamline classifier used both as a reward function during RL training and as an online stopping criterion during inference, so the system simultaneously generates and evaluates white-matter streamlines (Théberge et al., 2024). A subsequent robustness study generalized the term to an umbrella family of oracle-guided RL tractography methods, examining alternative RL backbones, broader state encodings, longer training, and iterative oracle updating across five diffusion MRI datasets (Levesque et al., 15 Jul 2025).

1. Conceptual basis and tractography formulation

TractOracle-RL was introduced to address a specific failure mode of prior RL tractography systems: the reward functions used in earlier methods, including Track-to-Learn, primarily encode local geometric and diffusion consistency. These rewards encourage alignment with the local fiber orientation distribution function (fODF) and with the previous direction, which is suitable for smooth local propagation but insufficient to determine whether the whole partially reconstructed streamline is anatomically plausible. The reported consequence is reward gaming: the agent can continue to accrue reward while drifting into anatomically implausible regions or shapes, producing spurious false positive tracts (Théberge et al., 2024).

The framework models tractography as an MDP with state space $S$, action space $A$, transition probability $p(s_{t+1}\mid s_t,a_t)$, and reward $r(s_t,a_t)$. The discounted return is written as

$G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$

The value and action-value functions are defined by

$$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$

and

$$Q_\pi(s, a) = \mathbb{E}_\pi \left[r(s_t, a_t) + G_{t+1} \mid s_t = s, a_t = a\right].$$

Within this formulation, the action $a_t$ is the tracking direction $\vec{u}_t$, rescaled by the environment to a fixed step size $\Delta$, and the state $A$0 contains the fODF at the current position $A$1, the six surrounding fODFs, and the previous 100 tracking directions (Théberge et al., 2024).

The streamline itself is represented as an ordered set

$A$2

and propagation follows

$A$3

or, equivalently in the original tractography paper,

$A$4

Initial states are selected at the WM/GM interface. This formulation preserves the standard iterative structure of streamline tracking while inserting learned anatomical assessment directly into the control loop (Théberge et al., 2024, Levesque et al., 15 Jul 2025).

2. Oracle network, reward design, and anatomical supervision

The central component is TractOracle-Net, a transformer network that takes a streamline as input and outputs a scalar plausibility score in $A$5. The streamline is resampled to a fixed number of points, converted into directions between consecutive coordinates, and prepended with a special SCORE token. The network uses a 32-dimensional embedding, positional encoding, 4 transformer encoder blocks, 4 attention heads per block, and a linear layer plus sigmoid to convert the SCORE token into a scalar plausibility score. The model has about 550k trainable parameters (Théberge et al., 2024).

TractOracle-Net is trained as a regression model on streamlines labeled as plausible versus implausible, with mean squared error as the loss. A threshold of $A$6 is used for classification: score $A$7 indicates plausible, while score $A$8 indicates implausible. Data augmentation includes random flipping of streamlines, random cutting followed by re-resampling to 128 points, and point-wise Gaussian noise (Théberge et al., 2024).

Once trained, the network becomes the reward oracle $A$9. The reward used by TractOracle-RL has two components: a local term based on alignment with the local fODFs and previous direction, and an anatomical term derived from TractOracle-Net. The paper writes the reward as a local term plus $p(s_{t+1}\mid s_t,a_t)$0, with the anatomical indicator defined by

$p(s_{t+1}\mid s_t,a_t)$1

The reported hyperparameters are $p(s_{t+1}\mid s_t,a_t)$2, $p(s_{t+1}\mid s_t,a_t)$3, and $p(s_{t+1}\mid s_t,a_t)$4 (Théberge et al., 2024).

The architectural and algorithmic rationale is that the anatomical prior is applied to the partial streamline itself rather than solely to the next local step. This makes TractOracle-RL not just a generator but also a learned online filter: the agent is shaped toward globally plausible white-matter trajectories instead of only locally admissible ones. A plausible implication is that the framework shifts tractography supervision from purely stepwise geometric consistency toward prefix-level anatomical regularization, which is the specific mechanism by which it targets false-positive reduction (Théberge et al., 2024).

3. Online tracking, termination, and joint generate-and-judge behavior

A distinguishing property of TractOracle-RL is that the same oracle is also used for online termination. During tracking, the environment evaluates the current partial streamline prefix $p(s_{t+1}\mid s_t,a_t)$5. If the score drops below $p(s_{t+1}\mid s_t,a_t)$6 after a minimum number of steps, tracking is terminated early. This learned stopping rule is combined with two standard termination conditions: leaving the WM mask and excessive turning angle between successive steps. The stopping criterion is given as

$p(s_{t+1}\mid s_t,a_t)$7

This means the tracker does not need to wait until the streamline exits the WM mask or violates a curvature rule; it can stop as soon as the learned anatomy prior judges the prefix implausible (Théberge et al., 2024).

The inference pipeline is correspondingly incremental. At each step, the agent observes the local diffusion context $p(s_{t+1}\mid s_t,a_t)$8, the actor proposes a 3D direction $p(s_{t+1}\mid s_t,a_t)$9, the environment converts it to a fixed-length step, the streamline point is updated, the current partial streamline is scored by TractOracle-Net, and that score both contributes to reward and can trigger early stopping (Théberge et al., 2024). The later robustness study describes the same family-level idea in broader terms: an RL tracker generates streamlines step by step, an oracle network evaluates anatomical plausibility, and a reward mechanism adds anatomical feedback to the usual local tracking reward (Levesque et al., 15 Jul 2025).

This online use of the oracle is the basis for the claim that TractOracle simultaneously evaluates and reconstructs WM streamlines. A common misconception is to treat the method as post hoc filtering layered on top of a standard tracker. The published description is narrower and more specific: the oracle is integrated into training-time reward shaping and inference-time stopping, so evaluation and reconstruction occur in the same control loop rather than as two disconnected stages (Théberge et al., 2024).

4. Optimization, datasets, baselines, and evaluation protocols

The original TractOracle-RL implementation uses Soft Actor-Critic. The actor and twin critics have 3 fully connected layers of width 1024; the actor outputs the mean and standard deviation of a 3D Gaussian, and the action is sampled from that Gaussian. The learning rate is $r(s_t,a_t)$0 and the discount factor is $r(s_t,a_t)$1 (Théberge et al., 2024). The later robustness study retains SAC as the baseline optimizer and explicitly formulates its entropy-regularized objective and critic structure, while extending the framework to DroQ and CrossQ variants (Levesque et al., 15 Jul 2025).

The initial paper evaluates on three datasets. ISMRM2015 is a synthetic dataset derived from global tractography on one HCP subject, with 5 tractography algorithms used to generate 100k streamlines each and Tractometer segmentation providing positive and negative examples. BILGIN contains 453 healthy adult participants, with 39 randomly selected subjects used to train TractOracle-Net following the FINTA procedure to obtain callosal fibers. TractoInferno is a multi-site, multi-protocol dataset with 284 subjects; negative examples are generated by running tractography and treating unrecognized streamlines as negatives (Théberge et al., 2024).

The robustness study expands the dataset suite to five benchmarks: ISMRM2015, BILGIN, TractoInferno, HCP, and Penthera-3T. Across datasets, it uses order-6 fODFs, i.e. 28 spherical harmonics coefficients, because order 8 was too expensive and order 4 too coarse (Levesque et al., 15 Jul 2025).

For tractography generation, the reported baselines in the original paper are sd_stream and ifod2 from MRtrix3, and Track-to-Learn. For classification of streamlines, the baselines are Recobundles and FINTA (Théberge et al., 2024). The robustness study compares oracle-guided RL variants not only to classical tractography methods and Track-to-Learn, but also across multiple RL backbones and state encodings, including DroQ-1K, CrossQ-3K, CrossQ-AE, and CrossQ-IRT (Levesque et al., 15 Jul 2025).

Evaluation differs by setting. On ISMRM2015, the papers use Tractometer-style metrics: VC, VB, IC, IB, OL, OR, F1, and NC. On in vivo datasets, where no ground-truth tractography exists, evaluation is based on the number of streamlines recognized by RecobundlesX, extractor_flow, and Verifyber as proxies for anatomical plausibility (Théberge et al., 2024, Levesque et al., 15 Jul 2025).

5. Reported empirical performance and robustness extensions

The original paper reports that TractOracle-Net achieves strong streamline-classification performance. On ISMRM2015, accuracy is $r(s_t,a_t)$2, sensitivity $r(s_t,a_t)$3, precision $r(s_t,a_t)$4, and F1-score $r(s_t,a_t)$5. On BILGIN, accuracy is $r(s_t,a_t)$6, sensitivity $r(s_t,a_t)$7, precision $r(s_t,a_t)$8, and F1-score $r(s_t,a_t)$9. On TractoInferno, TractOracle-Net trained on that dataset reaches $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$0 accuracy on the test split (Théberge et al., 2024).

For tractography itself, the ISMRM2015 results reported for TractOracle are $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$1, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$2, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$3, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$4, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$5, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$6, $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$7, and $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$8. The paper emphasizes $G_t = \sum_{k=t..T}[\gamma^k r(s_{t+k},a_{t+k})].$9 better VC than the second-best method, $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$0 lower IC than the second-best method, and $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$1 lower NC than the second-best method. The abstract summarizes the effect as an improvement of true positive ratios by almost $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$2 and a reduction of $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$3 of false positive ratios on one dataset, together with an increase between $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$4 and $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$5 in the number true positive streamlines on another dataset (Théberge et al., 2024).

On TractoInferno in vivo reconstructions, the number of plausible streamlines recovered by TractOracle-RL is reported as 25,028,287 by Recobundles, 55,761,074 by extractor_flow, and 184,355,412 by TractOracle-Net, compared with 9,309,681, 46,135,363, and 85,447,863 respectively for Track-to-Learn. The paper summarizes this as roughly $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$6 improvement over sd_stream and ifod2 and roughly $$V_\pi(s) = \mathbb{E}_{s \sim \pi} \left[G_t \mid s_t = s\right]$$7 improvement over Track-to-Learn (Théberge et al., 2024).

The later robustness study extends these findings in two directions. First, it reports that oracle-guided RL remains robust across SAC, DroQ, CrossQ, longer training, widened context, and Iterative Reward Training, and that combining an oracle with the RL framework consistently leads to robust and reliable tractography regardless of the specific method or dataset used (Levesque et al., 15 Jul 2025). Second, it introduces Iterative Reward Training (IRT), an RLHF-inspired procedure in which the oracle is repeatedly updated using bundle filtering methods rather than human feedback. The loop is explicitly three-step: train the RL agent with the current oracle until a plateau of about 150–200 episodes, generate tractograms on 5 randomly selected subjects until approximately 250K streamlines are collected, annotate them with RecobundlesX, extractor_flow, or Verifyber, and then fine-tune the oracle after splitting the new data into 80% training, 10% validation, and 10% testing (Levesque et al., 15 Jul 2025).

The IRT schedule reported in the hyperparameter table uses 60 iterations, 5 epochs for the first IRT oracle training, 1 epoch for subsequent IRT oracle training, 250k streamlines per iteration, a maximum dataset size of 4M, oracle batch size 1024, and oracle learning rate 0.0005. The dataset is rebalanced to 50% plausible and 50% implausible before appending new data (Levesque et al., 15 Jul 2025). The robustness study further reports that reducing oracle input from 128 to 64 to 32 points does not hurt accuracy and greatly reduces inference time, leading it to adopt 32-point streamline representations (Levesque et al., 15 Jul 2025).

Setting	Reported result
ISMRM2015 classification	Accuracy 0.97, Sensitivity 0.98, Precision 0.94, F1-score 0.96
ISMRM2015 tractography	VC 88.05 ± 0.35%, IC 11.95 ± 0.35%, F1 57.00 ± 0.47%, NC 0.73 ± 0.12
TractoInferno oracle test split	89.21% accuracy
TractoInferno plausible streamlines for TractOracle-RL	25,028,287 by Recobundles; 55,761,074 by extractor_flow; 184,355,412 by TractOracle-Net

A plausible implication is that much of the framework’s empirical advantage comes from coupling generation with learned online rejection of anatomically implausible prefixes, rather than from substituting one actor-critic optimizer for another. That interpretation is consistent with the 2025 result that the oracle-guided strategy remains beneficial across multiple RL backbones and training regimes (Levesque et al., 15 Jul 2025).

6. Limitations, contested points, and prospective directions

Both papers identify limitations. The original TractOracle paper states that overlap with reference bundles is still not perfect and suggests providing the agent with more global context of the diffusion volume to address tractography’s ill-posedness more fundamentally (Théberge et al., 2024). The robustness study adds that each filtering method has its own biases, so the oracle may emulate a particular filter rather than a general anatomical truth; extractor_flow is harder to learn because it depends on information beyond geometry alone; the state representation remains mostly local; overlap seems to decrease as training extends, suggesting possible edge-avoidance behavior; scaling the oracle or combining multiple filtering methods into a single reward model is promising future work; and validation in pathological cases remains open (Levesque et al., 15 Jul 2025).

These limitations clarify several misconceptions. TractOracle-RL does not remove the ill-posedness of tractography; it changes the supervisory signal by injecting a learned anatomical prior into reward and stopping. It also does not replace classical tractography evaluation with ground truth anatomy in vivo; instead, it uses bundle recognition and plausibility scoring tools such as RecobundlesX, extractor_flow, Verifyber, and TractOracle-Net itself as evaluators or filters (Théberge et al., 2024, Levesque et al., 15 Jul 2025). Likewise, the oracle is not a generic anatomical verifier independent of data provenance, because IRT was introduced precisely to address distributional shift between the oracle’s training data and the RL agent’s generated streamline distribution (Levesque et al., 15 Jul 2025).

Within RL-based tractography, the framework’s enduring significance is the generate-and-judge design: a tracker proposes local propagation directions, while a learned oracle evaluates the evolving streamline prefix and can terminate it early if plausibility collapses. In the original paper, this design is described as a unique method that evaluates and reconstructs WM streamlines at the same time (Théberge et al., 2024). In the robustness paper, the same idea becomes a broader family of oracle-guided RL tractography systems, with IRT as the main mechanism for adapting the reward model as the policy changes (Levesque et al., 15 Jul 2025).