Conversation Forests in Reinforcement Learning

• Conversation Forests are a reinforcement learning framework using branched architectures to generate and compare multiple dialogue trajectories in multi-turn settings.
• The approach employs sibling-relative rewards and depth-wise normalization to stabilize credit assignment across conversational turns, enhancing diagnostic accuracy.
• Experimental results with models like Llama-3.1-8B-Instruct show SCF improves performance by up to 4% over linear architectures in clinical diagnostic tasks.

Savage Conversation Forests (SCF) are a reinforcement learning (RL) framework designed for fine-tuning LLMs in complex multi-turn conversational tasks, most notably in clinical diagnostic settings. Unlike linear conversation architectures—which treat each doctor-patient dialogue as a single trajectory—SCF induces a branched structure at every conversational turn, explicitly enabling the model to compare and learn from alternative conversational continuations and their downstream effects on diagnostic outcomes. SCF leverages sibling-relative rewards and depth-wise normalization to provide a decomposed and stable training signal, facilitating more effective credit assignment across multiple turns and promoting diagnostic strategies that better reflect real-world physician decision making (Savage, 5 Jul 2025).

1. Architecture: Branched vs. Linear Dialogue Modeling

Linear RL conversation architectures generate a single model response at each turn, forming a trajectory (root → u₁ → u₂ → ... → u_D → reward). These methods, including Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO), treat each trajectory in isolation, without access to "counterfactual" outcomes—what might have occurred if a different action had been taken at an earlier step. In contrast, the SCF framework generates B > 1 alternative doctor responses at every turn, with each of these branches subsequently receiving a frozen patient model's reply. This process continues recursively for a predefined depth D, producing a combinatorial explosion of conversational paths (total leaves = B^D), all rooted in a shared initial prompt but diverging immediately after the first doctor response.

This architecture enables explicit comparison of alternative dialogic paths. For example, at the first branch point, the model is tasked with evaluating how each of the B possible questions at turn 1 influences entire downstream dialogues, including the ultimate diagnostic reward. This structure is unavailable to linear methods, which can neither synthesize nor propagate these counterfactual rewards through the policy gradient (Savage, 5 Jul 2025).

2. Training Procedure and Algorithmic Details

The SCF training loop consists of building the conversation forest, computing rewards at each leaf, propagating rewards back up the tree, and performing policy updates (in GRPO or PPO style). The pseudocode, with branching factor B and depth D, operates as follows:

• At each 'doctor' turn, B model continuations are sampled independently.
• The patient LLM (frozen) generates a single response for each branch.
• When all dialogues reach the maximum length (2D+1 turns), each leaf is scored by a frozen "diagnostician" and a grader (GPT-4), producing a scalar reward r_i per dialogue.
• Sibling-relative rewards and depth-wise normalized advantages are computed at each depth, providing the decomposition necessary for inter-turn credit assignment.
• The policy is updated by summing (negative) advantage-weighted log-probabilities over all leaf responses, with an optional KL penalty to a reference policy (Savage, 5 Jul 2025).

3. Mathematical Formulation of the Objective

For a conversation forest with branching factor B and depth D, there are L = B^D leaves. The sibling group Sₚ at depth d for parent node p contains all leaves sharing that immediate predecessor. The sibling-relative reward for each leaf i at depth d is

$$\tilde r_i^d = r_i^d - \frac{1}{|S_p|}\sum_{j \in S_p} r_j^d$$

where r_i^d is the reward allocated to leaf i.

For normalization at each depth,

$$A_i^d = \frac{\tilde r_i^d - \mu_d}{\sigma_d + \epsilon}$$

with μ_d and σ_d^2 denoting mean and variance of all sibling-relative rewards at depth d, respectively.

The SCF policy is optimized via the GRPO-style surrogate loss

$$\mathcal{L}(\theta) = - \sum_{i=1}^L \; A_i^{d(i)} \,\log \pi_\theta(a_i \mid s_i) + \lambda_{\rm KL} \sum_{i=1}^L \mathrm{KL}[\pi_\theta(\cdot\mid s_i)\| \pi_{\rm ref}(\cdot\mid s_i)]$$

where s_i is the prefix history for leaf i, a_i the final doctor utterance, and π_ref a static reference policy. The optional KL penalty can be omitted (the GRPO default is λ_kl=0) (Savage, 5 Jul 2025).

4. Experimental Design and Hyperparameters

The experimental evaluation deploys SCF in simulating multi-turn doctor-patient interviews using the following protocol:

• Trainable doctor model: Llama-3.1-8B-Instruct or Mistral-8B-Instruct.
• Patient simulator: frozen Llama-3.1-8B-Instruct, case-conditioned.
• Diagnostician and reward grader: frozen GPT-4 instances.
• Data: MedQA-derived, multi-turn clinical roleplays.
• B = 4 (branching factor), D = 2 (doctor turns), for 16 leaves per forest.
• Linear SCF variant uses B = 1 and repeats the process for 10 independent trajectories per case.
• Optimization hyperparameters: AdamW, learning rate 2e-7, max generation 20 tokens/turn, optional KL weight λ_{KL} = 0.01 in ablations (Savage, 5 Jul 2025).

Evaluation metrics include percent of total possible points per case (average graded score), mean training reward, and loss per case.

Model	Base	Linear SCF	SCF w/ Branching
Llama-3.1-8B-Instruct	45.1%	45.4%	49.2%
Mistral-8B-Instruct	45.5%	36.2%	48.8%

Branched SCF consistently yields higher mean training rewards and achieves more negative loss earlier than both base and linear variants, with improvements of approximately 3–4 percentage points in diagnostic accuracy. No formal statistical significance is reported (Savage, 5 Jul 2025).

5. Advantage Propagation and Credit Assignment

SCF's architectural innovation allows explicit decomposition of reward attribution across turns. By analyzing rewards among siblings at each branch, the model isolates the impact of specific decisions (e.g., "Did asking question Q₁ᵃ at turn 1 lead to superior outcomes relative to Q₁ᵇ?"). The sibling-relative and depth-normalized advantage computation stabilizes gradient magnitudes even for early turns, counteracting signal decay typically observed in deep RL for language (Savage, 5 Jul 2025).

Such decomposed credit assignment is fundamental for procedural skill acquisition, such as developing efficient diagnostic schemas where early-section questions funnel patient responses towards informative diagnostic conclusions. This structure is unavailable in linear methods, which only back-propagate reward from the outcome of a single trajectory.

6. Implementation Considerations and Limitations

Branch generation in SCF is managed through careful sampling strategies:

• Doctor model: B completions per turn, independently sampled.
• Patient model: one completion per branch, temperature T=0.7.
• Diagnostician and graders: single completions per leaf.

To ensure computational parity, B=1 linear SCF samples multiple independent trajectories, while B>1 branching generates fewer forests but more branches per tree (e.g., 4 forests × 4 branches = 16 leaves, matching ~20 total completions per case) (Savage, 5 Jul 2025).

The primary limitation is computational: branching incurs O(B^D) complexity per root prompt, constraining current experiments to shallow depths (D=2). Increasing D or applying the framework to larger models may further amplify its benefits but remains an open research direction.

Pre-finetuning with supervised (SFT) methods or imposing a KL penalty in linear SCF provided negligible gains, indicating that branching, rather than data augmentation or conservative updates, is the principal driver of SCF's success. No linearized approach matched the improvements observed with branching (Savage, 5 Jul 2025).

7. Implications, Observations, and Future Directions

SCF demonstrates the significance of branching architectures for RL-based fine-tuning of LLMs in settings where inter-turn dependencies critically impact downstream outcomes. Improvements are attributed to:

• Enabling direct comparison of counterfactual branches.
• Robust decomposed credit assignment for each turn via sibling-relative rewards.
• Gradient normalization across dialog depths.
• Qualitative emergence of more focused, "funnel-like" questioning strategies.

A plausible implication is that SCF, though evaluated on medical dialog, could generalize to other domains where conversational structure is deep and causal (e.g., education, legal reasoning). The authors note that extending to greater conversational depths or broader domains is a promising direction, but computational demands are a key practical obstacle (Savage, 5 Jul 2025).