---
title: Self-Consistency Training with RL (Self-CTRL)
url: https://www.emergentmind.com/topics/self-consistency-training-with-reinforcement-learning-self-ctrl
type: topic
---

# Self-Consistency Training with RL (Self-CTRL)

Searching arXiv for Self-CTRL and closely related self-consistency RL papers.
Self-Consistency Training with Reinforcement Learning (Self-CTRL) is a reinforcement-learning framework for aligning a language model’s self-explanations with its actual behavior on related inputs. The method treats explanation–behavior agreement as an explicit trainable objective rather than a post hoc desideratum. In Self-CTRL, a model generates a self-explanation on a meta-level prompt and a behavior on a paired object-level prompt, and an external evaluator scores whether the explanation predicts or justifies the behavior. Reinforcement learning then updates explanations to better predict behavior, behavior to better match explanations, or both [2606.18327]. The framework is motivated by the observation that language models can produce plausible statements about their own policies while behaving inconsistently on downstream tasks, undermining auditability, trust, and controllability [2606.18327].

## 1. Conceptual basis and scope

Self-CTRL adopts a simulatability view of explanation. In this view, a good explanation is one that helps a simulator predict a model’s behavior on related inputs. The paper operationalizes this using paired prompts. A meta-level prompt \(x_*\) asks the model to articulate a rule, latent bias, or policy, while an object-level prompt \(x\) elicits the behavior to be predicted. The explanation \(y_*\) and behavior \(y\) are sampled independently from the same model under different contexts, so the target is not intra-answer coherence but cross-context consistency between separate outputs [2606.18327].

This framing distinguishes Self-CTRL from several adjacent paradigms. It is not self-consistency decoding, which aggregates multiple sampled answers at inference time without changing model parameters. It is not standard supervised explanation tuning, because the consistency target is defined relationally between generated explanations and generated behaviors. It is also distinct from many cycle-consistency or round-trip methods, although those are close methodological relatives in other domains. For example, Round-Trip Reinforcement Learning (RTRL) in chemistry optimizes consistency between forward and inverse tasks using reverse-model likelihood as reward [2510.01527], and CoVo uses consistency of intermediate reasoning states across trajectories as an intrinsic reward for reasoning RL [2506.08745]. Self-CTRL differs in that the paired objects are explicitly “self-explanation” and “behavior” rather than two task inverses or multiple reasoning paths [2606.18327].

A central implication is that Self-CTRL treats transparency itself as behaviorally grounded. A self-explanation is not rewarded for sounding reasonable; it is rewarded only insofar as it predicts what the model later does, or behavior is rewarded only insofar as it matches what the model said it would do [2606.18327].

## 2. Objective and optimization directions

The framework defines a scalar consistency function
\[
\phi(y_*, y; x_*, x),
\]
which is high when explanation \(y_*\) and behavior \(y\) are consistent for the paired prompts \((x_*, x)\). The full expected consistency objective is
\[
\mathcal{L}(x, x_*) = - \mathbb{E}_{y \sim p_\theta(\cdot \mid x),\; y_* \sim p_\theta(\cdot \mid x_*)} \big[\phi(y_*, y;x_*, x)\big].
\]
Because \(\phi\) can be non-differentiable and depends on sampled outputs, Self-CTRL uses reinforcement learning. The policy gradient is written as
\[
\nabla \mathcal{L}(x, x_*) = - \mathbb{E}_{y, y_*} \Big[ \big( \nabla \log p_\theta(y \mid x) + \nabla \log p_\theta(y_* \mid x_*) \big)\phi(y_*, y;x_*, x) \Big]
\]
[2606.18327].

The paper defines two one-sided variants. In explanation training, the objective is
\[
\mathcal{L}_\text{explanation}(x, x_*) = - \mathbb{E}_{y, y_*} \big[\log p_\theta(y_* \mid x_*)\,\phi(y_*, y;x_*, x)\big],
\]
so only the explanation policy is updated. In behavior training, the objective is
\[
\mathcal{L}_\text{behavior}(x, x_*) = - \mathbb{E}_{y, y_*} \big[\log p_\theta(y \mid x)\,\phi(y_*, y;x_*, x)\big],
\]
so only behavior is updated. Mixed training interpolates between them:
\[
\mathcal{L} = \lambda \, \mathcal{L}_\text{behavior} + (1-\lambda)\,\mathcal{L}_\text{explanation},
\]
with \(\lambda \in [0,1]\) controlling the direction of adaptation [2606.18327].

This directional split is one of the method’s defining features. Explanation-side updates target faithfulness of model self-report. Behavior-side updates target controllability by pushing actions toward articulated principles. Mixed updates seek simultaneous convergence of both [2606.18327].

## 3. Reinforcement-learning procedure

Self-CTRL is implemented with a GRPO-style estimator. For each paired prompt \((x, x_*)\), the model samples a group of \(k\) candidate explanations from \(p_\theta(\cdot \mid x_*)\) and a group of \(k\) candidate behaviors from \(p_\theta(\cdot \mid x)\). To score explanations, the method chooses a reference behavior \(y^{(\mathrm{ref})}\) and evaluates each explanation candidate with
\[
r^{(i)} = \phi(y_*^{(i)}, y^{(\mathrm{ref})}; x_*, x).
\]
To score behaviors, it chooses a reference explanation \(y_*^{(\mathrm{ref})}\) and evaluates each behavior candidate with
\[
r^{(i)} = \phi(y_*^{(\mathrm{ref})}, y^{(i)}; x_*, x).
\]
The reference is selected as the highest-probability completion,
\[
y^{(\mathrm{ref})} = \arg\max_y \log p_\theta(y \mid x),
\]
and analogously for \(y_*^{(\mathrm{ref})}\) [2606.18327].

Group-normalized advantages are then computed and used for policy-gradient updates on the explanation side, behavior side, or both. The paper notes that averaging over all \(k \times k\) pairs is possible, but it uses the reference-based estimator for efficiency [2606.18327].

This training scheme places Self-CTRL in the broader class of RL methods that use relational rewards across multiple generated outputs. Related designs appear elsewhere in the literature. CoVo constructs intrinsic rewards from cross-trajectory consistency and volatility of intermediate states [2506.08745]. RTRL converts invertibility across paired tasks into a reverse-likelihood reward [2510.01527]. SCS for multimodal RL shapes rewards using answer stability under perturbation [2511.10648]. Self-CTRL’s distinctive contribution is to apply this style of RL reward to explanation–behavior relations rather than answer agreement or inverse-task recovery [2606.18327].

## 4. Consistency evaluators and domain instantiations

Self-CTRL is intentionally evaluator-agnostic. The framework is defined by the consistency scorer \(\phi\), whose concrete form depends on the domain [2606.18327].

### 4.1 Formal probabilistic reasoning domain

The first domain studies a family of named biased coin samplers. The model is asked to imitate each sampler behaviorally by generating 100 flips, and to explain the sampler by writing a Python function that returns the associated heads–tails probabilities [2606.18327].

For a rollout with \(h\) heads and \(t\) tails, and an explanation that specifies \(p_{\mathrm{heads}}\), the consistency score is the per-flip log-likelihood
\[
\phi(y_*^{(p_{\mathrm{heads})}, y_{h,t}; x_*, x) = \frac{1}{h+t} \left[ h \log p_{\mathrm{heads}} + t \log(1-p_{\mathrm{heads}}) \right].
\]
This is an exact executable evaluator: the explanation induces a Bernoulli model, and the behavior is scored by how well it matches that model [2606.18327].

The experimental design uses 100 biased coins with biases uniformly in \([0.1, 0.9]\), partitioned into 50 fully supervised coins, 40 experimental rollout-only coins used in Self-CTRL training, and 10 held-out rollout-only coins used only for evaluation [2606.18327]. The key question is whether consistency training can recover accurate self-reports of latent bias without direct explanation labels.

### 4.2 Constitutional AI domain

The second domain studies natural-language rules about refusal and compliance behavior. Meta-level prompts ask the model to describe how it handles requests in a category such as stereotypes/discrimination, privacy, harmful activity, or hostile communication. Object-level prompts are concrete user requests from those categories [2606.18327].

Here \(\phi\) is not executable code but an LM-jury score. Eight jurors, corresponding to different philosophical frameworks and anti-frameworks, assess whether the response follows the stated rule. Each juror yields a soft yes/no score
\[
s_j(y_*, y; x) = \frac{P_j(\text{yes}\mid y_*, y, x)} {P_j(\text{yes}\mid y_*, y, x)+P_j(\text{no}\mid y_*, y, x)},
\]
and the final consistency score is
\[
\phi(y_*, y; x) = \frac{1}{8}\sum_{j=1}^8 s_j(y_*, y; x)
\]
[2606.18327].

Because pure consistency can reward degenerate fixed points such as universal refusal, behavior-side runs also add an engagement reward:
\[
r_{\mathrm{beh}} = r_{\mathrm{jury}} + \alpha \, r_{\mathrm{engage}},
\]
with \(\alpha = 0.4\). This penalizes generic refusal when the stated rule does not require refusal for the specific prompt [2606.18327].

## 5. Empirical results

The paper reports two main empirical findings: explanation-side consistency training can substantially improve the faithfulness of self-reports, and behavior-side consistency training can substantially improve alignment while preserving usefulness [2606.18327].

### 5.1 Biased coin results

In the coin domain, before Self-CTRL the model generalizes poorly from behavior to explicit self-report on rollout-only coins. On experimental coins, the articulated biases have \(R^2 = -0.23\) against empirical rollout bias and \(R^2 = -0.40\) against ground-truth latent bias. After explanation-only Self-CTRL on the experimental coins, these improve to \(R^2 = 0.65\) and \(R^2 = 0.60\), respectively [2606.18327].

On the 10 held-out coins, which are never used in consistency training, the model reaches \(R^2 = 0.64\) against empirical rollout bias and \(R^2 = 0.63\) against ground truth. The abstract highlights an improvement from \(R^2 = 0.24\) to \(R^2 = 0.64\) on held-out distributions, matching the generalization of direct ground-truth supervision [2606.18327].

This near-matches an oracle model trained with explicit explanation supervision on the experimental coins, which reaches \(R^2 = 0.62\) against empirical rollout and \(R^2 = 0.69\) against ground truth on held-out coins [2606.18327]. The result suggests that explanation–behavior consistency can recover latent structure without direct labels when the evaluator is exact and the explanation space has executable semantics.

### 5.2 Constitutional AI results

In the constitutional domain, explanation-side Self-CTRL improves how well third-party auditors can predict the model’s refusal/compliance behavior from the model’s generated rules. The abstract reports that a third-party auditor’s refusal prediction improves from \(36\%\) to \(92\%\) when given Self-CTRL-generated rules rather than baseline rules [2606.18327].

Behavior-side Self-CTRL improves alignment directly. The abstract reports that HarmBench failure rate drops from \(15.0\%\) to \(0.5\%\) without substantially increasing refusal on harmless prompts [2606.18327]. The significance is that the model can be trained so that its behavior more faithfully tracks its stated rule while still engaging on benign requests.

Together, these two domains illustrate the bidirectionality of the framework. Explanation training can increase transparency by making self-reports more predictive. Behavior training can increase controllability by making downstream behavior more obedient to self-articulated policies [2606.18327].

## 6. Relation to adjacent methods

Self-CTRL sits at an intersection of several lines of work but is not reducible to any of them.

First, it differs from self-consistency decoding, which aggregates multiple sampled outputs at inference and leaves model weights unchanged. Certified Self-Consistency formalizes majority-vote self-consistency as mode recovery and shows how label-free post-training can sharpen the answer distribution [2511.17472]. Self-CTRL instead performs parameter updates using a reward defined over explanation–behavior consistency [2606.18327].

Second, it differs from self-correction RL methods such as SCoRe, which train multi-turn self-revision behavior using improvement reward across attempts [2409.12917], and from ICRL, which trains a solver and critic jointly so critique-induced gains become critique-free capability [2605.15224]. These methods focus on revision quality and internalization of critique, whereas Self-CTRL focuses on the alignment between what a model says about itself and what it subsequently does [2606.18327].

Third, it is closely related in spirit to RL methods that turn consistency into a reward signal. RTRL operationalizes round-trip consistency between inverse chemistry tasks [2510.01527]. CoVo operationalizes consistency and volatility of reasoning trajectories [2506.08745]. SCS uses answer stability under perturbations as a reward-shaping signal in multimodal RL [2511.10648]. Self-CTRL generalizes this pattern to explanation–behavior pairs, where the reward is not answer agreement per se but whether an explanation behaviorally simulates the model [2606.18327].

Fourth, it differs from critique-training frameworks such as CTRL, where a critic is rewarded for producing feedback that improves a fixed generator [2502.03492]. CTRL enforces behavioral usefulness of critique, but it does not target consistency between a model’s abstract self-description and its downstream behavior in paired contexts [2502.03492].

## 7. Training design and regularization

The paper emphasizes that consistency training requires anchoring to avoid degenerate solutions. In the coin setting, explanation training includes a behavior-preserving NLL regularizer weighted by \(\lambda_{\mathrm{KL}} = 0.3\), implemented by sampling one rollout from the SFT-initialized policy and penalizing its negative log-likelihood under the current policy [2606.18327]. This approximates a forward-KL anchor that preserves behavior while improving explanation faithfulness.

In constitutional runs with \(\lambda \in \{0,1\}\), the side not being updated is similarly anchored with a forward-KL approximation [2606.18327]. Behavior-side runs also include continued instruction tuning on Nemotron SFT data to prevent blanket-refusal collapse and preserve general instruction-following [2606.18327].

These regularizers are methodologically important. The raw consistency objective has trivial optima. In constitutional AI, a model that states “I always refuse” and indeed always refuses could score high on explanation–behavior consistency while being unusable. The auxiliary engagement reward and KL-style anchoring are therefore integral to making behavior-side Self-CTRL practical [2606.18327].

## 8. Interpretation, significance, and limitations

Self-CTRL’s main significance is that it turns explanation–behavior faithfulness into a direct optimization target. The method shows that self-descriptions can become materially more predictive of behavior, and that behavior can be steered toward stated principles, without requiring full direct supervision of both sides [2606.18327].

The framework is especially compelling when explanations have executable semantics, as in the biased coin task, because \(\phi\) becomes exact and latent variables are directly measurable. In natural-language constitutional settings, the results are still strong, but they depend on the quality of the LM-jury and auxiliary reward design [2606.18327].

Several limitations follow from the reported setup. First, the framework depends on having a meaningful paired structure \((x_*, x)\) such that the explanation prompt really ought to constrain the behavior prompt. Second, the choice of \(\phi\) is domain-specific and may itself be noisy or biased, especially when instantiated with LM judges rather than exact programs [2606.18327]. Third, pure consistency is not sufficient in safety settings because it can reward degenerate fixed points; auxiliary engagement or task rewards are required [2606.18327]. A plausible implication is that behavior-side Self-CTRL is best viewed as a consistency-constrained alignment method rather than a standalone alignment criterion.

A further implication is that Self-CTRL optimizes a specific notion of transparency: behavioral predictiveness of self-report. This suggests strong relevance for auditing and governance, but it does not by itself guarantee truthfulness about internal mechanisms beyond what is behaviorally simulatable. In the constitutional case, the model learns to produce rules that help a third-party predictor anticipate refusal behavior; this is a practical transparency gain, but it remains a behavioral criterion rather than an interpretability guarantee in the mechanistic sense [2606.18327].

## 9. Legacy and broader context

Self-CTRL should be understood as part of a broader shift from using consistency merely as an evaluation heuristic to using it as a trainable RL objective. Earlier and parallel work established that self-consistency can function as a reward or pseudo-reward in multiple settings: inverse-task recoverability in chemistry [2510.01527], self-rewarded reasoning based on trajectory convergence [2506.08745], critique usefulness for code revision [2502.03492], multimodal answer stability under perturbation [2511.10648], and label-free self-training from majority-vote agreement [2505.21444]. Self-CTRL extends this trend to the relation between self-explanation and action [2606.18327].

Its distinctive contribution is therefore not simply “consistency with RL,” but a specific recipe for training models whose articulated principles and measured behaviors are more tightly coupled. In the explanation direction, this improves model auditability. In the behavior direction, it improves rule-following alignment. The paper presents these as two sides of the same objective: explanations should predict behavior, and behavior should instantiate explanations [2606.18327].

Source: https://www.emergentmind.com/topics/self-consistency-training-with-reinforcement-learning-self-ctrl