Self-Consistency Training with RL (Self-CTRL)

• The paper demonstrates that training models with paired self-explanations and behaviors significantly improves auditability and behavioral alignment.
• Self-CTRL uses paired meta-level and object-level prompts to generate explanations and actions, linking them with a trainable consistency score.
• Reinforcement learning updates explanation, behavior, or both, with empirical results showing improved predictive accuracy and safer, more accountable outputs.

Searching arXiv for Self-CTRL and closely related self-consistency RL papers. Self-Consistency Training with Reinforcement Learning (Self-CTRL) is a reinforcement-learning framework for aligning a LLM’s self-explanations with its actual behavior on related inputs. The method treats explanation–behavior agreement as an explicit trainable objective rather than a post hoc desideratum. In Self-CTRL, a model generates a self-explanation on a meta-level prompt and a behavior on a paired object-level prompt, and an external evaluator scores whether the explanation predicts or justifies the behavior. Reinforcement learning then updates explanations to better predict behavior, behavior to better match explanations, or both (Pres et al., 16 Jun 2026). The framework is motivated by the observation that LLMs can produce plausible statements about their own policies while behaving inconsistently on downstream tasks, undermining auditability, trust, and controllability (Pres et al., 16 Jun 2026).

1. Conceptual basis and scope

Self-CTRL adopts a simulatability view of explanation. In this view, a good explanation is one that helps a simulator predict a model’s behavior on related inputs. The paper operationalizes this using paired prompts. A meta-level prompt $x_*$ asks the model to articulate a rule, latent bias, or policy, while an object-level prompt $x$ elicits the behavior to be predicted. The explanation $y_*$ and behavior $y$ are sampled independently from the same model under different contexts, so the target is not intra-answer coherence but cross-context consistency between separate outputs (Pres et al., 16 Jun 2026).

This framing distinguishes Self-CTRL from several adjacent paradigms. It is not self-consistency decoding, which aggregates multiple sampled answers at inference time without changing model parameters. It is not standard supervised explanation tuning, because the consistency target is defined relationally between generated explanations and generated behaviors. It is also distinct from many cycle-consistency or round-trip methods, although those are close methodological relatives in other domains. For example, Round-Trip Reinforcement Learning (RTRL) in chemistry optimizes consistency between forward and inverse tasks using reverse-model likelihood as reward (Kong et al., 1 Oct 2025), and CoVo uses consistency of intermediate reasoning states across trajectories as an intrinsic reward for reasoning RL (Zhang et al., 10 Jun 2025). Self-CTRL differs in that the paired objects are explicitly “self-explanation” and “behavior” rather than two task inverses or multiple reasoning paths (Pres et al., 16 Jun 2026).

A central implication is that Self-CTRL treats transparency itself as behaviorally grounded. A self-explanation is not rewarded for sounding reasonable; it is rewarded only insofar as it predicts what the model later does, or behavior is rewarded only insofar as it matches what the model said it would do (Pres et al., 16 Jun 2026).

2. Objective and optimization directions

The framework defines a scalar consistency function

$\phi(y_*, y; x_*, x),$

which is high when explanation $y_*$ and behavior $y$ are consistent for the paired prompts $(x_*, x)$. The full expected consistency objective is

$$\mathcal{L}(x, x_*) = - \mathbb{E}_{y \sim p_\theta(\cdot \mid x),\; y_* \sim p_\theta(\cdot \mid x_*)} \big[\phi(y_*, y;x_*, x)\big].$$

Because $\phi$ can be non-differentiable and depends on sampled outputs, Self-CTRL uses reinforcement learning. The policy gradient is written as

$x$0

(Pres et al., 16 Jun 2026).

The paper defines two one-sided variants. In explanation training, the objective is

$x$1

so only the explanation policy is updated. In behavior training, the objective is

$x$2

so only behavior is updated. Mixed training interpolates between them: $x$3 with $x$4 controlling the direction of adaptation (Pres et al., 16 Jun 2026).

This directional split is one of the method’s defining features. Explanation-side updates target faithfulness of model self-report. Behavior-side updates target controllability by pushing actions toward articulated principles. Mixed updates seek simultaneous convergence of both (Pres et al., 16 Jun 2026).

3. Reinforcement-learning procedure

Self-CTRL is implemented with a GRPO-style estimator. For each paired prompt $x$5, the model samples a group of $x$6 candidate explanations from $x$7 and a group of $x$8 candidate behaviors from $x$9. To score explanations, the method chooses a reference behavior $y_*$0 and evaluates each explanation candidate with

$y_*$1

To score behaviors, it chooses a reference explanation $y_*$2 and evaluates each behavior candidate with

$y_*$3

The reference is selected as the highest-probability completion,

$y_*$4

and analogously for $y_*$5 (Pres et al., 16 Jun 2026).

Group-normalized advantages are then computed and used for policy-gradient updates on the explanation side, behavior side, or both. The paper notes that averaging over all $y_*$6 pairs is possible, but it uses the reference-based estimator for efficiency (Pres et al., 16 Jun 2026).

This training scheme places Self-CTRL in the broader class of RL methods that use relational rewards across multiple generated outputs. Related designs appear elsewhere in the literature. CoVo constructs intrinsic rewards from cross-trajectory consistency and volatility of intermediate states (Zhang et al., 10 Jun 2025). RTRL converts invertibility across paired tasks into a reverse-likelihood reward (Kong et al., 1 Oct 2025). SCS for multimodal RL shapes rewards using answer stability under perturbation (Wang et al., 13 Nov 2025). Self-CTRL’s distinctive contribution is to apply this style of RL reward to explanation–behavior relations rather than answer agreement or inverse-task recovery (Pres et al., 16 Jun 2026).

4. Consistency evaluators and domain instantiations

Self-CTRL is intentionally evaluator-agnostic. The framework is defined by the consistency scorer $y_*$7, whose concrete form depends on the domain (Pres et al., 16 Jun 2026).

4.1 Formal probabilistic reasoning domain

The first domain studies a family of named biased coin samplers. The model is asked to imitate each sampler behaviorally by generating 100 flips, and to explain the sampler by writing a Python function that returns the associated heads–tails probabilities (Pres et al., 16 Jun 2026).

For a rollout with $y_*$8 heads and $y_*$9 tails, and an explanation that specifies $y$0, the consistency score is the per-flip log-likelihood

$y$1

This is an exact executable evaluator: the explanation induces a Bernoulli model, and the behavior is scored by how well it matches that model (Pres et al., 16 Jun 2026).

The experimental design uses 100 biased coins with biases uniformly in $y$2, partitioned into 50 fully supervised coins, 40 experimental rollout-only coins used in Self-CTRL training, and 10 held-out rollout-only coins used only for evaluation (Pres et al., 16 Jun 2026). The key question is whether consistency training can recover accurate self-reports of latent bias without direct explanation labels.

4.2 Constitutional AI domain

The second domain studies natural-language rules about refusal and compliance behavior. Meta-level prompts ask the model to describe how it handles requests in a category such as stereotypes/discrimination, privacy, harmful activity, or hostile communication. Object-level prompts are concrete user requests from those categories (Pres et al., 16 Jun 2026).

Here $y$3 is not executable code but an LM-jury score. Eight jurors, corresponding to different philosophical frameworks and anti-frameworks, assess whether the response follows the stated rule. Each juror yields a soft yes/no score

$y$4

and the final consistency score is

$y$5

(Pres et al., 16 Jun 2026).

Because pure consistency can reward degenerate fixed points such as universal refusal, behavior-side runs also add an engagement reward: $y$6 with $y$7. This penalizes generic refusal when the stated rule does not require refusal for the specific prompt (Pres et al., 16 Jun 2026).

5. Empirical results

The paper reports two main empirical findings: explanation-side consistency training can substantially improve the faithfulness of self-reports, and behavior-side consistency training can substantially improve alignment while preserving usefulness (Pres et al., 16 Jun 2026).

5.1 Biased coin results

In the coin domain, before Self-CTRL the model generalizes poorly from behavior to explicit self-report on rollout-only coins. On experimental coins, the articulated biases have $y$8 against empirical rollout bias and $y$9 against ground-truth latent bias. After explanation-only Self-CTRL on the experimental coins, these improve to $\phi(y_*, y; x_*, x),$0 and $\phi(y_*, y; x_*, x),$1, respectively (Pres et al., 16 Jun 2026).

On the 10 held-out coins, which are never used in consistency training, the model reaches $\phi(y_*, y; x_*, x),$2 against empirical rollout bias and $\phi(y_*, y; x_*, x),$3 against ground truth. The abstract highlights an improvement from $\phi(y_*, y; x_*, x),$4 to $\phi(y_*, y; x_*, x),$5 on held-out distributions, matching the generalization of direct ground-truth supervision (Pres et al., 16 Jun 2026).

This near-matches an oracle model trained with explicit explanation supervision on the experimental coins, which reaches $\phi(y_*, y; x_*, x),$6 against empirical rollout and $\phi(y_*, y; x_*, x),$7 against ground truth on held-out coins (Pres et al., 16 Jun 2026). The result suggests that explanation–behavior consistency can recover latent structure without direct labels when the evaluator is exact and the explanation space has executable semantics.

5.2 Constitutional AI results

In the constitutional domain, explanation-side Self-CTRL improves how well third-party auditors can predict the model’s refusal/compliance behavior from the model’s generated rules. The abstract reports that a third-party auditor’s refusal prediction improves from $\phi(y_*, y; x_*, x),$8 to $\phi(y_*, y; x_*, x),$9 when given Self-CTRL-generated rules rather than baseline rules (Pres et al., 16 Jun 2026).

Behavior-side Self-CTRL improves alignment directly. The abstract reports that HarmBench failure rate drops from $y_*$0 to $y_*$1 without substantially increasing refusal on harmless prompts (Pres et al., 16 Jun 2026). The significance is that the model can be trained so that its behavior more faithfully tracks its stated rule while still engaging on benign requests.

Together, these two domains illustrate the bidirectionality of the framework. Explanation training can increase transparency by making self-reports more predictive. Behavior training can increase controllability by making downstream behavior more obedient to self-articulated policies (Pres et al., 16 Jun 2026).

6. Relation to adjacent methods

Self-CTRL sits at an intersection of several lines of work but is not reducible to any of them.

First, it differs from self-consistency decoding, which aggregates multiple sampled outputs at inference and leaves model weights unchanged. Certified Self-Consistency formalizes majority-vote self-consistency as mode recovery and shows how label-free post-training can sharpen the answer distribution (Jiménez et al., 21 Nov 2025). Self-CTRL instead performs parameter updates using a reward defined over explanation–behavior consistency (Pres et al., 16 Jun 2026).

Second, it differs from self-correction RL methods such as SCoRe, which train multi-turn self-revision behavior using improvement reward across attempts (Kumar et al., 2024), and from ICRL, which trains a solver and critic jointly so critique-induced gains become critique-free capability (Lin et al., 13 May 2026). These methods focus on revision quality and internalization of critique, whereas Self-CTRL focuses on the alignment between what a model says about itself and what it subsequently does (Pres et al., 16 Jun 2026).

Third, it is closely related in spirit to RL methods that turn consistency into a reward signal. RTRL operationalizes round-trip consistency between inverse chemistry tasks (Kong et al., 1 Oct 2025). CoVo operationalizes consistency and volatility of reasoning trajectories (Zhang et al., 10 Jun 2025). SCS uses answer stability under perturbations as a reward-shaping signal in multimodal RL (Wang et al., 13 Nov 2025). Self-CTRL generalizes this pattern to explanation–behavior pairs, where the reward is not answer agreement per se but whether an explanation behaviorally simulates the model (Pres et al., 16 Jun 2026).

Fourth, it differs from critique-training frameworks such as CTRL, where a critic is rewarded for producing feedback that improves a fixed generator (Xie et al., 5 Feb 2025). CTRL enforces behavioral usefulness of critique, but it does not target consistency between a model’s abstract self-description and its downstream behavior in paired contexts (Xie et al., 5 Feb 2025).

7. Training design and regularization

The paper emphasizes that consistency training requires anchoring to avoid degenerate solutions. In the coin setting, explanation training includes a behavior-preserving NLL regularizer weighted by $y_*$2, implemented by sampling one rollout from the SFT-initialized policy and penalizing its negative log-likelihood under the current policy (Pres et al., 16 Jun 2026). This approximates a forward-KL anchor that preserves behavior while improving explanation faithfulness.

In constitutional runs with $y_*$3, the side not being updated is similarly anchored with a forward-KL approximation (Pres et al., 16 Jun 2026). Behavior-side runs also include continued instruction tuning on Nemotron SFT data to prevent blanket-refusal collapse and preserve general instruction-following (Pres et al., 16 Jun 2026).

These regularizers are methodologically important. The raw consistency objective has trivial optima. In constitutional AI, a model that states “I always refuse” and indeed always refuses could score high on explanation–behavior consistency while being unusable. The auxiliary engagement reward and KL-style anchoring are therefore integral to making behavior-side Self-CTRL practical (Pres et al., 16 Jun 2026).

8. Interpretation, significance, and limitations

Self-CTRL’s main significance is that it turns explanation–behavior faithfulness into a direct optimization target. The method shows that self-descriptions can become materially more predictive of behavior, and that behavior can be steered toward stated principles, without requiring full direct supervision of both sides (Pres et al., 16 Jun 2026).

The framework is especially compelling when explanations have executable semantics, as in the biased coin task, because $y_*$4 becomes exact and latent variables are directly measurable. In natural-language constitutional settings, the results are still strong, but they depend on the quality of the LM-jury and auxiliary reward design (Pres et al., 16 Jun 2026).

Several limitations follow from the reported setup. First, the framework depends on having a meaningful paired structure $y_*$5 such that the explanation prompt really ought to constrain the behavior prompt. Second, the choice of $y_*$6 is domain-specific and may itself be noisy or biased, especially when instantiated with LM judges rather than exact programs (Pres et al., 16 Jun 2026). Third, pure consistency is not sufficient in safety settings because it can reward degenerate fixed points; auxiliary engagement or task rewards are required (Pres et al., 16 Jun 2026). A plausible implication is that behavior-side Self-CTRL is best viewed as a consistency-constrained alignment method rather than a standalone alignment criterion.

A further implication is that Self-CTRL optimizes a specific notion of transparency: behavioral predictiveness of self-report. This suggests strong relevance for auditing and governance, but it does not by itself guarantee truthfulness about internal mechanisms beyond what is behaviorally simulatable. In the constitutional case, the model learns to produce rules that help a third-party predictor anticipate refusal behavior; this is a practical transparency gain, but it remains a behavioral criterion rather than an interpretability guarantee in the mechanistic sense (Pres et al., 16 Jun 2026).

9. Legacy and broader context

Self-CTRL should be understood as part of a broader shift from using consistency merely as an evaluation heuristic to using it as a trainable RL objective. Earlier and parallel work established that self-consistency can function as a reward or pseudo-reward in multiple settings: inverse-task recoverability in chemistry (Kong et al., 1 Oct 2025), self-rewarded reasoning based on trajectory convergence (Zhang et al., 10 Jun 2025), critique usefulness for code revision (Xie et al., 5 Feb 2025), multimodal answer stability under perturbation (Wang et al., 13 Nov 2025), and label-free self-training from majority-vote agreement (Shafayat et al., 27 May 2025). Self-CTRL extends this trend to the relation between self-explanation and action (Pres et al., 16 Jun 2026).

Its distinctive contribution is therefore not simply “consistency with RL,” but a specific recipe for training models whose articulated principles and measured behaviors are more tightly coupled. In the explanation direction, this improves model auditability. In the behavior direction, it improves rule-following alignment. The paper presents these as two sides of the same objective: explanations should predict behavior, and behavior should instantiate explanations (Pres et al., 16 Jun 2026).