Self-CTRL: Self-Consistency Training with Reinforcement Learning

Abstract: LLMs (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from $R^2=0.24$ to $R^2=0.64$ on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from $36\%$ to $92\%$. In the other direction, behavior updates improve alignment, reducing HarmBench failure rate from $15.0\%$ to $0.5\%$ without substantially increasing refusal on harmless prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.

• The paper introduces a dual-agent framework that uses reinforcement learning to enforce consistency between self-generated explanations and observed behaviors.
• It demonstrates significant improvements in explanation-behavior correlation, with R² scores rising from negative values to around 0.60–0.65 on probabilistic inference tasks.
• It offers practical insights for balancing simulatability and safety, enabling scalable audit and alignment strategies without compromising compliance performance.

Self-CTRL: Self-Consistency Training with Reinforcement Learning

Introduction and Motivation

"Self-CTRL: Self-Consistency Training with Reinforcement Learning" (2606.18327) introduces a protocol for optimizing LLMs not just for high-quality outputs, but explicit consistency between their self-explanations and observable behaviors. Addressing the common dissociation between stated principles and actual completions in models, the work leverages RL-based optimization of a paired meta-level (explanation-eliciting) and object-level (behavior-eliciting) process. This paradigm serves the dual goal of making LMs more predictable and auditable to users and model developers, with direct implications for alignment, interpretability, and controllability.

Methodology: Self-Consistency as RL Objective

The core Self-CTRL framework treats faithful self-explanation as an RL-derived objective, casting the model as a two-role agent: one generating explanations in response to meta-level queries, and one generating behaviors for concrete task prompts. Self-consistency is operationalized via a cross-context reward function, $\phi(y_e, y_b; x_e, x_b)$, which scores how well a sampled explanation $y_e$ (from input $x_e$) predicts the behavior $y_b$ (on input $x_b$).

In this schema, explanation training (updating $y_e$) pushes explanations toward better prediction of fixed behaviors, while behavior training (updating $y_b$) enforces that observed model decisions satisfy the sampled explanations. The framework supports arbitrary convex combinations of these, enabling gradual tradeoffs (with a parameter $\lambda$) between simulatability and behavioral alignment.

Learning is conducted via a group policy gradient (GPO) variant, with candidate explanations and behaviors pooled and rewards computed via a $\phi$ function instantiated as either a formal likelihood (for programmatic explanations) or LLM juries (for free-form NL explanations).

Formal Explanation Experiments: Latent Structure Articulation

The first empirical setting is a probabilistic reasoning task, where the LM is tasked to simulate flips from one of many coins with latent, variable biases, and then, via explanations, articulate the probability parameter for each coin. For held-out coins with only rollout (behavioral) supervision—no direct explanation labels—Self-CTRL explanation training substantially increases the $R^2$ correlation between stated explanations and empirical sampling bias (from $y_e$0 to $y_e$1) and likewise against ground truth ($y_e$2 to $y_e$3). Notably, generalization to held-out coins matches models with direct explanation supervision, indicating strong out-of-context inference of latent variables through self-consistency.

(Figure 1)

Figure 1: Self-CTRL narrows the gap to oracle supervision on experimental and held-out coins, aligning articulated bias with empirical and ground-truth bias.

This formal instantiation demonstrates that mere behavioral supervision permits the derivation of principled model self-knowledge, provided consistency between explanations and behavior is enforced.

Constitutional AI: Natural Language Rule Alignment

In a higher-level, practical alignment domain, Self-CTRL is tasked with harmonizing stated constitutional rules (NL explanations) with model responses on behaviorally challenging safety-constrained prompts (SpecEval). Here, $y_e$4 is computed with an LM jury, whose composition reflects diverse ethical frameworks, scoring how well a response coheres with the provided rule.

Self-CTRL is evaluated in three modes: explanation training ($y_e$5), behavior training ($y_e$6), and mixed ($y_e$7). Baselines include SFT-only, and one-sided RL training using standalone behavioral or explanation rewards.

Empirically, Self-CTRL:

• Improves the average jury consistency reward ($y_e$8) on both validation and held-out splits, indicating out-of-category and out-of-principle generalization.
• Traces an explicit Pareto optimal trade-off between safety (HarmBench attack success rate) and simulatability (Normalized Simulatability Gain, NSG). Explanation training yields maximal simulatability increases (NSG from $y_e$9 to $x_e$0); behavior training yields maximal safety reduction (attack success $x_e$1 to $x_e$2).

(Figure 2)

Figure 2: Self-CTRL enhances both simulatability and safety, with mixed training achieving a balance and pushing the observed Pareto frontier.

• In counterfactual simulatability, explanation and mixed training yield sharp improvements in refusal accuracy (from $x_e$3 to $x_e$4 and $x_e$5, respectively), with compliance accuracy unchanged. Behavior training yields smaller gains.
• Capabilities (MMLU), and over-refusal rates are only minimally affected. Mixed and explanation training preserve non-toxic compliance, while behavior training induces modest over-refusal.

Additional Model Analyses

Qwen3-8B Case Study

Self-CTRL’s effectiveness is bounded by the underlying behavioral diversity present in the base model. For Qwen3-8B, which is highly permissive, consistency training can only marginally improve simulatability because there are few refusal examples to constrain the rule boundary. This case highlights the necessity of near-boundary data for robust rule learning.

(Figure 3)

Figure 3: For Qwen3-8B, consistency training improves counterfactual consistency over baseline but simulatability on the refusal side remains poor due to low refusal rates in the base model.

Regularization and Failure Modes

The framework is susceptible to model collapse toward vacuous, trivially consistent explanations (e.g., “be safe” for everything). This is mitigated with structured prompts, auxiliary engagement rewards to discourage generic refusals, and regularization (KL and SFT-anchoring). Nevertheless, explanation faithfulness remains sensitive to both prompt engineering and jury instantiation.

Theoretical and Practical Implications

Interpretability and Alignment

Self-CTRL advances model interpretability and self-audit by grounding explanations in empirical behavior rather than independent SFT or prompt-crafting. For deployment, this approach circumvents the annotation bottleneck: models generate and grade their own explanation-behavior pairs, enabling scalable audit pipelines and supporting user trust via explicit, testable policy communication.

Safety/Alignment Tradeoffs

A salient contribution is the quantification and management of safety-simulatability tradeoffs. Self-CTRL’s $x_e$6 continuum allows practitioners to calibrate models for precision in self-explanation or strict behavioral constraint, according to deployment requirements. Importantly, models can be made safer (lower HarmBench attack rates) without commensurate increases in over-refusal and corresponding utility degradation, a core bottleneck in real-world alignment.

(Figure 4)

Figure 4: Behavior training makes output behavior more predictable—even absent explanations—raising predictor accuracy but potentially depressing explanation-induced simulatability gains.

Generalization and Data Requirements

Self-CTRL achieves out-of-context generalization of behavioral explanations, but only over the support of observable behaviors in training. Identifying behavioral rules for rare, adversarial, or weakly represented patterns may require explicit data augmentation to elicit sufficient coverage for the RL objective.

Conclusion

Self-CTRL establishes self-explanation/behavior concordance as a tractable RL objective, with robust empirical benefit in both formal probabilistic inference and high-level alignment domains. It regularizes LMs to produce explanations predictive of future behavior, refines their safety-policy boundaries, and allows continuous tuning between capabilites and interpretability. Data selection, prompt scaffolding, and jury design remain key determinants of success, and further exploration in broader domains and with larger base models is warranted. Self-CTRL stands as a complementary paradigm to external interpretability, with implications for deployment auditing, red-teaming, and trust calibration in next-generation LLMs.