SafeCtrl-RL: Inference-Time RL for Dialogue Safety

• SafeCtrl-RL is an inference-time behavioural control framework that refines LLM prompts to suppress unsafe dialogue without modifying model parameters.
• It formulates dialogue generation as a sequential decision process where an RL agent selects from 11 prompt-adjustment strategies based on multi-dimensional contextual feedback.
• Empirical results demonstrate significant safety improvements and efficiency across various LLMs, with partially sustained behavioural retention after safeguard removal.

SafeCtrl-RL is an inference-time behavioural control framework for LLM dialogue that enables adaptive safety regulation without model retraining or parameter modification. It formulates dialogue generation as a sequential decision process in which a reinforcement learning agent dynamically selects prompt adjustment strategies from contextual feedback, so that unsafe behaviours are suppressed through iterative refinement; the authors conceptualise this as inference-time behavioural unlearning. Rather than treating safety as a static prompt filter or a parameter-space alignment problem, SafeCtrl-RL treats prompt construction itself as the control surface and implements a closed-loop generate–evaluate–refine process over dialogue turns (Orme et al., 25 May 2026).

1. Motivation and control-theoretic framing

The method is motivated by a limitation of many safety interventions for conversational LLMs. Parameter-based methods such as fine-tuning, model editing, or unlearning require access to model parameters, are usually offline, are expensive to update, and are not adaptive during conversation. SafeCtrl-RL instead assumes that dialogue safety is dynamic: the risk depends on the evolving context, previous turns, and how the model responds. On that basis, the paper frames dialogue safety as a closed-loop inference-time control problem: generate a response, evaluate it for safety and quality, update the prompt based on feedback, regenerate, and repeat until the response is acceptable (Orme et al., 25 May 2026).

This framing places the system prompt at the center of control. SafeCtrl-RL does not alter the underlying LLM weights and does not require white-box access. Its intervention is entirely prompt-level and inference-time. The paper therefore distinguishes it from traditional machine unlearning: the model’s unsafe behaviour is functionally suppressed through interaction and control, rather than deleted from parameters. In the authors’ terminology, this is inference-time behavioural unlearning (Orme et al., 25 May 2026).

2. Sequential decision process and prompt-control mechanism

Let $M$ be a fixed LLM and $S_t^{(k)}$ the system prompt at dialogue turn $t$ and refinement iteration $k$. Given user input $U_t$, the model generates

$R_t^{(k)} = M(U_t, S_t^{(k)}).$

That response is scored by a safety-quality evaluator. If the score is too low, the prompt is revised and generation repeats, yielding a closed-loop generate–evaluate–refine process (Orme et al., 25 May 2026).

The RL state is a 36-dimensional state vector built from a buffer of prior iterations. The paper groups its features into nine categories: Training/progress features, Performance features, Strategy performance features, Current episode features, Category/context features, Risk/safety features, Exploration features, Prompt sophistication features, and Hash-based user prompt features. The point of this representation is that the agent does not merely observe the latest score; it also observes optimization history, safety trends, prompt characteristics, and exploration status, allowing context-appropriate refinement decisions (Orme et al., 25 May 2026).

The action space is discrete: $\mathcal{A} = \{a_1, a_2, \dots, a_{11}\}.$ The 11 prompt-adjustment strategies are Minimal, Raw History, AI Summary Only, AI Enhanced, Progressive Summary, Hybrid, Best–Worst–Recent, Performance Tiered, Trajectory Focused, Contrast Learning, and Adaptive Performance. These strategies differ in how they package historical interaction data into the next system prompt: some use no history, some use raw history, some use summaries, and some emphasise performance trajectories or contrasts. The learned policy is written as

$\pi(a_t \mid S_{t(k)}),$

and after learning the deterministic action is selected as the argmax over the policy distribution (Orme et al., 25 May 2026).

3. Evaluator, reward structure, and DQN optimisation

The evaluator is implemented with DeepEval and Gemini 2.0 Flash as the evaluator model. It produces a quality score $q \in [0,1]$, based on coherence, relevance, and context relevance, and a safety score $s \in [0,1]$, based on absence of harmful content. Safety is assessed with a fine-grained harm taxonomy derived from PKU-SafeRLHF / Llama Guard-style categories: violent crimes, non-violent crimes, sex-related crimes, child sexual exploitation, specialized advice, privacy, intellectual property, indiscriminate weapons, hate, suicide/self-harm, sexual content, and rudeness (Orme et al., 25 May 2026).

The reward couples quality and safety multiplicatively: $S_t^{(k)}$0 with $S_t^{(k)}$1 and $S_t^{(k)}$2. In the paper’s interpretation, $S_t^{(k)}$3 controls the trade-off between quality and safety, while $S_t^{(k)}$4 sharpens the reward surface so that low values in either dimension are heavily penalized. Because of the multiplicative form, a response cannot obtain a high reward by being strong on one axis and weak on the other. The paper also imposes a hard safety threshold: if the minimum over the critical safety metrics falls below a threshold $S_t^{(k)}$5, for example $S_t^{(k)}$6, the final reward is $S_t^{(k)}$7, regardless of quality (Orme et al., 25 May 2026).

The RL agent is a DQN agent with policy type MlpPolicy, total timesteps 5,000,000, learning rate $S_t^{(k)}$8, batch size 256, replay buffer 100,000, Q-network architecture $S_t^{(k)}$9, exploration fraction 0.5, and final $t$0 0.1. The learned policy therefore optimises long-term prompt-adjustment behaviour, but the underlying LLM parameters are never updated. This separation is central to the method’s claim that it performs inference-time optimisation only (Orme et al., 25 May 2026).

4. Experimental design, baselines, and metrics

SafeCtrl-RL is evaluated on four LLMs with different alignment and architecture profiles: BlackSheep-Llama3.2-3B, DialoGPT-Large, DeepSeek-R1-Distill-Qwen-1.5B, and Evil-Alpaca-3B. The unsafe prompt corpus is built by merging PKU-SafeRLHF, TOXIC-DPO, BeaverTails, and DarkSide DPO. The resulting corpus contains 1,048,575 prompts, including 629,323 safe prompts and 415,199 unsafe prompts, organised into 12 harm categories. For evaluation, the paper samples a fixed benchmark of 50 prompts per category, yielding 600 evaluation prompts in total, and keeps that set fixed across methods and models (Orme et al., 25 May 2026).

The comparison set includes handcrafted prompt-adjustment strategies and prompt-optimisation methods. The handcrafted strategies include raw history, AI summary only, AI enhanced, best-worst-recent, contrast learning, hybrid, minimal, performance tiered, progressive, smart adaptive, and trajectory learning; the appendix also includes static prompting variants such as few-shot, roleplay, chain-of-thought, and value reinforcement. The prompt-optimisation baselines are Evolutionary, TextGradient, GRIPS, Dynamic Retrieval, and OPRO. Among static methods, the paper highlights Self-Correction because it performed best and matches the iterative generate–revise structure (Orme et al., 25 May 2026).

The paper reports four principal metrics. The Safety-Quality Score $t$1 is the final reward. Behavioral Improvement $t$2 is the safeguarded score minus the plain score. The Performance–Efficiency Ratio is

$t$3

where $t$4 is the average number of refinement iterations. Retention / Consistency after removing the safeguard is measured using $t$5 and Retention, defined as the percentage of samples with reward $t$6. The evaluation protocol generates a plain response, applies the safeguard loop, stops when the threshold is met or the iteration cap is reached, and scores the final response. For the retention study, the system records the best safeguarded step $t$7, removes the safeguard, re-queries the model with the history $t$8 plus a minimal consistency instruction, and measures how much of the improved behaviour remains (Orme et al., 25 May 2026).

5. Empirical results and observed retention

The main result is that SafeCtrl-RL achieves the best overall macro performance among the compared strategies, with Macro $t$9. The paper reports model-specific examples of BlackSheep $k$0 with improvement $k$1, DialoGPT $k$2 with improvement $k$3, DeepSeek-R1 $k$4 with improvement $k$5, and Evil-Alpaca $k$6 with improvement $k$7. The emphasis is not only on peak performance but on consistency across architectures: fixed strategies can work well on one model and poorly on another, whereas SafeCtrl-RL adapts (Orme et al., 25 May 2026).

Against prompt-optimisation baselines, the gap is large. The strongest baseline, OPRO, has macro performance around $k$8, substantially below SafeCtrl-RL’s $k$9. The paper also notes that several optimisation methods fall below the plain baseline on some models, indicating that prompt optimisation can worsen unsafe dialogue performance. This is used to support the claim that dialogue safety control is not simply a matter of finding a better prompt once; it depends on selecting refinement strategies based on the evolving interaction (Orme et al., 25 May 2026).

The method also shows a favourable performance-efficiency trade-off. The paper reports approximately $U_t$0 with $U_t$1, placing SafeCtrl-RL in a stronger region of the performance-efficiency space than most fixed strategies, which often require more iterations while achieving smaller gains. The authors further report that latency is dominated by the evaluator: safeguarded responses take about 80–130 seconds depending on strategy, the external DeepEval/Gemini evaluation takes about 20.4 seconds per call, and RL inference itself adds only about 4–5 seconds (Orme et al., 25 May 2026).

The retention study indicates that some of the improved behaviour persists after the safeguard is removed, although the effect is model-dependent.

Model	Plain $U_t$2 Safeguarded $U_t$3 Post-safeguard	Retention
BlackSheep	0.193 $U_t$4 0.910 $U_t$5 0.676	90.5%
Evil-Alpaca	0.451 $U_t$6 0.990 $U_t$7 0.902	94.5%
DeepSeek-R1-Distill	0.290 $U_t$8 0.793 $U_t$9 0.671	72.1%
DialoGPT	0.287 $R_t^{(k)} = M(U_t, S_t^{(k)}).$0 0.573 $R_t^{(k)} = M(U_t, S_t^{(k)}).$1 0.524	60.12%

The paper interprets these figures as evidence that inference-time control can induce partially sustained behavioural change, while also stressing that the persistence is uneven across models (Orme et al., 25 May 2026).

6. Interpretation, adjacent control-oriented RL methods, and limitations

The paper’s internal analysis attributes SafeCtrl-RL’s behaviour to several interacting factors. State awareness matters because the agent observes not only the latest score but also optimisation history, risk, and prompt structure. No single strategy is universally best, so adaptive selection outperforms fixed rules. Hard safety gating prevents reward hacking because unsafe outputs cannot be compensated for by high quality. Iterative refinement is well matched to dialogue, where safety problems often emerge across turns rather than in a single shot. These claims position SafeCtrl-RL as a control policy over prompt updates rather than as a static prompting heuristic (Orme et al., 25 May 2026).

A common misconception is to read the system as a form of machine unlearning in the standard parameter-space sense. The paper explicitly distinguishes its approach from retraining or editing. It does not erase weights, does not retrain the LLM, and does not require white-box access. The “unlearning” claim is functional and behavioural: unsafe tendencies are suppressed through repeated prompt-level control. The reported post-safeguard retention does not imply permanent parameter modification; it indicates that, in the tested interaction regime, some improved behaviour remains after safeguard removal (Orme et al., 25 May 2026).

A plausible contextualization is that SafeCtrl-RL belongs to a broader control-oriented RL literature in which the intervention locus differs. RLBF applies critic feedback during live generation and trains a model to emit a backtrack by x tokens signal so that unsafe spans can be retracted and generation can continue autoregressively (Sel et al., 9 Feb 2026). Ctrl-R defines guided behavior policies for targeted reasoning structures and uses tractable importance-sampling correction to learn from guided rollouts (Kung et al., 2 Mar 2026). Self-CTRL optimises consistency between meta-level self-explanations and object-level refusal/compliance behaviour, improving both auditor predictability and alignment outcomes (Pres et al., 16 Jun 2026). This suggests that SafeCtrl-RL is distinctive not because it is the only control-oriented RL method for LLMs, but because it locates control at the inference-time system-prompt refinement loop rather than at token-level rollback, structured decoding, or explanation–behaviour coupling.

The paper is explicit about several limitations. Evaluation focuses mainly on under-aligned or unsafe model families, and only four LLM families are tested. The retention study uses a sampled subset rather than the full distribution. The system relies on a single external evaluator, so robustness to alternative judges remains untested. Latency is substantial and dominated by evaluator calls. The paper proposes several future extensions: combining SafeCtrl-RL with soft prompting or other lightweight adaptation mechanisms to reduce latency, and extending the framework to multi-turn dialogue, multimodal LLMs, domain-specific safety settings, lighter evaluators, and alternative feedback signals (Orme et al., 25 May 2026).