---
title: Information Self-Locking (SeL) in RL Agents
url: https://www.emergentmind.com/topics/information-self-locking-sel
type: topic
---

# Information Self-Locking (SeL) in RL Agents

Information Self-Locking (SeL) is a failure mode in reinforcement learning (RL) for active reasoning with large language model (LLM) agents in which the agent ceases to ask informative questions and fails to internalize available information, resulting in stagnation of both exploration and learning. This phenomenon arises particularly when training with outcome-only rewards, as the bidirectional coupling of Action-Selection (AS) and Belief-Tracking (BT) capabilities leads to a low-information deadlock from which standard RL algorithms cannot readily recover [2603.12109].

## 1. Definition and Manifestation

Let $I_\mathrm{th}(w)$ denote the theoretical AS informativeness of a policy $\pi_w$, defined as the total Bayesian belief-gain an oracle update would obtain from the agent’s queries, and let $\mathrm{CBT}(w)$ quantify the absorbed progress in belief-tracking, i.e., the total positive drift in the agent’s own belief over on-policy rollouts. Information self-locking is formally characterized as the self-locking regime

$$
R_{\epsilon,\delta} = \{ w : I_\mathrm{th}(w) < \epsilon \ \text{and} \ \mathrm{CBT}(w) < \delta \}
$$

where both AS and BT are simultaneously deficient. Once the model enters $R_{\epsilon,\delta}$, outcome-only policy-gradient updates become inefficient at restoring either capability.

In practical settings such as preference estimation (PE-G), medical diagnosis (MediQ), and troubleshooting (FloDial), empirical traces demonstrate that, despite increasing episode rewards, the agent’s AS proxy (informative questions) and BT proxy (belief updates) plateau near zero. This manifests as the agent ceasing to ask questions that force trade-offs or elicit novel facts, and failing to improve its belief about the correct answer, even with access to on-policy evidence.

## 2. Underlying Formal Framework

Active reasoning in LLM agents is modeled as a partially observable Markov decision process (POMDP) with components $(S, Q, O, T, R, \gamma)$:

- $s^* \in S$: fixed latent ground-truth per episode.
- $q_t \sim \pi_w(\cdot|b_t) \in Q$: query-action selected given belief state $b_t$.
- $o_t = O(s^*, q_t) \in O$: deterministic feedback for each query.
- $b_{t+1} = U_w(b_t, q_t, o_t) \in \Delta(S)$: belief-update kernel induced by the LLM's internal state.
- $R(T)$: scalar final outcome reward, typically correctness post dialogue.

The trajectory probability under agent parameters $w$ factorizes as
$$
p_w(T) = \prod_{t=0}^{H-1} [\pi_w(q_t|b_t) \cdot U_w(b_{t+1}|b_t, q_t, o_t)]
$$
and the policy-gradient is given by
$$
\nabla_w J(w) = \mathbb{E}_{T \sim \pi_w}[R(T) \cdot \sum_{t=0}^{H-1} (\nabla_w \log \pi_w(q_t|b_t) + \nabla_w \log U_w(b_{t+1}|...))]
$$
This exposes the axis-aligned contributions of AS and BT, yielding channel-isolated advantages $A_t^{AS}$ and $A_t^{BT}$.

## 3. Mechanism and Theoretical Analysis

Information self-locking arises due to coupled failure modes:

- In the low-BT regime, informative queries provide negligible reward gains because the belief-update kernel $U_w$ fails to internalize new information, causing AS-channel advantages to collapse and query exploration to cease. 
- If AS halts informative querying, BT stagnates due to lack of novel evidence, resulting in interaction-insensitive shortcuts.

Theoretical analysis shows that in the self-locking region $R_{\epsilon,\delta}$, stepwise improvements in both informativeness and belief-tracking scale linearly with the current low values:
$$
\Delta^+ I_\mathrm{th}(w) \leq \eta C \, \mathrm{CBT}(w)
$$
$$
\Delta^+ \mathrm{CBT}(w) \leq \eta (B I_\mathrm{th}(w) + C\, \mathrm{CBT}(w))
$$
for step-size $\eta$, constants $B, C$, and where $\Delta^+$ represents the projected policy-gradient increment. Exiting the low-information region thus requires an exponentially large number of RL steps.

## 4. Solution: AREW (Advantage Reweighting with Directional Critiques)

To overcome self-locking, a lightweight auxiliary training objective is introduced: Advantage Reweighting with Directional Critiques (AREW). The method augments the RL signal with stepwise directional critiques easily obtained by inspecting user feedback or model readout.

At each step $t$, two binary critiques are extracted:
- $z_t^{AS} \in \{+1,-1\}$: was the query informative?
- $z_t^{BT} \in \{+1,-1\}$: did the belief move toward the ground truth?

When one or more of each type of critique appears within a trajectory, the intra-trajectory log-likelihood margin is computed:

$$
C(w; T) = \frac{1}{|P|}\sum_{t\in P} \log p_w(\cdot_t) - \frac{1}{|N|}\sum_{t\in N} \log p_w(\cdot_t)
$$

with $P = \{t : z_t=+1\}$ and $N = \{t : z_t = -1\}$, and corresponding gradient
$$
\nabla_w C(w; T) = \sum_{t=0}^{H-1} u_t \cdot \nabla_w \log p_w(\cdot_t)
$$
where $u_t$ is $+1/|P|$ if $z_t=+1$, $-1/|N|$ if $z_t=-1$, and 0 otherwise.

In RL training, the effective advantage per step is updated as
$$
A_t \leftarrow A_t + u_t
$$
Thereby, positively-critiqued steps receive increased learning signal and negatively-critiqued steps are down-weighted. Theoretical results demonstrate that as long as the accuracy $\mathrm{Acc}$ of the critiques exceeds 50%, the expected improvement in $I_\mathrm{th}(w)$ becomes proportional to $(2 \cdot \mathrm{Acc} - 1)$, ensuring eventual escape from the self-locking regime.

## 5. Empirical Evaluation

Experiments span 7 tasks in three domains: Preference Estimation (PE-G, PE-F), Medical Diagnosis (MediQ), and Troubleshooting (FloDial-Easy/Hard). Baselines included zero-shot LLMs (Direct Inference), vanilla PPO, group-based RL variants GRPO and GSPO, using Qwen-2.5-7B-Instruct and LLaMA-3.1-8B-Instruct.

Table: AREW Results Across 28 Settings

| Setting             | Vanilla PPO Score | AREW Score | Relative Gain (%) |
|---------------------|------------------|------------|------------------|
| PE-GS=3 (LLaMA)     | 11               | 73         | +62              |
| Other 27 Settings   | N/A              | Up to +60  | Median +X        |

Across 28 settings, AREW outperformed vanilla PPO in 27, with relative improvements up to +62%. Combined AS+BT critiques surpassed AS-only critiques in the majority of evaluated cases. AREW’s improvements in episode reward, AS proxy, and BT proxy persisted across RL algorithms and were robust to up to 40–50% random flips in critique labels.

## 6. Implications and Design Recommendations

The results underscore that credit-assignment in long-horizon RL for LLMs is intrinsically two-dimensional: both exploration (AS) and evidence-internalization (BT) are required for robust learning. Outcome-only rewards entangle these axes, predisposing models to SeL and associated stagnation.

Directional critiques provide a fine-grained, non-degenerate gradient signal that mitigates the pathological bidirectional coupling without recourse to heavyweight reward shaping or auxiliary discriminators. A practical implementation strategy consists of:

1. Instrumenting active-reasoning prompts to output a binary “informative?” flag per query.
2. Reading out a confidence score at each belief update.
3. Reweighting policy-gradient advantages per step as $A_t \leftarrow A_t + u_t$.

It is advisable to maintain tracking proxies for AS and BT, proactively intervene upon stagnation of either channel, and adopt minimal-injection strategies like AREW to preserve exploration–exploitation balance.

## 7. Future Perspectives

A plausible implication is that future RL-based agent architectures for complex reasoning should explicitly model and disentangle the AS and BT processes, ensuring that learning signals preserve both capacities, thereby avoiding convergence to degenerate low-information regimes. Continued exploration of lightweight, critique-driven credit-assignment mechanisms appears warranted for scalable and data-efficient training in active reasoning domains [2603.12109].

Source: https://www.emergentmind.com/topics/information-self-locking-sel