Reinforcement Learning with Internal Feedback

• Reinforcement learning with internal feedback is a method where internal computations generate control signals and reward estimations instead of relying solely on external rewards.
• It encompasses various methodologies including hidden state modulation, learned evaluators, feedback graphs, and latent human signals to refine decision-making policies.
• These approaches enhance policy adaptation and exploration while presenting challenges such as reward noise amplification and stability in optimization.

Reinforcement learning with internal feedback denotes a family of methods in which the signal driving sequential adaptation is produced, transformed, or mediated by internal computational structure rather than supplied only as an externally specified scalar reward. In the literature, this includes at least four distinct regimes: a policy that controls a model’s own hidden computation during inference, reward signals generated by learned internal evaluators, internally constructed side experiences induced by structural feedback graphs, and feedback channels filtered through hidden human or latent reward-states rather than directly observed rewards (Stollenga et al., 2014, Li et al., 2023, Kausik et al., 2024, Liu et al., 2024). The term is therefore not uniform: in some papers “internal” refers to the inside of the policy network, in others to the reward-generation mechanism, and in others to hidden cognitive or latent feedback processes.

1. Scope, terminology, and major formulations

Across the cited literature, “internal feedback” is polysemous. Some works study internal control, where the agent acts on its own hidden representations; others study internal reward generation, where the reward used for policy learning is computed by a learned model rather than emitted directly by the environment; still others study internally generated side observations, where one real interaction yields additional synthetic learning signals; and another line studies internal human feedback, where the evaluative signal is latent in biological or cognitive state and must be decoded (Stollenga et al., 2014, Li et al., 2023, Dann et al., 2020, Poole et al., 2021).

Regime	Internal object	Representative formulation
Internal control	Hidden activations or feature-map gains	Gated multi-pass CNN inference
Internal reward model	Learned posterior, discriminator, or predictive model	Joint policy–reward optimization or prediction-error reward
Internal side experience	Graph-structured auxiliary observations	Feedback graphs and side replay
Internal human signal	EEG or latent cognitive state	ErrP-based reward shaping
Internalized critique	Learned self-critique or feedback predictor	Distillation from critique-conditioned revisions

This variety matters because the same phrase can denote very different learning problems. In some cases the “environment” is effectively the model’s own future computation; in others the environment remains external but the reward channel is mediated by a learned internal estimator; and in still others the latent variable sits inside the human evaluator rather than inside the agent (Stollenga et al., 2014, Li et al., 2023, Kausik et al., 2024).

2. Internal control of hidden computation

An early and explicit instance of reinforcement learning with internal feedback is dasNet, which augments a Maxout CNN with multiplicative gates on convolutional output maps and treats those gates as continuous actions selected over multiple passes on the same image. If a convolutional layer normally computes

$$y_j^\ell = \sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$

dasNet replaces it with

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$

where $a_j^\ell$ is a gate chosen by a controller. The action vector concatenates these gains across gated layers, the first pass starts from $\mathbf a_0=\mathbf 1$, and later passes reuse the same image while modulating hidden processing online. The controller observes average activations of Maxout maps, intermediate classification-layer activations, and the current class-probability vector, then applies a deterministic linear map followed by a scaled softmax,

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$

so that gate values remain positive and average to $1$. The paper formulates this as an RL problem but solves it by black-box policy search with SNES because the controller has close to a million parameters; in the reported experiments the rollout horizon is $T=5$ and the SNES population size is $50$ (Stollenga et al., 2014).

Conceptually, this is internal feedback because neither observations nor actions primarily concern an external world state. The observation is a summary of the network’s own activations and uncertainty, and the action changes the network’s own next internal computation without rewriting base weights. The paper emphasizes that difficult examples can benefit from multiple feedforward sweeps with different internal emphases, and reports CIFAR-10 test error improving from $9.61\%$ for their Maxout implementation to $9.22\%$ for dasNet, and CIFAR-100 improving from $$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$0 to $$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$1; it also reports that gate settings encode task-relevant information and that the learned recurrent attention dynamics degrade gracefully when run beyond the training horizon (Stollenga et al., 2014).

3. Internally generated reward models

A second major formulation treats the reward itself as an internal object. In Internally Rewarded Reinforcement Learning, the policy is trained not on an externally supplied reward function, but on a reward generated by an internal model that is jointly optimized with the policy. The paper formalizes a latent variable $$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$2, a trajectory $$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$3, a discriminator $$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$4, and reward functions derived from this posterior. The variational mutual-information reward is

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$5

but the paper argues that joint policy–reward-model optimization is unstable because an immature discriminator produces noisy rewards and a weak policy generates uninformative trajectories. It therefore studies transformed rewards

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$6

and proposes the clipped linear reward

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$7

showing theoretically and empirically that lower-curvature transformations reduce reward-noise amplification relative to the logarithmic reward. The paper places hard attention, unsupervised skill discovery, active vision, and RLHF-like learned-reward pipelines inside this broader IRRL template (Li et al., 2023).

A related but distinct line learns an internal predictive model from expert observation trajectories alone and uses prediction discrepancy as reward. There the internal model is trained by

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$8

and the RL reward becomes

$$y_j^\ell = a_j^\ell\sum_{i=0}^{i=c}\phi (x_i \ast F_{i,j}^\ell),$$9

The agent is thus rewarded when its next state matches what the internal model predicts expert-like progression should produce. The paper uses this mechanism in low-dimensional control and game domains, including observation-only demonstrations for Flappy Bird and Super Mario Bros., arguing that reward can be estimated without expert action labels or a hand-crafted external task reward (Kimura et al., 2018).

These two lines share a structural property: the reward is not primitive. It is computed from a learned internal object—a discriminator posterior in IRRL, or a predictive model of expert state evolution in observation-based reward shaping—and that induces a coupled optimization problem between policy learning and the quality of the internal feedback channel (Li et al., 2023, Kimura et al., 2018).

4. Language-model post-training: certainty rewards and internalized critique

In language-model post-training, reinforcement learning with internal feedback has been formalized as KL-regularized policy optimization driven by self-derived certainty signals rather than human labels or externally trained reward models. One mathematical formulation defines a self-certainty reward over a response $a_j^\ell$0 as

$a_j^\ell$1

and optimizes

$a_j^\ell$2

The paper shows that

$a_j^\ell$3

so the internal reward is effectively anti-entropy. Under $a_j^\ell$4, the optimum is a power-sharpened reference policy, and under the idealized assumption $a_j^\ell$5 with $a_j^\ell$6, the paper derives an exact objective-level equivalence between RLIF and RLHF; under $a_j^\ell$7, the same sharpened distribution also coincides with soft best-of-$a_j^\ell$8 test-time scaling (Jiao et al., 4 Sep 2025).

A later empirical study evaluates self-certainty, token-level entropy, and trajectory-level entropy as RLIF rewards for math reasoning and finds that all three tend to reduce policy entropy. Token-level entropy is exactly negative expected policy entropy, while the other two are shown under stylized assumptions to induce entropy reduction through covariance arguments. Empirically, these objectives can improve base models early in training, sometimes matching or surpassing RLVR-style gains initially, but longer training degrades performance, sometimes below the pre-training baseline, and instruction-tuned models benefit little. The reported mechanism is overconfidence: entropy collapse reduces “transitional words,” shortens responses, suppresses exploratory reasoning, and can turn initial format compliance gains into shallow reasoning failures (Zhang et al., 20 Jun 2025).

A separate line asks how external textual critiques available during training can be internalized so that inference remains single-turn. RL from Text Feedback formalizes a multi-turn process with training-time critiques $a_j^\ell$9 and reward $\mathbf a_0=\mathbf 1$0, but optimizes single-turn performance

$\mathbf a_0=\mathbf 1$1

It proposes RLTF-SD, which distills feedback-conditioned second-turn generations into the first-turn policy, and RLTF-FM, which predicts feedback text with an auxiliary loss

$\mathbf a_0=\mathbf 1$2

Both methods are designed to turn external critiques into an internalized feedback mechanism that improves one-shot behavior, and the paper reports consistent gains over strong baselines across reasoning, competition math, and creative writing tasks (Song et al., 2 Feb 2026).

An adjacent model-mediated formulation replaces humans with a noisy language-model evaluator. There a state-score model $\mathbf a_0=\mathbf 1$3 is trained from pairwise rankings, and reward is used in potential-difference form,

$\mathbf a_0=\mathbf 1$4

rather than as a raw score. The key theoretical result is that repeated-query inconsistency pushes score differences toward zero, so uncertain internal feedback becomes uninformative rather than strongly misleading (Lin et al., 2024).

5. Feedback graphs and internally constructed side experience

Another major use of “internal feedback” does not alter the reward function at all. Instead, one interaction yields multiple internally constructed observations. In feedback-graph RL, the learner interacts with an episodic MDP but, after executing one state-action pair $\mathbf a_0=\mathbf 1$5, receives additional observations for every out-neighbor of $\mathbf a_0=\mathbf 1$6 in a directed graph $\mathbf a_0=\mathbf 1$7 over state-action pairs. At each step,

$\mathbf a_0=\mathbf 1$8

and the extra samples are distributed like correct transition/reward samples for those other pairs. The optimism-based analysis then depends not on $\mathbf a_0=\mathbf 1$9, but on graph complexity. In particular, the main regret bound scales as

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$0

where $$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$1 is the size of the maximum acyclic subgraph (mas-number), with a lower bound of order $$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$2 in terms of the independence number $$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$3. The same paper shows that a small dominating set is much subtler in RL than in graph-feedback bandits because informative vertices may themselves be hard to reach (Dann et al., 2020).

Lost-sales inventory control instantiates the same idea in a domain-specific way. RLFG+IME uses a feedback graph to turn one real interaction into many side experiences and then adds an intrinsic reward that prefers state-action pairs producing many uncertain side experiences. The feedback graph is dynamic: uncensored demand observations induce effectively complete graph coverage over state-action pairs, while censored observations induce only partial connectivity. The internal reward for a sampled experience is defined as

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$4

where the $$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$5 terms are ensemble-disagreement uncertainties from an $$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$6-head DQN, and training uses the shaped reward

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$7

Here “internal feedback” therefore refers to internally generated side experiences and intrinsic exploration bonuses rather than hidden-state control or self-certainty (Liu et al., 2024).

The common feature of these feedback-graph methods is that one external interaction is algorithmically expanded into a richer internal supervision set. The feedback is internal in the sense of being constructed inside the learning system from structure, not because it is a hidden neural activation or a self-generated scalar confidence score (Dann et al., 2020, Liu et al., 2024).

6. Human internal states and partially observed reward channels

A distinct strand defines internal feedback as feedback implied by a human’s internal biological or cognitive state. A review of interactive RL and BCI places such signals inside a taxonomy of demonstrations, advice, guidance, and feedback, and defines intrinsic feedback as feedback implied through a human’s intrinsically occurring biological signals such as brain activity. The paper stresses that this is not intrinsic motivation in the standard RL sense. It surveys integration mechanisms including reward shaping,

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$8

value shaping,

$$\pi_{\theta_i}(\mathbf{o}) = dim(A)\,\sigma(\boldsymbol{\theta}_i \mathbf{o_t}) = \mathbf a_t,$$9

and policy shaping, where the decoded human signal acts as a proxy for evaluative feedback or action preference (Poole et al., 2021).

An experimental realization uses EEG-based observation ErrPs as implicit human feedback. The environment is an MDP

$1$0

the human observes agent actions, and an EEG pipeline with xDAWN spatial filtering, tangent-space Riemannian features, and ElasticNet classification predicts whether the observed action elicited an ErrP. Rather than shaping reward directly from raw decoded labels alone, the paper learns a soft-Q human model

$1$1

adds a state baseline $1$2, and defines an auxiliary reward

$1$3

RL then uses

$1$4

The paper reports zero-shot ErrP transfer across games, improved label efficiency, and substantial query reduction relative to querying implicit feedback at every training step (Xu et al., 2020).

At a more abstract level, PORRL treats the reward channel itself as partially observed through latent reward-states. A cardinal PORMDP is

$1$5

with hidden reward-states $1$6, reward functions $1$7, and observed feedback only at timesteps $1$8. The learner does not observe $1$9 or $T=5$0, only noisy feedback $T=5$1. This framework subsumes traditional RL, once-per-episode RLHF, and reward machines, and it supports both cardinal and dueling feedback. The paper develops POR-UCRL and POR-UCBVI for cardinal feedback, proves regret and sample-complexity guarantees, shows that Markovian policies can be provably insufficient, and argues that recursive latent reward-state structure can make history-aware Bellman-eluder analysis exponentially sharper than naive history summarization (Kausik et al., 2024).

7. Conceptual tensions, misconceptions, and research directions

The most persistent misconception is that reinforcement learning with internal feedback is a single method class. The literature instead contains several non-equivalent constructions: hidden-computation control, internally generated reward models, graph-induced side observations, biologically internal human feedback, and latent reward-state models (Stollenga et al., 2014, Li et al., 2023, Dann et al., 2020, Poole et al., 2021). A second misconception is that internal feedback is automatically aligned with task success. Certainty-based RLIF in LLMs explicitly optimizes anti-entropy objectives, not correctness; the theoretical sharpening results and the reported late-stage degradation both show that internal confidence can diverge from externally verified performance (Jiao et al., 4 Sep 2025, Zhang et al., 20 Jun 2025).

The main technical limitations are mechanism-specific. DasNet acts only through feature-map gain modulation, uses a simple linear controller, and relies on SNES rather than differentiable credit assignment (Stollenga et al., 2014). IRRL exposes a bidirectional instability in which noisy internal rewards and weak policies reinforce each other, so the reward transform itself becomes part of the stability problem (Li et al., 2023). Feedback-graph methods require side observations that are interpretable as correct or approximately correct transition samples, and dominating-set structure is only useful when informative vertices are reachable (Dann et al., 2020). EEG-based feedback remains noisy, timing-sensitive, and attention-dependent, and the survey literature emphasizes decoder non-stationarity, cross-subject variability, and the distinction between biological intrinsic feedback and algorithmic intrinsic motivation (Xu et al., 2020, Poole et al., 2021). PORRL, finally, shows that naive history summarization can be statistically wasteful when latent reward-state structure is recursive or partially observed (Kausik et al., 2024).

Several research directions recur across these papers. One is richer internal action spaces: dasNet already identified spatially focused actions as future work beyond global feature-map gains (Stollenga et al., 2014). Another is replacing brittle or noisy internal objectives with hybrid schemes: LLM studies suggest early entropy-based sharpening may be useful, but prolonged optimization requires externally faithful signals or more structured critique internalization (Zhang et al., 20 Jun 2025, Song et al., 2 Feb 2026). A third is exploiting internal structure without assuming exact tabular side feedback, extending feedback-graph or PORRL ideas to richer function approximation and more realistic latent-state models (Dann et al., 2020, Kausik et al., 2024).

Taken together, these works suggest a unifying interpretation: reinforcement learning with internal feedback is best understood not by a single reward formula, but by the locus at which evaluative information is computed. That locus may be a network’s own hidden state, a jointly trained reward estimator, a structural side-observation mechanism, a learned critique model, or a hidden human/internal reward-state. The common agenda is to make learning depend on more than externally observed scalar rewards, while preserving enough statistical and algorithmic discipline that the extra internal signal improves, rather than destabilizes, policy optimization.