Neural Feedback as Reward Signal

• Neural Feedback as Reward Signal is a paradigm that integrates measurable brain activity into learning systems via explicit and implicit reward cues.
• It leverages both biophysical models and noninvasive techniques, such as EEG and fNIRS, to capture evaluative signals that inform agent behavior.
• Integration methods, including reward shaping and policy modulation, demonstrate improved learning speed, stability, and sample efficiency.

Neural feedback as a reward signal encompasses the interpretation and integration of neural activity—originating from human or animal brains—into the learning loops of artificial or biological agents through explicit or implicit reward-like signals. This concept bridges neuroscience, brain-computer interfaces (BCI), reinforcement learning (RL), and preference modeling to enable agents to learn from evaluative neural responses, either in real-time or from pre-recorded neural data. The field spans multiple levels, ranging from subcellular processes (dopaminergic modulation of plasticity) and spiking neural network models to macroscale noninvasive brain signals (EEG, fNIRS) driving policy updates in intelligent agents.

1. Biophysical Foundations: Reward Modulation in Biological Neural Circuits

Explicit neural feedback as a reward signal is rooted in the biology of reward-driven plasticity. In the pyramidal neuron combinatorial switch model, global modulatory signals such as dopamine encode reward prediction errors and directly gate synaptic plasticity rules (Rvachev, 2011). Synaptic clusters on dendrites act as nonlinear detectors of input conjunctions; back-propagating action potentials mark cluster activation, and a subsequent phasic dopamine reward signal triggers potentiation (if positive) or depression (if negative) of the cluster’s efficacy via plasticity equations such as:

$$\Delta\gamma_j = \eta \cdot R \cdot H(n_j - n_j^* - \bar n_j)$$

Here, $H(\cdot)$ is the Heaviside step function, $R$ the global reward, and $n_j, n_j^*, \bar n_j$ are counts of active synaptic types within the cluster. This cellular paradigm constitutes a biologically grounded instance of neural feedback as reinforcement.

At a broader scale, models such as BioLCNet implement reward-modulated spike-timing-dependent plasticity (R-STDP), where a global reward signal (emulated as a phasic scalar variable corresponding conceptually to dopamine) modulates eligibility traces and determines sign/integration of synaptic weight updates (Ghaemi et al., 2021). Both static ($r_i = \pm 1$ for correct/incorrect decisions) and temporally filtered reward-prediction-error variants have been explored, yielding substantial gains in learning stability and sample efficiency.

2. Noninvasive Neural Feedback: EEG, fNIRS, and Implicit Human-in-the-Loop RL

Noninvasive brain signals serve as accessible channels for neural feedback. Error-related potentials (ErrPs), typically detected with EEG, are stereotyped event-related potentials that reflect the human observer’s implicit detection of an agent’s error—a negative deflection (ERN/Ne) followed by a positive Pe—enabling low-latency, continuous feedback annotation (Poole et al., 2021, Kim et al., 17 Jul 2025). An EEG decoding pipeline usually involves:

• Signal acquisition (32–64 channels, 250–1000 Hz)
• Preprocessing (band-pass, artifact suppression, segmentation)
• Feature extraction (ERP amplitudes, spectral/temporal CNN features)
• Classification (EEGNet, SVM, LDA) yielding $p_t = P(\mathrm{ErrP}|X_t)$

The decoded probability is mapped to a scalar reward component, e.g., $r^\mathrm{EEG}_t = 1 - p_t$ or its scaled/centered variant. This signal is then either combined additively with environment rewards or used to bias action selection. Fully implicit human feedback via EEG ErrPs enables reinforcement learning from implicit human feedback (RLIHF), where agents in continuous-control tasks such as 7-DoF robotic manipulation receive reward shaping at every timestep from real-time decoded neural signals, demonstrably accelerating policy learning relative to sparse-reward or even manually designed dense-reward baselines (Kim et al., 24 Nov 2025, Kim et al., 17 Jul 2025).

Functional near-infrared spectroscopy (fNIRS) offers complementary high-level feedback by recording prefrontal hemodynamics correlated to the observer’s perceived agent performance. Machine learning models (SVM, MLP, RF) are trained to map sliding-windowed feature vectors from multi-channel fNIRS data to auxiliary performance or reward-like scores, with demonstrated cross-condition (watch/play) classification accuracies of 72–77% (Santaniello et al., 14 Jun 2025).

3. Neural Feedback Integration in Reinforcement Learning and Reward Shaping

Neural feedback signals are incorporated into RL frameworks through several algorithmic modalities:

• Reward shaping: $$r'(S_t,A_t) = r_{\text{env}}(S_t,A_t) + \beta \,\hat{f}(S_t,A_t)$$, where $\hat{f}$ is the decoded neural feedback signal and $\beta$ a gain factor (Poole et al., 2021, Kim et al., 24 Nov 2025).
• Value and policy shaping: neural feedback adjusts $H(\cdot)$0-values or biases the policy distribution, enabling more human-aligned exploration.
• Exploration guidance: in some frameworks, neural feedback is used to train a detached “human-feedback policy” via supervised learning, which then guides initial exploration before decaying to pure RL (Akinola et al., 2019).
• Inverse RL from neural data: neural signals are mapped to inferred intrinsic reward functions, allowing closed-form IRL solutions and decoding/prediction of animal behavior (Kalweit et al., 2022).

Explicit reward integration schemes often involve additive reward computation at each timestep, with the weight $H(\cdot)$1 (or $H(\cdot)$2) subject to tuning for optimal tradeoff between exploitation of neural trajectories and robustness to feedback noise (Kim et al., 24 Nov 2025). Control experiments and statistical analyses demonstrate significant learning speedup, robustness to decoder variability, and improved final performance in both simulated and real closed-loop tasks.

4. Neural Networks as Interpreters of Human Preference and Feedback

Neural-feedback-based reward modeling extends to high-level human feedback, including preference data and language critiques. Neural contextual bandit methods treat human pairwise preferences (e.g., “A preferred to B”) as noisy observations of latent reward differences, modeling the signal with deep ReLU networks and applying maximum likelihood estimation under the Bradley–Terry–Luce model. The resultant reward estimator $H(\cdot)$3 is used both for policy selection and as the subject of active data collection strategies, yielding sublinear regret and near-optimal sample efficiency in synthetic and real-world datasets (Verma et al., 16 Apr 2025).

In dialogue and LLM alignment, reward modeling from natural language human feedback (RM-NLHF) uses similarity between model-generated and human critiques, computed via core-argument F1 score, to construct a composite reward signal for RL (e.g., $H(\cdot)$4 for correct outcomes). The process reward is generalized to unannotated data using a meta reward model (MetaRM) with a transformer decoder, ensuring consistent reward alignment throughout policy optimization (Wang et al., 12 Jan 2026).

Feedback neural networks (FNNs) can also be used for reward shaping in deep RL from human-labeled episode data. Ensemble-based architectures provide calibrated confidence, with reward shaping terms gated by action/state confidence thresholds, and enable significant improvements over both human and baseline agent performance in sparse-reward high-dimensional games (Xiao et al., 2020).

5. End-to-End Pipelines and Evaluation Protocols

The standard pipeline for neural feedback integration comprises: (i) signal acquisition and labeling (e.g., ErrP or performance classes), (ii) signal-to-reward mapping via neural decoders (EEGNet, FNN, fNIRS regression), (iii) formulation of composite reward for RL, and (iv) empirical evaluation via standard RL metrics (learning curves, sample efficiency, final success rate). For reward modeling from feedback in LLMs, this is extended by self-mining preference pairs using follow-up likelihood signals (FLR), applying direct preference optimization (DPO), and validating on strong RM and alignment benchmarks (Zhang et al., 2024).

A sample integration pseudocode for EEG-based shaping (as in RLIHF) is:

$H(\cdot)$5

Empirical findings consistently show that implicit neural feedback signals—if decoded with sufficient accuracy (>60–70% depending on context)—are sufficient to densify reward, enable faster convergence, and often match or exceed the performance of policies trained with engineered dense rewards (Kim et al., 24 Nov 2025, Poole et al., 2021). Experiments account for subject variability, feedback noise, latency, and task complexity, with ablations confirming the impact of reward-weighting, signal class, and model confidence.

6. Limitations, Open Problems, and Future Directions

While neural feedback as a reward signal has demonstrated efficacy across a spectrum of domains, several technical and scientific challenges remain:

• Decoder robustness: As neural signals are subject to drift, motion artifacts, and inter-subject variability, ongoing research explores online adaptation, transfer learning, and uncertainty-aware shaping (Santaniello et al., 14 Jun 2025, Kim et al., 24 Nov 2025).
• Feedback quality and credit assignment: Noisy feedback (EEG classifiers with 65–85% accuracy) is tolerable up to a point; however, credit assignment across long action-histories and aligning neural latencies with eligibility traces remains an open problem (Poole et al., 2021).
• Scalability: Most demonstrations are in low- or moderate-dimensional domains. Extending to high-dimensional or multi-modal tasks (e.g., robotic manipulation with vision, LLM dialogue) requires more sample-efficient signal-to-reward models and tight RL integration (Kim et al., 17 Jul 2025, Zhang et al., 2024).
• Human workload and feasibility: Unobtrusive, passive feedback mechanisms (implicit ErrPs, fNIRS) minimize cognitive burden, but signal fidelity and the feasibility of continuous real-world integration are still being explored (Santaniello et al., 14 Jun 2025).
• Process-over-outcome supervision: In LLMs, process-based reward signals (critique similarity) yield more robust optimization than purely outcome-based rewards, but depend on high-quality reference critiques and raise cost/compute overheads (Wang et al., 12 Jan 2026).

Future research directions identified include adaptive or context-sensitive reward weighting, online co-adaptation of decoders and policies, multi-modal and cross-modal neural feedback integration, and rigorous benchmark development to quantify performance in real-world, safety-critical environments.

---

References

• "Neuron as a reward-modulated combinatorial switch and a model of learning behavior" (Rvachev, 2011)
• "BioLCNet: Reward-modulated Locally Connected Spiking Neural Networks" (Ghaemi et al., 2021)
• "Towards Interactive Reinforcement Learning with Intrinsic Feedback" (Poole et al., 2021)
• "Mapping Neural Signals to Agent Performance, A Step Towards Reinforcement Learning from Neural Feedback" (Santaniello et al., 14 Jun 2025)
• "Accelerated Robot Learning via Human Brain Signals" (Akinola et al., 2019)
• "Accelerating Reinforcement Learning via Error-Related Human Brain Signals" (Kim et al., 24 Nov 2025)
• "Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback" (Kim et al., 17 Jul 2025)
• "NeuRL: Closed-form Inverse Reinforcement Learning for Neural Decoding" (Kalweit et al., 2022)
• "Active Human Feedback Collection via Neural Contextual Dueling Bandits" (Verma et al., 16 Apr 2025)
• "FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback" (Xiao et al., 2020)
• "Aligning LLMs Using Follow-up Likelihood as Reward Signal" (Zhang et al., 2024)
• "Reward Modeling from Natural Language Human Feedback" (Wang et al., 12 Jan 2026)