Feedback Residuals in Control & Learning

• Feedback residuals are corrective signals added to base feedback systems to compensate for imperfections and improve system robustness.
• They integrate learned data-driven corrections with structured control laws, achieving efficient sample utilization and reduced estimator variance.
• Applications include reinforcement learning, derivative-free optimization, and wireless communications where they enhance performance and resilience.

A feedback residual is a general term for a corrective, additive, or complementary signal or estimator in a system governed by feedback that is designed to compensate for imperfections, limitations, or unmodeled variation in the primary feedback or control pathway. In modern research, this concept is central to a wide range of fields including reinforcement learning, black-box optimization, distributed online learning, signal processing, wireless communications, and information theory. The core idea is to superpose a data-driven correction (the residual) onto a structured feedback or control law to boost robustness, adaptability, or efficiency while maintaining desirable properties of the base system.

1. Fundamental Principles and Mathematical Formulations

A feedback residual is typically realized by decomposing the overall control, gradient, or signal update as

$$u_t^{\text{exec}} = u_t^{\text{base}} + u_t^{\text{res}},$$

where $u_t^{\text{base}}$ represents a nominal policy, controller, or feedback law—often based on hand-engineering, modeling, or imitation—while $u_t^{\text{res}}$ is a learned or adaptively optimized residual term. In learning scenarios, the residual term is tuned by maximizing cumulative rewards, minimizing regret, or optimizing other performance criteria under sparse or delayed feedback.

In black-box and derivative-free settings, residual feedback takes a statistical form, using the difference between function values or pseudo-gradient estimates at carefully chosen pairs of iterates to construct a low-variance estimator for descent directions, e.g.,

$$\widetilde{g}_t(x_t) = \frac{u_t}{\delta} \left[ f_t(x_t+\delta u_t) - f_{t-1}(x_{t-1}+\delta u_{t-1}) \right],$$

where only the first term requires querying the new function instance, thus preserving one-sample-per-iteration complexity (Zhang et al., 2020, Zhang et al., 2020).

In distributed multi-agent and game-theoretic environments, the residual corrects for feedback delays, asynchrony, and nonstationarity in "pseudo-gradients" or payoff dynamics through difference-based estimators or priority-buffered updates (Huang et al., 2023, Huang et al., 2023).

2. Residual Feedback in Reinforcement Learning and Control

Residual Reinforcement Learning (ResRL) employs a feedback-residual structure to decouple the tractable, model-driven components of a system from its complex, contact-rich or high-dimensional aspects. In robotic control, this is formalized as:

$$a_t^{\text{exec}} = \pi_0(s_t) + \pi_r(s_t, \pi_0(s_t)),$$

where $\pi_0$ is a base controller (PD, impedance, or a policy learned via behavioral cloning from demonstrations) and $\pi_r$ is a learned residual policy, typically trained by RL algorithms under sparse task-completion rewards (Alakuijala et al., 2021, Johannink et al., 2018).

Key features include:

• The base controller provides reliability and structure, reducing the burden on RL to within a manageable subset of the state-space.
• The residual policy is parametrized to output small corrective actions, learned with trust-region or actor-critic methods (e.g., distributional MPO).
• For systems with high-dimensional, image-based state spaces, fixing the visual backbone during residual policy learning avoids catastrophic forgetting and preserves low-level perception skills.
• In contact-rich manipulation with inner feedback (e.g., impedance controllers), residual learning must move beyond naïvely adding control signals, as the base controller may actively "fight" the residual due to internal feedback loops. The "residual feedback learning" formulation addresses this by permitting RL to directly adjust internal feedback references (setpoints), shifting the controller's "virtual goal" and enabling smooth, robust adaptation (Ranjbar et al., 2021).

3. Residual Feedback for Zeroth-Order Optimization and Bandit Learning

Derivative-free and bandit optimization settings classically face a trade-off between sample complexity and estimator variance. Traditional one-point estimators,

$$g^{(1)}_t = \frac{u_t}{\delta} f_t(x_t+\delta u_t),$$

suffer $O(1/\delta^2)$ variance, which is prohibitive for small smoothing radii. Two-point schemes,

$$g^{(2)}_t = \frac{u_t}{\delta}[f_t(x_t+\delta u_t) - f_t(x_t)],$$

ameliorate variance at the cost of needing two function queries per round—unrealistic when the underlying loss is nonstationary or only one measurement can be made per instance.

The residual feedback approach bridges this gap. By reusing the previous round's query,

$$\widetilde{g}_t = \frac{u_t}{\delta}[f_t(x_t+\delta u_t) - f_{t-1}(x_{t-1}+\delta u_{t-1})],$$

one maintains single-query-per-iteration efficiency and achieves comparable regret and convergence rates to two-point schemes even for nonconvex and distributed problems (Zhang et al., 2020, Zhang et al., 2020, Wang et al., 21 Mar 2025, Shen et al., 2021, Hua et al., 2024). The estimator is unbiased for the gradient of a smoothed loss and contracts in variance under mild bounded-drift assumptions. In asynchronous distributed optimization, agents cache prior measurements and use them as baseline for local block updates, yielding provable $O(T^{-1/3})$ nonconvex rates (Shen et al., 2021).

In multi-player continuous games, the residual pseudo-gradient (differences of one-point estimates over time) achieves $O(\delta_k^2)$ variance and accommodates delayed, asynchronous feedback, ensuring robust convergence properties with aggressive learning rates (Huang et al., 2023, Huang et al., 2023).

4. Feedback Residuals in Information and Communication Systems

Residual feedback is central in quantifying and regulating the impact of imperfect or quantized channel-state (CSI) feedback in multi-user wireless systems. In MIMO interference channels with cooperative (finite-rate) CSI feedback, the quantization of inner precoders inevitably produces residual interference ("feedback residuals") that cannot be suppressed by the base zero-forcing architecture alone (Huang et al., 2010, Huang et al., 2010).

Key methodologies include:

• Analytical quantification of the Grassmannian quantization error and its role in setting the interference floor, e.g.,

$$I_{\text{res}} \leq \nu M N_p \lambda_{\max} P_j \epsilon_j,$$

where $\epsilon_j$ is the precoder feedback residual.

• Joint design of inner precoders and equalizers to minimize the worst-case impact under finite-bit quantization.
• Scalar feedback loops that dynamically regulate transmit power based on real-time measurements of residual interference, employing fixed-margin, maximum sum-throughput, or outage-minimization criteria.
• The rate at which feedback bits must scale with SNR to prevent residual interference floors—linearly in $\log_2$SNR and the subspace dimensionality.

In information theory, the concept of residual directed information—defined as

$$I^W(X^n \to Y^n) = I(X^n \to Y^n) - I(X^n \to Y^n|W) = I(W;Y^n),$$

represents the message-bearing component of forward information flow in feedback channels, separating effective (message-related) capacity from spurious flow induced by noisy feedback. This provides both operational meaning and a basis for computable capacity bounds in feedback systems (Li et al., 2011).

5. Neural Network Architectures: Residual Feedback via Skip and Cross-Attention Connections

In deep neural network and equilibrium propagation architectures, "feedback residual" denotes structural modules that supplement canonical feedback pathways to enhance trainability and convergence:

• In brain-inspired recurrent neural networks, feedforward and feedback pathways are regulated to manage the spectral radius, ensuring fast contraction to equilibrium. Cross-layer residual (skip) connections are inserted to counter vanishing gradients in deep (weakly coupled) networks, enabling state-of-the-art performance with local, biologically plausible learning rules (Liu et al., 5 Aug 2025).
• In massive MIMO CSI feedback, transformer-based architectures integrate residual cross-attention blocks that fuse local user channel embeddings with complementary features from neighboring users. The cross-attention residual computes the difference between the user's embedding and a multi-head attention fusion, which is then projected, normalized, and propagated. These residual modules are embedded in multi-user decoder stacks, supporting performance gains under tight feedback and uplink SNR constraints (2505.19465).

6. Empirical Impact and Applications

Feedback residual architectures consistently demonstrate improved sample efficiency, robustness, and generalization across a diverse set of domains:

• In robotic manipulation, residual RL from behavioral-cloning or conventional controllers achieves $>95$% success in high-DoF, sparse-reward tasks—significantly outperforming RL-from-scratch or RL-only finetuning, particularly in novel scenarios beyond the demonstration distribution (Alakuijala et al., 2021).
• In distributed zeroth-order and online optimization, one-point residual feedback methods approach the dynamic regret performance of classical two-point schemes while maintaining minimal sampling and communication cost—even with rapidly varying objectives, heterogeneous network topologies, and asynchronous updates (Hua et al., 2024, Shen et al., 2021, Wang et al., 21 Mar 2025).
• In multi-user wireless feedback, residual-regulated architectures recover much of the performance lost to quantization and limited-rate feedback by scalar power-control loops and inner-precoder assignment, avoiding interference floors at high SNR (Huang et al., 2010, Huang et al., 2010).
• In astrophysical environments (AGN feedback in galaxy clusters), the term "residual cooling" denotes persistent, spatially structured cooling flows that survive powerful AGN heating, fueling ongoing star formation and black hole accretion at $4{-}8$\% of classical rates despite extensive feedback—a direct astrophysical analogy to residual feedback phenomena maintaining essential system function amid strong regulatory mechanisms (Tremblay et al., 2012).

7. Practical Considerations and Design Guidelines

Several universal principles emerge for deploying feedback residual methodologies:

• The base feedback or control law should be strong enough to ensure baseline performance and stability, reducing state exploration demands.
• The residual module must be parametrized and regularized to produce controlled corrections—residual magnitude constraints, network freezing, and adaptive weighting are standard (Alakuijala et al., 2021).
• In distributed or asynchronous systems, memory-efficient caching is critical for maintaining bias and variance reduction, and stepsizes must accommodate the bounded variation in local functions or payoffs (Shen et al., 2021, Wang et al., 21 Mar 2025).
• In communication and signal processing, the dimensionality of feedback residual quantization must be scaled with SNR and system degrees-of-freedom to avoid throughput "floors" or capacity penalties (Huang et al., 2010, Li et al., 2011).
• In deep learning, the placement and structure of residual connections (skip, cross-attention, or spectral regulation) should match the feedback pathway's locality, network depth, and dynamical constraints, balancing convergence speed and representation power (Liu et al., 5 Aug 2025, 2505.19465).

---

References:

• Residual RL for robot manipulation and learning from demonstration: (Johannink et al., 2018, Alakuijala et al., 2021)
• Residual feedback learning for contact-rich systems: (Ranjbar et al., 2021)
• One-point residual feedback in black-box and distributed optimization: (Zhang et al., 2020, Zhang et al., 2020, Shen et al., 2021, Wang et al., 21 Mar 2025, Hua et al., 2024, Huang et al., 2023)
• Feedback residual in MIMO wireless/interference channels: (Huang et al., 2010, Huang et al., 2010)
• Residual directed information for noisy-feedback channels: (Li et al., 2011)
• Residuals in large-scale neural net architectures and equilibrium propagation: (Liu et al., 5 Aug 2025, 2505.19465)
• Residual cooling and feedback in galaxy clusters: (Tremblay et al., 2012)