Feedback-Conditional Policy Overview

• Feedback-Conditional Policy is a strategy where actions adapt based on real-time and historical feedback, including sensor data, human input, and environmental cues.
• In quantum control applications, FCPs significantly improve success rates, for example, raising quantum state steering success from 0.56 to 0.66 with fewer measurements.
• In human-in-the-loop and reinforcement learning settings, FCPs effectively integrate policy-dependent feedback to accelerate learning and boost robustness in complex tasks.

Feedback-Conditional Policy (FCP) refers to a broad family of learning and decision-making strategies in which the policy—mapping from states (or contexts) to actions—is adaptively conditioned on feedback signals that may arise from environment measurements, human instructions, model outputs, or other sources. Unlike static or open-loop policies, FCPs incorporate and respond to observed information in sequential decision processes, allowing for more effective, adaptive control and learning across diverse domains such as quantum systems, reinforcement learning, robotics, LLMing, and federated learning.

1. Foundations and Core Definitions

Feedback-Conditional Policies are defined by the principle that current and historical feedback information is used in policy selection or adjustment. Rather than pre-determined mappings or schedules, action choice at time step $k$ depends on a conditioning variable, which may be:

• Measurement results (quantum trajectory, sensor outputs)
• Policy-dependent human feedback (e.g., signals shaped by agent behavior)
• Relative feedback or corrections (e.g., directional cues)
• Verbal/natural language feedback (instructional, evaluative)
• Explicit environmental or mediator signals (trajectories sampled under past policies)

Formally, a feedback-conditional policy is a function $\pi(a|s, F)$, with $F$ summarizing relevant feedback up to the current step. This feedback may be discrete, continuous, probabilistic, or fuzzy, depending on the domain and context.

2. FCP in Quantum Measurement-Based Control

In quantum state manipulation, FCPs are applied to adaptively select measurement operators, leveraging the outcome of prior measurements to maximize success probability for steering a system toward a target state (Fu et al., 2014):

• Quantum state evolution is described by recursive application of measurement updates:

$$\rho_{k+1} = \mathcal{M}_{u_k}^{y_k}(\rho_k) = \frac{M_{u_k}(y_k) \rho_k M_{u_k}(y_k)^\dagger}{\mathrm{tr}[M_{u_k}(y_k) \rho_k M_{u_k}(y_k)^\dagger]}$$

• Optimal control problem: maximize $$J_\pi(N) = P_\pi(\rho_N = |\psi_{\text{target}}\rangle\langle\psi_{\text{target}}|)$$ over measurement selection policies.
• Dynamic programming is used to synthesize Markovian feedback policies, yielding recursively computed cost-to-go functions and selection rules:

$$V(t, x) = \max_{u \in \mathcal{A}} \sum_{y \in \mathcal{Y}} P(y|u, x) V(t-1, \mathcal{M}_u^y(x))$$

• Numerical examples show substantial improvement (e.g., for three measurements, closed-loop FCP raises success probability from 0.56 to 0.66, and for ten measurements from 0.8 to 0.9968).

Alternative objectives such as expected fidelity and minimal arrival time are also addressed, with FCP frameworks adjusted for each.

3. FCP in Human-in-the-Loop and Policy-Dependent RL

Feedback-conditional policies are central to interactive learning from human feedback, especially when feedback is shaped by the learner’s current policy (MacGlashan et al., 2017, Shah et al., 2021):

• Human feedback $f(s,a)$ is often policy-dependent: identical state–action pairs may be evaluated differently based on the agent’s exhibited behavior.
• COACH and E-COACH algorithms use the advantage function $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ to align human feedback with policy improvement, using actor–critic updates:

$$\Delta\theta_t = \alpha_t \nabla_\theta \pi(s_t, a_t) \frac{f_{t+1}}{\pi(s_t, a_t)}$$

• Eligibility traces and discounting mechanisms aggregate feedback over sequences for improved credit assignment and convergence.
• Empirical findings confirm that FCPs accelerate learning and enable stable acquisition of complex or compositional behaviors, outperforming baselines such as TAMER and Q-learning under policy-dependent or advantage feedback settings.

4. Feedback-Driven Exploration and Robustness

FCPs meaningfully enhance exploration and robustness in reinforcement learning and control:

• PPMP (Predictive Probabilistic Merging of Policies) weighs corrective feedback against policy uncertainty, dynamically scaling corrections during training (Scholten et al., 2019):

$$G = \Sigma_{a_p a_p} (\Sigma_{a_p a_p} + \Sigma_{hh})^{-1}$$

Final action is merged via $a = a_Q + \hat{e} \cdot h$ with selector modules and Q-filters.

• Relative feedback models (e.g., directional human cues) are efficiently integrated with off-policy RL, allowing rapid bootstrapping and adaptation to environmental or user changes (Schiavi et al., 7 Jul 2025).
• Sample efficiency and noise robustness are empirically validated in settings such as OpenAI Gym, robotic navigation, and sparse reward environments, often with significant reduction in required human labels.

5. Conditional Generation, Verbal Feedback, and Expressive Policy Adaptation

Recent FCP work moves beyond scalar reward-based learning, especially with LLMs or language-conditioned robotics:

• Verbal feedback is treated as a conditioning variable, reframing policy learning as conditional generation (Luo et al., 26 Sep 2025):
  • Offline data consists of $$(\text{instruction}, \text{response}, \text{verbal feedback})$$ triples.
  • Conditional maximum likelihood is performed on $$P_\text{off}(r|f,x) \propto \pi_\text{ref}(r|x)\,p_\text{env}(f | x, r)$$ via cross-entropy optimization.
  • Online bootstrapping: model generates under positive feedback condition, receives fresh annotations, and iteratively updates.
• This approach preserves feedback nuance, enables precise user control (e.g., stylistic, structural), and offers holistic data efficiency since every feedback instance informs policy behavior.

6. Multi-Modal, Dynamic, and Federated FCPs

FCPs are extended to multi-modal, dynamic, and distributed systems:

• Conditional policy generators for dynamic constraint satisfaction use RL with GAN-style architectures, conditioning on random noise and class labels to produce solution distributions matching both static (reward-based) and dynamic (class label) constraints (Lee et al., 21 Sep 2025).
• In personalized federated learning, sample-specific conditional policies separate global and personalized feature information, enhancing performance and privacy resilience during intermittent client participation (Zhang et al., 2023).
• Continual alignment with evolving human preferences is achieved by regularizing a current policy against historical optimal distributions, addressing catastrophic forgetting and leveraging replay buffers and scoring modules for unlabeled adaptation (Zhang et al., 2023).

7. Practical Implications and Future Directions

FCPs facilitate:

• Adaptive control under uncertainty, as in quantum state manipulation and robust robotics.
• Efficient interactive RL with nuanced human or model feedback, leveraging advantage-based learning, eligibility traces, and credit assignment.
• Data-efficient learning with conditional generation and bootstrapping, flexible to mixed or natural language feedback.
• Dynamic optimization and constraint satisfaction solutions via GAN- or RL-inspired conditional generation and multi-modal mapping.
• Personalized federated learning and privacy improvement by conditioning feature paths and heads at a fine-grained sample level.

Ongoing research addresses scaling FCPs to higher-dimensional problems, more complex feedback modalities (multi-turn dialogue, compositional tasks), integration with probabilistic and fuzzy causal structures, privacy-preserving aggregations, and real-time adaptation in dynamic environments.

---

The concept of Feedback-Conditional Policy thus encompasses a rigorous, flexible framework applicable across theoretical control, machine learning, interactive RL, federated systems, LLMing, and dynamic optimization, integrating diverse feedback forms to efficiently and robustly adapt policy behavior. With empirical and theoretical advances across these domains, FCPs now occupy a central role in contemporary approaches to adaptive learning and control.