CAD Premium: Control Feedback in RL

• CAD premium is a formal measure quantifying how reinforcement learning methods achieve surplus by incorporating feedback effects directly into system dynamics.
• Advanced stochastic calculus techniques, including Malliavin calculus and the BEL representation, are used to derive bounds and sensitivity operators for quantifying incremental gains.
• Empirical findings indicate that while RL with CAD feedback may secure measurable premiums in finance and enhanced precision in robotic assembly, it requires careful risk management to address variance and phantom profit issues.

The Control-Affects-Dynamics (CAD) premium is a formal construct quantifying the surplus or edge obtainable by reinforcement learning (RL) strategies when their controls (actions) have explicit feedback effects on the evolution of the underlying system dynamics. In financial portfolio management, the CAD premium measures the difference in achievable performance between RL methods that internalize market feedback and myopic optimizers (MO) that treat controls as exogenous. This notion generalizes to other domains, such as robotic assembly, where CAD information encoded in geometric data can be leveraged to enhance sample efficiency and robustness through the direct integration of environmental structure into control and learning algorithms.

1. Definition and Mathematical Formulation of the CAD Premium

The CAD premium arises when the classical assumption of exogenous (non-influential) control on system dynamics no longer holds. Consider a stochastic process describing the system (e.g., portfolio state $Y_t$), traditionally modeled as:

$dY_t = V_0(t, Y_t)dt + V(t, Y_t) \circ dB_t,$

where $V_0$ and $V$ are drift and diffusion vector fields, and $B_t$ is Brownian motion. Under CAD, the control $P_t$ (e.g., trade decisions) enters the dynamics:

$$dY_t = [V_0(t, Y_t) + \varepsilon F_0(t, Y_t, P_t, D_t)] dt + [V(t, Y_t) + \varepsilon F(t, Y_t, P_t, D_t)] \circ dB_t \qquad (2.161)$$

where $F_0$ and $F$ represent feedback effects, $D_t$ collects auxiliary state (e.g., order book), and $\varepsilon$ is a tunable feedback coefficient.

The CAD premium is realized as the discernible incremental value:

$$A_{\mathrm{RL}-\mathrm{MO}}(\varepsilon) := J(\theta_{\mathrm{RL},*}) - J(\theta_{\mathrm{MO},*}) \quad \geq 0,$$

with $J(\cdot)$ the objective (expected utility, profit), $\theta_{RL,*}$ the RL-optimal parameter incorporating CAD feedback, and $\theta_{MO,*}$ the MO-optimal solution under exogenous dynamics.

A first-order expansion yields the bound:

$$0 \leq A_{\mathrm{RL}-\mathrm{MO}}(\varepsilon) \leq \varepsilon \cdot \mathcal{R} + O(\varepsilon^2), \qquad (2.166)$$

where $\mathcal{R}$ depends on temporary impact matrices, risk-shadow prices, and dual norms characteristic of the market structure. The premium density at each time $t$ is defined by

$$X_t := \mathbb{E}_t [Y_{o,t} J_t^+ - t9T + \mathrm{BEL}_{t,T_{Y_t}} 9T] \qquad (2.165)$$

where $\mathrm{BEL}_{t,T}$ maps diffusion perturbations forward and $J_t^+$ is the marginal gain associated with infinitesimal control perturbations.

2. Theoretical Foundations and Stochastic Calculus Approach

Evaluation and quantification of the CAD premium leverage advanced functional calculus, particularly Malliavin calculus, to analyze state-control sensitivity. Key technical steps include:

• Augmentation of the stochastic differential equations (SDEs) for the system state to include explicit control feedback.
• Definition of CAD sensitivity operators:

$$P_t^o = \partial_o F_0(t, Y_t, P_t, O_t), \qquad Q_t = \partial_o F(t, Y_t, P_t, O_t) \quad (2.162)$$

• First-order analysis using linear Stratonovich SDEs for state variations, supporting tractable computation of the CAD premium density.
• Utilization of the Bismut–Elworthy–Li (BEL) representation in conjunction with the Clark–Ocone formula, yielding explicit expressions for anticipative profit decomposition, policy gradients, and assessment of "phantom profit"—illusory gains arising from improper backtesting or model misspecification.
• Establishment of necessary conditions for premium realization via a Hamiltonian surplus constraint:

$$(X_t, \theta_{RL,*}) > (\theta_{MO,*} + \text{risk/execution terms}) \qquad (2.167)$$

3. Empirical Findings and Implications

Empirical and theoretical findings consistently indicate:

• In genuine CAD settings—where control has measurable market/state impact (e.g., through execution-induced price adjustments, regime shifts, or significant liquidity extraction)—the RL-driven policy can realize a strictly positive CAD premium, typically manifesting as a few basis points per execution episode in "strong" CAD regimes.
• However, in standard asset-immediate (AIM) or non-atomic double auction models, the CAD feedback vanishes or is negligible (Corollary 2.2), making the surplus effectively zero to leading order.
• The complexity overhead, risk profile, and variance floor associated with RL approaches often outweigh the incremental premium in normal, friction-limited environments, as documented by performance comparisons (lower or negative returns, higher cost, heavier CVaR, higher model risk for RL).
• Phantom profit contamination remains a notable risk; accordingly, Malliavin-based backtest decomposition is critical for isolating genuine execution gains from anticipative artifacts.

4. RL Versus Myopic Optimization: Comparative Analysis

Systematic comparison of RL and MO reveals:

Method	Feedback to Dynamics	Performance (CAD)	Variance/Floor
Myopic Optimization	None (exogenous)	No CAD premium	Low (PL) contraction
RL with CAD	Yes (endogenous)	Possible premium ($\varepsilon \mathcal{R}$)	Variance persists, higher model risk

• MO efficiently solves single-period convex programs, with rapid geometric error contraction under Polyak–Łojasiewicz conditions and minimal variance floor.
• RL may capture residual premium through systematic exploitation of feedback, but is susceptible to higher variance, phantom profit risk, and convergence issues if CAD intensity is weak.
• The surplus criterion in Equation (2.167) governs the regime where RL superiority is theoretically attainable.

5. Practical Applications and Model Auditing

Practical utilization and risk management of the CAD premium involve:

• Application domains where trading actions measurably move the underlying state—large-order execution, regime changes (as in Corollary 2.3), or illiquid markets—provide settings where RL tuned to CAD dynamics is justifiable.
• Audit methodology such as Malliavin policy-gradient decomposition (see Equations 2.125–2.134) can partition observed profit into genuine gains and leakage/phantom terms, providing ex-post validation and risk control.
• In robotic assembly, similar logic applies: leveraging CAD-derived geometric information to dynamically guide RL via motion planning costs yields dramatic improvements in sample efficiency, generalization, and success in contact-rich manipulation tasks without relying on state estimation accuracy (Thomas et al., 2018).

6. Relevance in High-Precision Industrial Assembly and Broader Implications

The "CAD Premium" in industrial robotics manifests as enhanced task success and adaptability by fusing geometric priors (from CAD files) into RL-driven policy synthesis. Notable findings:

• Using motion-plan–guided RL informed by CAD data, robots achieve substantially higher precision, success rates, and generalizability in challenging assembly tasks relative to either pure RL or classical planners.
• Neural network architectures explicitly encoding reference trajectories (with attention over CAD-derived waypoints) enable automatic compensation for object placement variability, essential in modern "high-mix" flexible manufacturing.
• The CAD premium lowers the trial complexity required to train robust controllers and mitigates policy convergence pathologies in high-precision applications.

In finance and robotics alike, the intrinsic value of CAD feedback is environment-dependent: substantial and critical in systems with strong state-control coupling, marginal or negligible otherwise. The necessity of rigorous audit and risk methodology is universal due to the persistent risk of phantom profit or spurious statistical performance. The concept of the CAD premium thus serves as a unifying formalism for understanding, quantifying, and exploiting the structural surplus achievable when control directly modifies system dynamics.