Papers
Topics
Authors
Recent
Search
2000 character limit reached

Differentiable Weightless Controllers (DWC)

Updated 6 February 2026
  • DWCs are symbolic-differentiable controllers that use discrete logic circuits to map continuous observations to actions with energy efficiency.
  • They employ thermometer-encoded inputs and sparse boolean lookup-table layers to achieve interpretable, gradient-optimized policy learning.
  • Empirical results on MuJoCo tasks demonstrate that DWCs match or exceed neural network policies while enabling low-latency, power-efficient FPGA deployments.

Differentiable Weightless Controllers (DWCs) are a symbolic-differentiable architecture for continuous control, designed to represent and learn policies as discrete logic circuits rather than traditional continuous neural networks. A DWC maps real-valued state observations to action outputs through thermometer-encoded input vectors, sparsely connected layers of boolean lookup tables (LUTs), and lightweight action heads. The system is fully amenable to end-to-end gradient-based optimization, while enabling direct compilation to FPGA-compatible circuits with minimal latency and exceptionally low energy per action. Empirical evidence demonstrates that DWCs can achieve returns competitive with both full-precision and quantized neural network policies on a range of MuJoCo continuous control tasks, with distinct properties of structural sparsity and interpretability (Kresse et al., 1 Dec 2025).

1. Input Encoding via Thermometer Codes

A DWC accepts an observation x∈Rdinx \in \mathbb{R}^{d_{\rm in}} and first applies per-coordinate normalization and clipping: x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in} where μj,σj\mu_j, \sigma_j are running statistics. Each normalized value x^j\hat{x}_j is discretized using a BB-bit thermometer code: τm=s Φ−1(qm),qm=mB, m=1,…,B,s=10∣Φ−1(1B)∣\tau_m = s\,\Phi^{-1}\left(q_m\right), \quad q_m = \tfrac{m}{B}, \ m=1,\ldots,B, \quad s = \frac{10}{\left|\Phi^{-1}(\tfrac{1}{B})\right|} with Φ−1\Phi^{-1} as the inverse CDF of the standard normal. The thermometer code is: Ej(x^j)=[ 1{x^j≥τ1}, 1{x^j≥τ2}, …, 1{x^j≥τB} ]∈{0,1}BE_j(\hat{x}_j) = [\,\mathbf{1}\{\hat{x}_j \ge \tau_1\},\,\mathbf{1}\{\hat{x}_j \ge \tau_2\},\,\dots,\,\mathbf{1}\{\hat{x}_j \ge \tau_B\}\,] \in \{0,1\}^B Concatenating across all dimensions yields a binary vector b(0)∈{0,1}D0b^{(0)} \in \{0,1\}^{D_0}, where D0=BdinD_0 = B d_{\rm in}.

2. Boolean Lookup-Table Layers

DWCs are composed of x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}0 layers of boolean lookup tables. For each layer x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}1, x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}2 LUTs compute binary activations x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}3. Each LUT in layer x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}4 selects x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}5 inputs from the previous layer (learned indices x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}6), forming an address vector: x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}7 Each LUT x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}8 stores a truth-table x^j=clip(xj−μjσj, −10, 10),j=1,…,din\hat x_j = \mathrm{clip}\left(\frac{x_j - \mu_j}{\sigma_j},\ -10,\ 10\right), \quad j = 1, \ldots, d_{\rm in}9. The output is computed as: μj,σj\mu_j, \sigma_j0 The sparsity and wiring pattern of interconnects is learned; straight-through estimators facilitate differentiability during training.

Surrogate gradients for discrete truth-table updates are computed using Extended Finite Difference (EFD), a method that perturbs input bits and measures the downstream loss change, aggregating weighted finite differences over the Hamming ball of addresses: μj,σj\mu_j, \sigma_j1

3. Action Head and Output Parameterization

The final binary layer μj,σj\mu_j, \sigma_j2 is divided into μj,σj\mu_j, \sigma_j3 disjoint groups μj,σj\mu_j, \sigma_j4 (one per action dimension). For action μj,σj\mu_j, \sigma_j5: μj,σj\mu_j, \sigma_j6 A per-dimension affine transformation is then applied: μj,σj\mu_j, \sigma_j7 For Soft Actor-Critic (SAC) policies, μj,σj\mu_j, \sigma_j8 yields the action mean. All μj,σj\mu_j, \sigma_j9 parameters are updated via standard gradients.

4. Training and Optimization Strategy

DWCs are trained end-to-end using the SAC algorithm. Policy parameters consist of all LUT truth-tables (x^j\hat{x}_j0), LUT input selectors (x^j\hat{x}_j1), and output head parameters (x^j\hat{x}_j2). The policy loss is: x^j\hat{x}_j3 where x^j\hat{x}_j4 is entropy weight (automatically tuned), x^j\hat{x}_j5 are critics, and x^j\hat{x}_j6 is the replay buffer. Backpropagation flows through the differentiable action head, uniform popcount, LUT updates via EFD surrogate gradients, and LUT input selectors via straight-through estimation.

5. FPGA Deployment and Efficiency Characteristics

At deployment, all floating-point operations are replaced by discrete, hardware-compatible operations. For Xilinx Artix-7 XC7A15T FPGAs at 100 MHz, a two-layer DWC with x^j\hat{x}_j7 6-input LUTs per layer entails:

  • 3228 LUT6s and 1667 FFs used (no DSPs, no large BRAM, one small SRAM per action)
  • Two-stage pipeline for 100 MHz timing
  • Latency: 2–3 cycles (x^j\hat{x}_j8–x^j\hat{x}_j9 μs)
  • Throughput: BB0 actions/s
  • Energy per action: BB1 J

Comparatively, 2–3 bit quantized neural policies require thousands to hundreds of thousands of cycles per action, throughput as low as BB2 actions/s, and energy per action from BB3 J up to BB4 J.

6. Empirical Performance on Benchmark Tasks

DWCs were evaluated on Ant, HalfCheetah, Hopper, Humanoid, and Walker2d continuous control tasks in MuJoCo, each with 10 training seeds and evaluation over 1000 rollouts. Comparison was made to a 256-neuron full-precision MLP ("FP") and 2–3 bit quantized MLP ("Quant"):

Environment FP Return Quant Return DWC Return
Ant 5598 [4253,5802] 4717 [3888,4887] 5677 [5516,5906]
HalfCheetah 11529 [10113,11922] 10465 [9622,10956] 7549 [7097,7881]
Hopper 2797 [2062,3349] 1931 [1096,3270] 3120 [2777,3386]
Humanoid 6186 [5956,6650] 5954 [5800,6054] 6141 [5819,6605]
Walker2d 5044 [4697,5194] 4656 [4445,5019] 5025 [4509,5196]

Bracketed intervals indicate [25%, 75%] quantiles. For all tasks except HalfCheetah (which is capacity-limited), DWCs achieved returns equal or superior to both FP and quantized controllers. Capacity ablations show that for Ant, Hopper, Humanoid, and Walker2d, performance saturates at BB5. On HalfCheetah, performance increases up to BB6; a DWC with BB7 and BB8 matches full-precision returns. Further ablations on LUT arity (BB9) and bit resolution (τm=s Φ−1(qm),qm=mB, m=1,…,B,s=10∣Φ−1(1B)∣\tau_m = s\,\Phi^{-1}\left(q_m\right), \quad q_m = \tfrac{m}{B}, \ m=1,\ldots,B, \quad s = \frac{10}{\left|\Phi^{-1}(\tfrac{1}{B})\right|}0) indicate most tasks do not require high arity or fine bit-depth except for HalfCheetah.

7. Structural Sparsity and Interpretability

Sparsity and wiring patterns in DWCs provide interpretability absent from conventional neural policies. Analysis of first-layer LUT connectivity reveals:

  • On the Ï„m=s Φ−1(qm),qm=mB, m=1,…,B,s=10∣Φ−1(1B)∣\tau_m = s\,\Phi^{-1}\left(q_m\right), \quad q_m = \tfrac{m}{B}, \ m=1,\ldots,B, \quad s = \frac{10}{\left|\Phi^{-1}(\tfrac{1}{B})\right|}1 dimensional Humanoid task, 275 dimensions receive zero connections, indicating they are ignored.
  • The most-connected dimensions correspond to torso-velocity, highly correlated with reward.
  • Per-threshold connectivity patterns across thermometer bits reveal a bimodal emphasis on thresholds just to the left and right of the normalized zero-point, highlighting informative value regions.

This direct inspection of wiring elucidates which input dimensions and thresholds influence decisions, addressing interpretability by exposing the sparse, symbolic logic structure inherent to the trained policy (Kresse et al., 1 Dec 2025).

Definition Search Book Streamline Icon: https://streamlinehq.com
References (1)

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Differentiable Weightless Controllers (DWC).