State-Aware Randomizer (STAR)

• STAR is a state-aware mechanism that applies context-sensitive randomization to both improve exploration in reinforcement learning and mitigate reliability issues in flash storage.
• In RL, STAR replaces static noise with a state-conditioned variational posterior, leading to notable performance gains on Atari benchmarks compared to traditional exploration methods.
• In SSD controllers, STAR dynamically selects optimal bit-flip masks to suppress weak data patterns, significantly extending device lifetimes and reducing read latencies.

The State-Aware Randomizer (STAR) denotes two distinct mechanisms in recent literature, unified by a core principle: introducing data- or state-conditioned randomization to improve real-world system performance. One application of STAR targets reinforcement learning (RL), where state-conditioned parameter perturbation is used to enhance exploration in Deep Q-Networks (DQN). Another addresses reliability in modern high-capacity SSDs, where STAR eliminates deleterious data patterns to mitigate retention errors from lateral charge spreading (LCS) in 3D NAND flash. Both implementations exploit state awareness to target and remediate domain-specific weaknesses, with rigorously quantified gains.

1. STAR in Reinforcement Learning: Variational Thompson Sampling for DQN

State-Aware Noisy Exploration (SANE), also referred to as STAR, is a generalization of NoisyNet-based exploration for RL agents. NoisyNet injects state-independent Gaussian noise into network parameters to induce exploration; SANE/STAR replaces this with a state-conditioned variational posterior, enabling the degree of exploration to be modulated according to the agent’s current environment context.

Variational Thompson Sampling Derivation

• The RL objective is to optimize over a parameter posterior $q(\theta)$ for the DQN weights, ideally matching the Bayesian posterior given data $\mathcal{D} = (X, Y)$.
• The variational objective maximizes the evidence lower bound (ELBO):

$$\text{ELBO}(q) = \int q(\theta) \log p(Y|X,\theta) d\theta - \text{KL}[q(\theta) \| p(\theta)]$$

• For squared-error loss (from the DQN’s TD target), the ELBO reduces to minimizing the expected squared Bellman error under parameter perturbations, with or without the KL term.
• In NoisyNet, $q(\theta)$ is state-independent: $q(\theta) = \mathcal{N}(\mu, \Sigma)$. In SANE/STAR, $$q(\theta|s) = \mathcal{N}(\mu, \text{Diag}(\sigma^2(h(s))))$$, with $\sigma$ state-conditional.

Network Architecture

Component	Structure / Functionality	Notes
Backbone DQN	3 conv layers, 2 FC layers, $|A|$ output units	Conv outputs serve as state features
Auxiliary perturbation module $g_\Theta$	FC layer (256, ReLU) → output $\sigma(s)$	$\sigma(s)$ used to scale parameter noise
Parameter partition	$\theta = \{\theta^b, \theta^p\}$ (conv, perturbed FC)	Only FC weights are perturbed

The perturbation module $g_\Theta$ receives activations after the last convolutional layer and emits a scalar $\sigma(s)$ controlling noise magnitude for all perturbed fully connected (FC) weights.

State-Dependent Reparameterization and Application

• At each action selection, the agent computes $\sigma(s) = g_\Theta(h(s;\theta^b))$.
• Noise realization:

$$\epsilon = \sigma(s) \cdot \hat{\epsilon}, \quad \hat{\epsilon} \sim \mathcal{N}(0, 1)$$

• All perturbed weights for FC layers share this $\sigma(s)$. The factored Gaussian parametrization is used for efficiency.
• The Q-value computation employs perturbed weights $W + \epsilon_w$, $b + \epsilon_b$ for input $x$.

End-to-End Optimization

• The DQN (with state-aware noise) and target network jointly compute targets and predictions, both with fresh independent draws of $\epsilon$ (see pseudocode below).
• Loss for minibatch:

$$L(\theta, \Theta) = \mathbb{E}_{(s,a,r,s') \sim \text{Replay}} \left\{ [Q(s,a;\theta,\Theta) - [r + \gamma \max_{a'} Q(s',a';\theta',\Theta')]]^2 \right\}$$

• Gradients flow through both backbone (including via reparameterization for perturbed weights) and auxiliary perturbation module.

Algorithm Pseudocode

Initialize Q-network θ = (θ^b, θ^p) and perturb module Θ Initialize target (θ', Θ') ← (θ, Θ) for step = 1 to MaxSteps do Observe current state s h ← conv_forward(s; θ^b) σ ← g_Θ(h) Sample standard noise 𝜖̂, set 𝜖 = σ · 𝜖̂ θ^p_pert ← θ^p + 𝜖 Choose a = argmax_a Q_forward(s; θ^b, θ^p_pert) Execute a, observe (s', r), store (s, a, r, s') in buffer Periodically: Sample batch from buffer For each sample i: Compute state features and σ, sample and apply noise for both Q-network and target Compute loss as squared TD-error Average loss, propagate gradients w.r.t. θ, Θ Periodically update target network parameters

Empirical Results

• On 8 Atari games with clear high/low-risk states, SANE/STAR achieves mean human-normalized score (HNS) 5.5 (Simple-SANE), versus 4.3 (NoisyNet) and 3.3 ($\epsilon$-greedy), with evaluation noise off.
• Keeping noise active at evaluation further improves Simple-SANE and Q-SANE.
• On control games with no evident risk states, SANE performance matches or slightly exceeds NoisyNet.
• Q-SANE (using non-noisy Q-value input to the perturbation module) marginally outperforms Simple-SANE, suggesting exploitation of additional context can tune exploration more effectively.

2. STAR in SSD Controllers: Lateral Charge Spreading Mitigation

In high-density 3D NAND SSDs, STAR is a hardware-optimized group-based data transformation layer that removes a broad class of “weak” patterns responsible for LCS-induced retention failures. This enables significantly longer device lifetimes and lower read latencies without modifying the flash chip itself.

LCS-Induced Retention Failures

• 3D NAND utilizes vertically stacked charge-trap (CT) cells sharing a silicon-nitride layer.
• Charge diffusion laterally through this layer—lateral charge spreading (LCS)—causes threshold voltage ($V_{th}$) drift, especially when high-$V_{th}$ cells neighbor low-$V_{th}$ (“worst-case” patterns).
• Empirical measurement reveals up to $4.6\times$ state differences in bit error rates, and tri-cell patterns like $E$–$P15$–$E$ (in QLC) or $E$–$P7$–$E$ (in TLC) can increase retention errors by an order of magnitude.

STAR Transformation Mechanism

Step	Function	Description
1	LFSR randomization	Applies standard data whitening (page level)
2	Bit-flip per group (128-cell granularity)	Evaluates all $2^m$ bit-flip masks per group; picks optimal mask to minimize group retention error

• Each $N$-cell page is partitioned into $G=128$-cell groups. For each group, all $2^m$ bit-flip masks (bits $b_0, \ldots, b_{m-1} \in \{0,1\}$; $m=4$ for QLC) are tested.
• The error metric uses pre-profiled per-state error probabilities $e_s$. The selected mask $f^*$ minimizes

$$\Delta E_{G,f} = \sum_{i=1}^G \left[ e_{f(s_i)} - e_{s_i} \right]$$

• This mask is recorded (using FIB metadata), ensuring on read the exact inversion is performed, preserving data consistency.

Hardware Pipeline and Overheads

• Datapath extension consists of LFSR randomizer, parallel error estimator (with $2^m$ parallel units), bit-flipper, and FIB encoder.
• Zig-zag I/O scheduling delivers all $m$ bits of each group together.
• Three-stage micro-pipelining across groups maintains throughput, amortizing per-group latency.
• Implementation overhead:
  • Latency: +100 ns per page (negligible versus $200$–$2\,000 \mu\text{s}$ program times)
  • Area: $0.036 \text{mm}^2$ ($<0.1\%$ SoC)
  • Power: +15.3 mW ($0.3\%$ SSD power)
  • Metadata: $0.7\%$ spare area

Controller Integration

• STAR operates transparently between the host FIFO and the flash DMA engine.
• On write: replaces the standard randomizer; on read: applies mask inversion followed by standard de-randomization.
• No changes to ECC code, wear-leveling, or GC are necessary.

Experimental Quantification

• Real chip characterization: 160 TLC/QLC chips, measuring raw bit error rates and weak pattern prevalence after up to 2 K P/E cycles and prolonged high-temperature retention.
• SSD emulation: Extended MQSim/Virt to model LCS and read-retry behavior, with workloads including Web, File, Mail, OLTP, Proxy.

Metric	TLC Baseline	TLC STAR	QLC Baseline	QLC STAR
Lifetime (K P/E cycles)	5.0	9.9	1.0	2.3
Top weak state reduction	–	30–42%	–	30–42%
Read latency reduction	–	28–46%	–	33–50%

• STAR suppresses weak QLC patterns (e.g., $P0$/$P1$ by $33.0\%$/$30.1\%$, $P14/P15$ by $41.6\%/38.6\%$).
• Top weak tri-cell pattern suppression: $72.3\%$.
• SSD lifetime increases: TLC $+98\%$, QLC $+130\%$.
• End-to-end read latency decrease: QLC $~50\%$, TLC $~46\%$ (across five workloads).

3. Comparative Summary of STAR Implementations

While the domain-specific mechanics differ, both STAR variants utilize data- or state-awareness to non-uniformly inject randomness, targeting weaker system points.

Application Area	State-Conditioned Target	Resulting Effect
RL (DQN)	Parameter noise for exploration	Risk-sensitive, adaptive exploration
SSD Controller	Data pattern masking, groupby state	Weak pattern suppression, longer life

The principle of modulating randomness contingent on input or operational state is central.

4. Limitations and Constraints

• RL STAR (SANE): Only the FC layers are perturbed; conv backbone remains deterministic. Optimality of this restriction is not fully explored. The demonstration focuses on Atari games; generalization to other RL domains is not addressed. No KL regularization is used in practice.
• SSD STAR: Offline error profiling is static; adaptation to wear-induced drift is not implemented. The group size is fixed; dynamic adaptation could refine the error/overhead trade-off. Application to >4 bits/cell (PLC) will require larger, but still practical, parallel estimation units.

5. Significance and Future Directions

In RL, state-aware noise enables the agent to tailor its exploration strategy adaptively, directly improving policy learning when high- and low-risk states coexist—a setting where uniform randomization can induce catastrophic failures or wasted exploration. In storage, STAR’s transparent and lightweight integration allows existing controllers to greatly extend flash lifetime and reduce read-service times, with only modest area and power cost.

Future work in both domains could encompass dynamic adaptation of state/error models, automated profiling, and application of the approach to other architectures or domains (e.g., PLC flash, multi-agent RL). The broad spectrum of STAR’s targeted randomness demonstrates its potential as a general technique where system vulnerabilities can be mapped to quantifiable state-dependent distributions.