ReLMXEL: RL for DRAM Controller Optimization

• The paper introduces ReLMXEL, a multi-agent reinforcement learning framework that dynamically tunes DRAM control settings via reward decomposition to optimize energy, latency, and bandwidth.
• It employs detailed DRAM behavior metrics and a SARSA-based, ε-greedy strategy to adapt controller parameters in response to evolving workload demands.
• Experiments using DDR4 simulations show that ReLMXEL outperforms static controller configurations by achieving better energy efficiency and cumulative rewards with transparent decision-making.

Searching arXiv for the cited papers to ground the article. Search query: ReLMXEL (Sai et al., 18 Mar 2026) ReLMXEL, short for Reinforcement Learning for Memory Controller with Explainable Energy and Latency Optimization, is an online, multi-agent reinforcement learning framework embedded inside the memory controller to dynamically tune DRAM control policies. It is designed for settings in which static controller configurations become suboptimal under changing workload behavior, and it couples online adaptation with an explicit explainability layer based on reward decomposition. In the formulation presented in the paper, ReLMXEL observes detailed memory behavior metrics, selects controller parameters, learns with SARSA and $\epsilon$-greedy exploration, and justifies decisions in terms of metric-level tradeoffs involving energy, latency, bandwidth, and related DRAM behaviors (Sai et al., 18 Mar 2026).

1. Terminology and scope

ReLMXEL is a memory-systems method rather than a language-model evaluation method. The title expands the acronym as Reinforcement Learning for Memory Controller with Explainable Energy and Latency Optimization, and the paper positions it as a framework for energy, latency, and related memory-system optimization inside the DRAM controller (Sai et al., 18 Mar 2026).

The distinction matters because a separate line of work uses the term ReLM for a formal-language interface to LLM generation. The reproduction study of that work explicitly states that it “does not introduce a different method called ‘ReLMXEL’; rather, the relevant terminology in this paper is ReLM itself,” where ReLM denotes a system for constraining, evaluating, and analyzing LLM outputs via regular expressions, finite automata, and LLM-specific automata (Adamson et al., 16 Apr 2025). ReLMXEL is therefore a separate method with a different problem domain, objective, and technical stack.

Within memory systems, the motivating problem is the difficulty of selecting DRAM controller parameters under workload-dependent behavior. The paper emphasizes that page policy, scheduler policy, buffer organization, arbitration behavior, refresh policy, and transaction limits interact with access locality, row-buffer hit rate, bank conflicts, refresh interference, and bandwidth usage. A fixed configuration can therefore perform well for one workload and poorly for another, which motivates online adaptation (Sai et al., 18 Mar 2026).

2. Control target and tunable parameter space

ReLMXEL treats the memory controller as an environment in which RL agents observe fine-grained DRAM behavior metrics and learn how to choose controller parameters that better balance performance tradeoffs. The actions are not limited to issuing individual DRAM commands; they are selections over a structured controller configuration space (Sai et al., 18 Mar 2026).

Parameter	Options or range
PagePolicy	Open, OpenAdaptive, Closed, ClosedAdaptive
Scheduler	FIFO, FR-FCFS, FR-FCFS Grp
SchedulerBuffer	Bankwise, ReadWrite, Shared
Arbiter	Simple, FIFO, Reorder
RespQueue	FIFO, Reorder
RefreshPolicy	NoRefresh, AllBank
RefreshMaxPostponed	$0,\dots,7$
RefreshMaxPulledin	$0,\dots,7$
RequestBufferSize	configurable parameter
MaxActiveTransactions	$2^x$, $x=0,\dots,7$

These knobs jointly affect row activation behavior, scheduling, refresh timing, buffering, and concurrency. The paper’s framing is that controller quality cannot be reduced to a single setting choice in isolation, because the benefit of one policy depends on the current access pattern and on the accompanying configuration of the other policies. This suggests that ReLMXEL is intended for adaptive policy composition rather than one-shot design-space selection.

3. Multi-agent online reinforcement learning architecture

ReLMXEL is described as a multi-agent RL system rather than a monolithic controller. The framework associates parameters and reward components with multiple agents and Q-tables, and each agent interacts with the memory-controller environment while learning from feedback derived from observed DRAM metrics (Sai et al., 18 Mar 2026).

The RL interaction is written in the standard form

$S_t \xrightarrow{A_t} R_t,\, S_{t+1},$

where at timestep $t$, the agent observes state $S_t$, selects action $A_t$, receives reward $R_t$, and transitions to $0,\dots,7$0. The algorithm initializes the number of agents $0,\dots,7$1, a base seed $0,\dots,7$2, per-agent seeds $0,\dots,7$3, a warmup or exploration threshold $0,\dots,7$4, a learning rate $0,\dots,7$5, and a discount factor $0,\dots,7$6. During the warmup period the controller explores more broadly, and after the threshold $0,\dots,7$7, exploration is reduced via a smaller $0,\dots,7$8 so that learning becomes more exploitative and stable (Sai et al., 18 Mar 2026).

Learning uses the SARSA update rule: $0,\dots,7$9 Each configurable parameter is associated with a Q-table, and action selection follows an $0,\dots,7$0-greedy policy. In exploitation mode, the selected action is

$0,\dots,7$1

so the chosen action maximizes the summed Q-values across reward components. The controller proceeds iteratively by applying the current joint action vector $0,\dots,7$2, observing DRAM metrics, computing reward components, updating $0,\dots,7$3 with respect to the warmup threshold, choosing the next action vector $0,\dots,7$4, and updating each agent’s Q-values using SARSA (Sai et al., 18 Mar 2026).

A further mechanism is the trace split parameter, which partitions a trace into fixed-size segments. Rather than adapting once over the entire trace, the controller can adapt repeatedly over time. The paper presents this as support for online adaptation to workload phase changes (Sai et al., 18 Mar 2026).

4. State signals, reward decomposition, and explainability

A major feature of ReLMXEL is that it does not optimize from a single scalar indicator alone. The paper explicitly uses the following memory behavior metrics as state and reward signals: Latency per request, Average power, Total energy consumption, Bandwidth utilization, Bank switches, Bank-group switches, Row-buffer hits, and Row-buffer misses (Sai et al., 18 Mar 2026).

The role of these metrics is operational rather than merely descriptive. Latency reflects request delay; power and energy quantify DRAM cost; bandwidth captures interface utilization; bank and bank-group switching capture locality and conflict behavior; and row-buffer hits and misses reflect how well the page policy and access patterns align. The paper defines a reward for each metric $0,\dots,7$5 so that the closer the observed metric is to the target, the larger the reward, and it defines the total reward as the sum of the per-metric rewards,

$0,\dots,7$6

This decomposition is central to the framework because the controller learns from a vector of objectives corresponding to latency, energy, bandwidth, and other DRAM-behavior features rather than from a single undifferentiated objective (Sai et al., 18 Mar 2026).

The explainability mechanism follows reward decomposition in the sense of Juozapaitis et al. The paper decomposes the Q-function into component-wise Q-values $0,\dots,7$7, so that action preference can be justified by comparing

$0,\dots,7$8

To explain why one action is preferred over another, it uses Reward Difference Explanation (RDX): $0,\dots,7$9 where each component $2^x$0 measures how much better $2^x$1 is than $2^x$2 with respect to reward component $2^x$3. When there are many components, the paper uses Minimal Sufficient Explanation (MSX), defining a positive subset $2^x$4 as the smallest subset of components whose total advantage outweighs the total disadvantage, with an additional necessity check based on whether a subset of negative components $2^x$5 can offset that advantage (Sai et al., 18 Mar 2026).

In practical terms, the explanation can take forms such as “Choose closed-page policy because the energy gain outweighs the latency penalty,” or “Choose reorder arbitration because the combined latency and bandwidth benefits justify the higher energy cost.” The paper presents this as its accountability mechanism: controller decisions are traceable to measurable tradeoffs rather than treated as opaque outputs of an RL policy (Sai et al., 18 Mar 2026).

5. Experimental setup and evaluation protocol

The evaluation is simulation-based. ReLMXEL is tested on DDR4 memory simulated in DRAMSys, with DRAMPower used for energy estimation. The modeled system includes a burst length of 8, 4 bank groups, 4 banks per group, 32,768 rows per bank, 1024 columns, 8-byte device width, and single-channel, single-rank $2^x$6 DRAM devices (Sai et al., 18 Mar 2026).

The baseline memory controller uses OpenAdaptive page policy, FR-FCFS scheduling, a bank-wise scheduler buffer with up to 8 requests, All-bank refresh with up to 8 postponed and 8 pulled-in refreshes, 128 active transactions, and request reordering arbitration. Relative to this baseline, the evaluation reports baseline reward versus ReLMXEL reward, average energy, average bandwidth, and average latency. For energy and latency, lower is better; for bandwidth, higher is better (Sai et al., 18 Mar 2026).

The workload suite spans both synthetic and benchmark traces. The evaluated traces include GEMM, STREAM, BFS, and SPEC CPU 2017 workloads divided into memory-intensive (fotonik_3d_s, mcf_s, lbm_s, roms_s), compute-intensive (xalancbmk_s, gcc_s), and mixed or network-simulation style (omnetpp_s) categories. Trace generation uses Intel Pin Tool for some workloads and ChampSim for SPEC CPU 2017 traces captured via LLC misses over long simulations (Sai et al., 18 Mar 2026).

The design-space exploration considers

$2^x$7

The selected evaluation settings are $2^x$8, $2^x$9, trace split parameter $x=0,\dots,7$0, and $x=0,\dots,7$1 (Sai et al., 18 Mar 2026).

6. Reported results, significance, and limitations

The paper’s central empirical claim is that ReLMXEL consistently outperforms the baseline across the tested workloads, with higher cumulative reward for every workload and strong gains in energy reduction and bandwidth improvement, while latency is generally maintained or only slightly affected (Sai et al., 18 Mar 2026).

Representative observations are workload-specific. For STREAM, the reported reward improves from 15555.06 to 17597.07, with energy reduced and bandwidth increased. For GEMM, reward increases from 6572.88 to 7121.46. BFS shows improved reward and lower energy. fotonik_3d shows a substantial reward increase and strong energy improvement. roms and lbm show especially strong gains in reward and bandwidth. mcf and omnetpp improve in energy and bandwidth even though latency may slightly increase, reflecting an explicit tradeoff. The paper further notes that memory-bound workloads such as STREAM and GEMM benefit from improved energy efficiency and bandwidth utilization, irregular and graph-based workloads such as BFS and fotonik_3d also see good gains due to controller adaptation, compute-intensive workloads like gcc and xalancbmk show improved scheduling behavior, and heavy-traffic workloads like mcf and omnetpp reveal tradeoffs where latency can rise slightly even as energy and bandwidth improve (Sai et al., 18 Mar 2026).

The significance of explainability is presented as substantive rather than decorative. The paper attributes Transparency, Accountability, Interpretability, and Trustworthiness to the combination of reward decomposition, RDX, and MSX. This matters because memory policies involve competing objectives: improving energy may worsen latency, increasing row-buffer hits may alter bandwidth behavior, and refresh postponement can save energy but risks interference. ReLMXEL makes these tradeoffs explicit in terms of measurable components (Sai et al., 18 Mar 2026).

The paper also indicates several limitations and future directions. The evaluation is simulation-based, and workload-specific tuning remains relevant; although one $x=0,\dots,7$2 pair is selected across workloads for evaluation, the paper notes that each workload can have its own optimal hyperparameters. A further limitation is that the quality of explanation depends on the chosen reward decomposition and therefore on how well the selected metrics capture the true operational tradeoffs. Suggested extensions include heterogeneous memory architectures, including hybrid nonvolatile memory systems, hardware-in-the-loop setups to bridge simulation and real hardware, and RL-based DRAM security applications such as detecting and mitigating rowhammer attacks by recognizing malicious access patterns and adjusting controller behavior (Sai et al., 18 Mar 2026).