Dual-Grained Efficiency-Aware RL

• The paper demonstrates that dual-grained RL divides control, reward, and memory into fast and slow components to optimize decision latency and sample efficiency.
• It employs specialized mechanisms like dual-memory replay, coarse/fine reward signals, and optimistic/pessimistic actors to address conflicting optimization objectives.
• Empirical evaluations across benchmarks show significant gains in convergence speed, task success rates, and computational resource allocation without sacrificing performance.

Searching arXiv for the cited papers to ground the article in current literature. Dual-Grained Efficiency-Aware Reinforcement Learning denotes a family of reinforcement-learning designs that organize learning, control, reward, or memory into two complementary levels in order to improve efficiency without discarding task performance. In the cited literature, the phrase does not refer to a single canonical algorithm. Rather, it appears in several distinct formulations: fast and slow decision systems in embodied agents, coarse and fine reward signals for manipulation, macro and micro optimization for multimodal search, optimistic and pessimistic actors for continuous control, dual-agent control of federated training, dual expectile-quantile estimation in distributional RL, and dual-memory replay structures (Dou et al., 13 May 2025, Chen et al., 28 Jan 2025, Jullien et al., 2023, Liu et al., 13 Feb 2026, Li et al., 8 May 2026, Yang et al., 2023, Ko et al., 2019). This suggests that the term currently functions as a design pattern: split a computationally expensive or statistically difficult RL problem into two interacting grains, then use their complementarity to trade off accuracy, latency, sample efficiency, or resource usage.

1. Scope and historical development

A dual-memory formulation appears in Ko & Chang’s "A Dual Memory Structure for Efficient Use of Replay Memory in Deep Reinforcement Learning" (Ko et al., 2019). There, the two grains are a large-capacity Main Memory $\mathcal{M}$ that preserves coverage and diversity, and a small-capacity Cache Memory $\mathcal{C}\ll\mathcal{M}$ on which prioritized operations are carried out efficiently. Later work extends the same dual-grained logic to other bottlenecks: exploration and utilization in OPARL (Yang et al., 2023), expectile and quantile estimation in distributional RL (Jullien et al., 2023), heterogeneous model allocation and training intensity in federated learning (Chen et al., 28 Jan 2025), fast and slow reasoning in DSADF (Dou et al., 13 May 2025), dual-granularity contrastive rewards in DEG (Liu et al., 13 Feb 2026), and trajectory-level plus token-level efficiency optimization in HyperEyes (Li et al., 8 May 2026).

Across these works, the efficiency target varies. In some cases it is wall-clock or inference latency, as in DSADF and HyperEyes (Dou et al., 13 May 2025, Li et al., 8 May 2026). In others it is sample efficiency, as in DEG, OPARL, and the dual-memory replay design (Liu et al., 13 Feb 2026, Yang et al., 2023, Ko et al., 2019). In HAPFL, efficiency is defined through overall training time and straggling latency in heterogeneous federated systems (Chen et al., 28 Jan 2025). In IEQN, efficiency refers to the statistical and optimization advantages of $L_2$-based updates while retaining a full return distribution (Jullien et al., 2023).

A common misconception is to treat dual-grained RL as synonymous with temporal hierarchy. The cited literature shows a broader picture. The two grains may be temporal, but they may also be representational, computational, supervisory, or infrastructural. This suggests that the central idea is not hierarchy alone, but the explicit separation of two roles that would otherwise interfere if forced into a single mechanism.

2. Canonical two-grain decompositions

The recurrent structural motif is the pairing of a cheap, local, or reactive component with a slower, more global, or more corrective component. The exact decomposition depends on the domain (Dou et al., 13 May 2025, Chen et al., 28 Jan 2025, Jullien et al., 2023, Liu et al., 13 Feb 2026, Li et al., 8 May 2026, Yang et al., 2023, Ko et al., 2019).

Formulation	Two grains	Efficiency target
DSADF	System 1 and System 2	latency and decision quality
HAPFL	Agent A and Agent B	training time and straggler latency
IEQN	expectiles and quantiles	efficient learning without collapse
DEG	coarse reward and fine reward	sample-efficient dense rewards
Dual-memory replay	Main Memory and Cache Memory	efficient prioritized replay
OPARL	optimistic actor and pessimistic actor	exploration and exploitation
HyperEyes	TRACE and OPD	tool-use efficiency and credit assignment

In DSADF, System 1 is a goal-conditioned RL agent with a memory space for fast and intuitive decision making, while System 2 is a VLM for deep and analytical reasoning that decomposes high-level tasks into sub-tasks, self-reflects to correct errors, and scores RL proficiency (Dou et al., 13 May 2025). In HAPFL, Agent A allocates heterogeneous model sizes and Agent B adjusts per-client training intensity shares, both using PPO (Chen et al., 28 Jan 2025). In IEQN, a single neural network predicts expectiles, while a mapper learns a quantile-to-expectile transformation so that the method can exploit asymmetric $L_2$ updates without losing distributional structure (Jullien et al., 2023). In DEG, the coarse-grained reward advances sequential imitation of a generated guidance video, while the fine-grained reward detects precise milestone matches (Liu et al., 13 Feb 2026). HyperEyes separates a macro-level trajectory reward, TRACE, from micro-level On-Policy Distillation, which injects dense token-level corrective signals on failed rollouts (Li et al., 8 May 2026).

These pairings are complementary rather than redundant. One grain typically absorbs the high-frequency control burden or the low-variance update burden, while the other handles long-horizon planning, exact correction, or adaptive cost control. A plausible implication is that dual-grained efficiency-aware RL is especially attractive when a single estimator or policy would otherwise face mutually incompatible objectives.

3. Formal objectives and coordination rules

Several papers make the dual-grained structure explicit through selectors, coupled rewards, or dual Bellman operators. In DSADF, coordination is governed by a selector

$$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$

where $p(g)$ is the stored proficiency from memory $\mathcal{M}$ and $\mathcal{T}$ is a threshold. The combined efficiency-aware objective is

$$J_{\mathrm{total}}=\mathbb{E}\Big[\sum_t R(s_t,a_t)-\lambda\,c(s_t)\Big],$$

with $c_2\gg c_1$ when System 2 is used (Dou et al., 13 May 2025). This formulation makes the cost of deliberation explicit rather than treating it as an external engineering constraint.

HAPFL formalizes its two grains through two rewards. For model allocation, Agent A uses

$\mathcal{C}\ll\mathcal{M}$0

which pushes the allocation toward balanced compute times across clients. For training-intensity adjustment, Agent B uses

$\mathcal{C}\ll\mathcal{M}$1

so maximizing the reward minimizes the straggler gap (Chen et al., 28 Jan 2025). The dual-grained formulation is therefore not only architectural; it is encoded directly in the optimization criterion.

In DEG, the reward is explicitly additive:

$\mathcal{C}\ll\mathcal{M}$2

and, when a sparse terminal success signal is available,

$\mathcal{C}\ll\mathcal{M}$3

The coarse reward is the gain in cosine similarity toward the current guidance frame, whereas the fine reward is a thresholded indicator tied to the best-matching frame that extends the reached prefix of the guidance sequence (Liu et al., 13 Feb 2026).

HyperEyes defines a trajectory reward

$\mathcal{C}\ll\mathcal{M}$4

where $\mathcal{C}\ll\mathcal{M}$5 depends on whether the rollout is correct, whether the number of tool-call rounds $\mathcal{C}\ll\mathcal{M}$6 and total tool invocations $\mathcal{C}\ll\mathcal{M}$7 stay within dynamically tightened reference budgets, and the rank of the trajectory within a sampled group (Li et al., 8 May 2026). The reference budgets $\mathcal{C}\ll\mathcal{M}$8 and $\mathcal{C}\ll\mathcal{M}$9 are updated monotonically during training, which directly couples policy improvement to efficiency improvement.

IEQN supplies a different kind of duality. Its dual Bellman operator is

$L_2$0

where $L_2$1 projects a distribution through jointly learned expectile and quantile statistics. The paper proves that, in the limit of infinitely many quantile and expectile fractions, the operator converges to the exact distributional Bellman operator (Jullien et al., 2023). Here, “dual-grained” refers not to action timing or system control, but to the coexistence of two asymmetric regression regimes with different efficiency and correctness properties.

4. Optimization mechanisms, memory structures, and supervision

Efficiency gains in these methods do not arise from decomposition alone; they depend on specific optimization and systems choices. DSADF uses a lightweight ImageBind backbone for images, a small CLIP text encoder for text+goal, a DQN head over actions, a fixed-size table of $L_2$2 pairs for memory, cached VLM plans and gating results per unique sub-goal, early-exit when System 1 succeeds quickly, and an annealed gating threshold $L_2$3 so that the agent relies more on its own policy over time (Dou et al., 13 May 2025). These are concrete mechanisms for converting a conceptual fast/slow split into a latency-aware decision system.

HAPFL uses two PPO agents at the server side. Agent A employs a multi-head MLP actor and a 3-layer fully-connected critic with clipping parameter $L_2$4, learning rate $L_2$5, and batch size $L_2$6. Agent B uses 4-layer fully-connected actor and critic networks with clipping $L_2$7, learning rate $L_2$8, and batch size $L_2$9 (Chen et al., 28 Jan 2025). On each client, a heterogeneous local model and a homogeneous LiteModel undergo mutual learning through knowledge distillation, with LiteModel aggregated by Fed-Avg. The dual-agent RL and the bidirectional distillation are jointly responsible for the reported efficiency gains.

The dual-memory replay design isolates expensive prioritization operations inside the cache. PER sampling uses

$L_2$0

while PSMM eviction uses

$L_2$1

The DQN loss is

$L_2$2

with all heavy priority computations carried out on $L_2$3 rather than $L_2$4 (Ko et al., 2019). This is a dual-grained infrastructural solution: preserve a broad history while restricting costly selection logic to a small working set.

OPARL uses two stochastic policies, $L_2$5 and $L_2$6, plus optimistic and pessimistic Q-ensembles. The optimistic target uses $L_2$7, the pessimistic target uses $L_2$8, and the optimistic actor is periodically warm-started by setting $L_2$9 every $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$0 steps (Yang et al., 2023). HyperEyes, by contrast, combines GRPO with On-Policy Distillation. On failed rollouts, it minimizes reverse KL from the student to the external teacher at each completion token:

$$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$1

This is a fine-grained supervisory signal layered on top of a coarse trajectory-level efficiency reward (Li et al., 8 May 2026).

5. Empirical behavior across domains

The reported gains are domain-specific but follow a consistent pattern: the dual-grained split improves an efficiency metric while preserving or improving task success. In Crafter, DSADF is evaluated on four benchmark tasks and compared against sparse-reward RL, curiosity RL, self-imitation RL, and LINVIT. In-domain, it reaches TSR 100% on wood-sword, 88% on stone-sword, 97% on iron-mine, and 100% on attack-cow, with lower completion times than baselines; it converges in ~0.5 M steps, whereas baselines require ~1–2 M. On the out-of-domain task "mine diamond" (15 steps), DSADF attains TSR 68% versus best baseline ≈21%, with time 2768 s versus ∼9–11 ks. On composite tasks with zombies, it reaches TSR 83% and survival 93%, while baselines reach TSR 30–75% and survival 61–81%. In Housekeep, DSADF attains Average Object Success Rate 92–96%, compared with RL sparse 71–79%, LINVIT 85%, GPT-4o alone 74–85%, and Qwen-2.5 70–89% (Dou et al., 13 May 2025).

HAPFL reports that overall training time is reduced by 20.9%–40.4% and straggling latency by 19.0%–48.0% compared to existing solutions. Compared to FedDdrl, training latency is down by 16.8%. In scalability tests with up to 100 clients and 50× performance variation, latency reduction remains ~37.8%. Ablation results attribute a 17.2% latency cut to model-allocation RL when training intensity is fixed, and a 19.2% cut to intensity-RL when model size is fixed (Chen et al., 28 Jan 2025).

IEQN matches the performance of the Huber-based IQN-1 baseline after 200M training frames but avoids distributional collapse and keeps estimates of the full distribution of returns. On Atari-5, the spread between the 0.1 and 0.9 quantiles is reported as 0.72 for IEQN versus 0.14 for IQN-1, while a pure expectile agent collapses to nearly deterministic value estimates and performs poorly (Jullien et al., 2023). DEG reports extensive experiments on 18 diverse tasks across both simulation and real-world settings. In reward-free Metaworld, it converges faster and to higher final success on 10/12 tasks despite using only 3–5 single-view videos versus tens of multi-view videos for TeViR. In real-world Franka experiments, DEG+ accelerated convergence and cut down needed human interventions by >50% over 25 K steps (Liu et al., 13 Feb 2026).

The dual-memory structure yields higher training and test scores than the conventional single memory structure in Assault-v0, SpaceInvaders-v0, and KungFuMaster-v0. Final test scores are reported as approximately $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$2 versus about $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$3 and $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$4 in Assault-v0; $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$5 versus about $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$6 and $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$7 in SpaceInvaders-v0; and $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$8 versus about $$g(g;\mathcal{M})= \begin{cases} 1 & \text{if } p(g)\ge \mathcal{T} \quad \text{(use RL)}\ 0 & \text{otherwise} \quad \text{(use VLM)} \end{cases}$$9 and $p(g)$0 in KungFuMaster-v0. Across all three games, DMS gives 2×–5× higher final test scores and converges in ≈60% fewer steps, with paired $p(g)$1-tests giving $p(g)$2 (Ko et al., 2019). OPARL reports average-return gains of 44.0%/51.3%/773.0% over SAC/TD3/PPO on MuJoCo Ant, and average normalized score gains of 14.3%, 4.1%, and 7.4% over REDQ, SAC, and TD3 on DMControl, with state-of-the-art results in 14/18 tasks (Yang et al., 2023). HyperEyes-30B surpasses the strongest comparable open-source agent by 9.9% in accuracy with 5.3x fewer tool-call rounds on average across six benchmarks, and introduces IMEB, a human-curated benchmark of 300 instances that jointly evaluates search capability and efficiency (Li et al., 8 May 2026).

Taken together, these results indicate that “efficiency” is not a single metric. It may denote lower per-decision latency, fewer tool-call rounds, faster convergence in environment steps, reduced straggler gaps, smaller wall-clock overhead, or improved distributional learning dynamics. The dual-grained pattern persists across all of these definitions.

6. Limitations, misconceptions, and future directions

The cited papers are explicit that dual-grained designs do not remove the underlying trade-offs; they redistribute them. DSADF depends on VLM APIs, incurring monetary cost and potential rate limits, and its VLM planner may generate impossible or sub-optimal sub-goals if not carefully prompt-engineered. Its gating threshold $p(g)$3 is static, and the paper notes that a learned gating policy might adapt better online (Dou et al., 13 May 2025). HyperEyes likewise treats inference efficiency as a first-class training objective, but its gains depend on a teacher model for OPD in the 30B variant and on reference budgets derived from successful trajectories (Li et al., 8 May 2026).

DEG reduces dependence on human annotations, but still requires a small number of expert videos for domain adaptation and a finetuned video-diffusion model for episodic guidance generation (Liu et al., 13 Feb 2026). IEQN assumes a shared mapper $p(g)$4 that may be violated by highly multimodal, state-dependent shapes, and the paper identifies extensions to risk-sensitive objectives and actor-critic frameworks as open directions (Jullien et al., 2023). OPARL introduces extra compute and storage for two actors and $p(g)$5 critics, requires tuning of ensemble size $p(g)$6, reset period $p(g)$7, and update frequencies, and does not provide a formal finite-sample complexity bound (Yang et al., 2023). HAPFL’s theoretical discussion distinguishes strongly convex and non-convex regimes for the mutual-distillation loss, but the broader systems problem remains sensitive to client heterogeneity and PPO policy quality (Chen et al., 28 Jan 2025).

A second misconception is to read “dual-grained” as merely duplicating model components. The stronger interpretation suggested by these works is functional specialization: each grain is assigned an objective that would be suboptimal if merged into a single update rule. This is visible in optimistic versus pessimistic Bellman targets, coarse versus fine reward channels, main versus cache memory, and trajectory-level versus token-level learning signals (Yang et al., 2023, Liu et al., 13 Feb 2026, Ko et al., 2019, Li et al., 8 May 2026).

The future directions stated in the papers are consistent with a broader move toward adaptive computation. DSADF proposes meta-gating with a learned network $p(g)$8, multi-agent extensions, on-device VLM distillation, and uncertainty estimates on VLM outputs (Dou et al., 13 May 2025). IEQN proposes learned fraction proposals and risk-sensitive objectives (Jullien et al., 2023). OPARL suggests adaptive balancing, multi-grained variants beyond two, and offline RL extensions (Yang et al., 2023). The dual-memory replay design suggests dynamic cache sizing and hierarchical or multi-agent settings (Ko et al., 2019). A plausible implication is that future work will preserve the two-grain principle while replacing fixed thresholds, fixed budgets, or hand-chosen decomposition rules with learned mechanisms for value-of-computation and uncertainty-aware allocation.