DF-ExpEnse: Diffusion Filtered Exploration

• DF-ExpEnse is an exploration strategy that uses a pretrained diffusion policy to generate a small, multimodal set of action candidates for continuous-control tasks.
• It employs a UCB-style ranking with an ensemble of critics that combine conservative value estimates and uncertainty (disagreement) to select the most promising action.
• Fleet normalization and BC-SR regularization ensure collaborative exploration and preserve candidate diversity, markedly improving sample efficiency in manipulation and locomotion benchmarks.

Searching arXiv for the main paper and closely related work on diffusion-policy finetuning and action-candidate selection. arXiv.search query: "DF-ExpEnse Diffusion Filtered Exploration for Sample Efficient Finetuning"

DF-ExpEnse, short for Diffusion Filtered Exploration via Ensembles, is an exploration strategy for online reinforcement-learning finetuning of pretrained generative control policies, especially diffusion policies, in continuous-control robotics (Luo et al., 17 Jun 2026). It is designed as a drop-in replacement for the action-selection step inside an RL finetuning loop: instead of executing a single sampled action, it samples a small candidate set from the pretrained generative policy, scores those candidates with an ensemble of critics using a UCB-style criterion, and executes the candidate with the highest exploration interest. In fleet settings, it further normalizes value and disagreement statistics across parallel agents so that exploration becomes collaborative rather than redundant. The method is used only during experience collection; evaluation rollouts use the underlying policy directly without exploration filtering (Luo et al., 17 Jun 2026).

1. Problem setting and design objective

DF-ExpEnse is formulated for a standard infinite-horizon MDP with continuous action space $\mathcal{A} \subset \mathbb{R}^d$, transition dynamics $p(s_{t+1}\mid s_t,a_t)$, reward $r(s_t,a_t)$, and discount factor $\gamma \in (0,1)$ (Luo et al., 17 Jun 2026). The starting point is a pretrained generative control policy $\pi^{dp}_\theta(a\mid s)$, typically a diffusion policy or flow model trained offline via behavior cloning on demonstrations. Online finetuning then overlays RL on top of this pretrained controller, using off-policy updates while collecting new transitions $(s_t,a_t,r_t,s_{t+1})$ into a replay buffer.

The central problem addressed by DF-ExpEnse is not policy pretraining itself, nor critic learning, but the quality of online experience collection. The method assumes that a pretrained generative controller already captures feasible and multimodal action structure from offline data, and asks how to exploit that structure to explore more effectively during RL finetuning. This yields a specific architectural separation: the underlying RL algorithm and the pretrained generative controller remain largely unchanged, while the executed action at each timestep is replaced by a filtered choice over multiple policy-generated candidates (Luo et al., 17 Jun 2026).

The paper situates this design in two concrete finetuning regimes. For diffusion policies steered by noise, denoted DSRL, RL acts through a SAC-style noise policy $\pi_\theta^{\text{noise}}(\epsilon\mid s)$. For residual finetuning, denoted ResFiT, RL acts through a residual policy $$\pi_\theta^{\text{res}}(a^{\text{res}}\mid s,a^{dp})$$ that modifies the diffusion output. In both cases DF-ExpEnse changes only the action chosen for environment interaction.

2. Candidate generation and exploration-interest scoring

The method rests on two linked ideas: using a pretrained multimodal generative policy as a tractable candidate generator in continuous action spaces, and using a critic ensemble to rank those candidates by a combination of conservative value and uncertainty (Luo et al., 17 Jun 2026).

At state $s_t$, DF-ExpEnse samples a small candidate set

$[a_1,\dots,a_M] \sim \pi^{dp}_\theta(a\mid s_t).$

The candidate set is intended to be expressive but small enough to permit exhaustive critic scoring. The paper states that values such as $p(s_{t+1}\mid s_t,a_t)$0–7 are sufficient in practice, with diminishing returns beyond that range.

For each candidate $p(s_{t+1}\mid s_t,a_t)$1, an ensemble of critics

$p(s_{t+1}\mid s_t,a_t)$2

produces per-critic value estimates. DF-ExpEnse then computes a conservative value term

$p(s_{t+1}\mid s_t,a_t)$3

a disagreement term

$p(s_{t+1}\mid s_t,a_t)$4

and an exploration-interest score

$p(s_{t+1}\mid s_t,a_t)$5

The executed action is

$p(s_{t+1}\mid s_t,a_t)$6

This scoring rule is explicitly UCB-style. The $p(s_{t+1}\mid s_t,a_t)$7 term makes the estimate conservative, while the standard deviation across critics acts as a proxy for epistemic uncertainty or underexploration. Setting $p(s_{t+1}\mid s_t,a_t)$8 recovers pure exploitation over the candidate set, referred to in the paper as Max-Q. For $p(s_{t+1}\mid s_t,a_t)$9, the selection rule expresses optimism in the face of uncertainty. The experiments use a fixed $r(s_t,a_t)$0 across tasks (Luo et al., 17 Jun 2026).

The paper also examined sampling candidates proportionally to $r(s_t,a_t)$1, described as V-GPS-style weighted sampling, but reports that the default argmax rule is slightly more stable while performing similarly overall. This suggests that the main benefit comes from the structure of the candidate set and the uncertainty-aware ranking, rather than from stochasticity in the final selector.

3. Generative-policy interface and BC-SR regularization

DF-ExpEnse is agnostic to the exact generative controller provided that it can generate multiple i.i.d. action samples conditioned on state and that it is genuinely multimodal (Luo et al., 17 Jun 2026). In the reported experiments, RoboMimic and Gym tasks use diffusion or flow policies conditioned on proprioception and, where relevant, vision. In the diffusion case, inference consists of a deterministic or stochastic denoising process, such as DDIM, over a small number of denoising steps; in flow-policy variants, actions are produced by flow integration.

The role of multimodality is foundational. A diffusion policy trained on demonstrations may represent several plausible behaviors for the same state, such as different grasp points, approach angles, task strategies, or locomotion patterns. Sampling only a single action exposes only one mode at a time. By sampling several candidates, DF-ExpEnse converts the continuous action space into a manageable discrete subset containing diverse, policy-supported behaviors. The critic ensemble then operates over this filtered subset rather than over the full $r(s_t,a_t)$2, making UCB-style exploration tractable in continuous control.

A specific regularization device, Behavior Cloning Sampling Regularization (BC-SR), is introduced to preserve candidate-set diversity during finetuning. The concern is that online RL may collapse the finetuned policy toward fewer modes under exploitative pressure. DF-ExpEnse therefore keeps a frozen copy of the offline behavior-cloned prior $r(s_t,a_t)$3 and replaces $r(s_t,a_t)$4 current candidates with samples from that frozen model: $r(s_t,a_t)$5 yielding the final candidate set

$r(s_t,a_t)$6

The empirical role of BC-SR is emphasized strongly. Removing BC-SR from DF-ExpEnse hurts performance, while adding BC-SR to Max-Q substantially improves Max-Q, although not enough to match full DF-ExpEnse (Luo et al., 17 Jun 2026). This is presented as evidence that preserving multimodality in the candidate set is not incidental but a key enabler of effective uncertainty-aware exploration.

4. Critic ensembles and fleet-normalized exploration

DF-ExpEnse reuses the critic ensembles already present in the underlying RL algorithm, rather than introducing a separate uncertainty model (Luo et al., 17 Jun 2026). Ensemble diversity comes from random initialization and stochastic minibatch sampling over a shared replay buffer. The paper details separate update equations for DSRL and ResFiT, including action critics, noise critics in the DSRL case, and SAC-style residual critics in the ResFiT case, but the essential point is that DF-ExpEnse does not alter critic losses. It only changes how the ensemble is queried at action-selection time.

The paper also reports that replacing $r(s_t,a_t)$7 with mean-Q in the exploration-interest score yields similar performance, indicating some robustness to the conservative-value aggregator. Nonetheless, min-Q remains the default.

In fleet settings with $r(s_t,a_t)$8 parallel agents, DF-ExpEnse extends the same principle from single-agent exploration to collaborative exploration. Each agent $r(s_t,a_t)$9 generates its own candidates

$\gamma \in (0,1)$0

and computes local statistics

$\gamma \in (0,1)$1

Instead of selecting locally from $\gamma \in (0,1)$2, the system aggregates all $\gamma \in (0,1)$3 scores across the fleet and computes fleet-level means and standard deviations: $\gamma \in (0,1)$4

$\gamma \in (0,1)$5

Each agent then normalizes its candidates: $\gamma \in (0,1)$6

$\gamma \in (0,1)$7

and selects

$\gamma \in (0,1)$8

This fleet normalization is intended to reduce redundant exploration and induce division of labor across agents. The communication requirement is minimal: agents need to share only scalar $\gamma \in (0,1)$9 and $\pi^{dp}_\theta(a\mid s)$0 values per candidate per timestep, rather than states or network parameters. Empirically, disabling fleet normalization reduces sample efficiency relative to the fleet-normalized variant (Luo et al., 17 Jun 2026).

5. Integration into finetuning pipelines

The method is explicitly described as a plug-in exploration / action selection module (Luo et al., 17 Jun 2026). The unchanged components include the RL algorithm, critic architectures, replay buffers, learning rates, and pretrained diffusion or flow-policy architecture. The only replaced component is the action actually executed in the environment.

The resulting online loop is simple. At each environment step, the current state is observed; $\pi^{dp}_\theta(a\mid s)$1 candidates are generated from the generative policy, optionally with BC-SR; each candidate is scored by the critic ensemble; fleet normalization is applied if relevant; the best candidate is executed; the resulting transition is stored in the replay buffer; and critics and the RL steering policy are updated off-policy as in the vanilla finetuning pipeline. No changes are required in replay-buffer structure or in the optimization losses of the underlying RL method.

This modularity is important for both implementation and evaluation. Because DF-ExpEnse is used only during experience collection, test-time evaluation rollouts disable the exploration filter and run the underlying steering policy directly. A plausible implication is that the method aims to improve training-time sample efficiency without changing deployment-time latency or inference complexity.

The paper reports typical hyperparameter choices of $\pi^{dp}_\theta(a\mid s)$2, $\pi^{dp}_\theta(a\mid s)$3, and fleet size $\pi^{dp}_\theta(a\mid s)$4, and characterizes the added computation as modest relative to diffusion inference, especially when amortized across GPUs and parallel environments (Luo et al., 17 Jun 2026). Recommended practical ranges are $\pi^{dp}_\theta(a\mid s)$5–5 initially, $\pi^{dp}_\theta(a\mid s)$6 as a minimum with $\pi^{dp}_\theta(a\mid s)$7 as a robust setting, and $\pi^{dp}_\theta(a\mid s)$8 for BC-SR as a default prior-sample count.

6. Empirical results, limitations, and research position

The empirical study spans three domains (Luo et al., 17 Jun 2026). In manipulation, the evaluation includes RoboMimic tasks—Lift, Can, Square, Tool Hang—and DexMimicGen bimanual tasks—Can Sort, Box Cleanup, Coffee. In locomotion, it includes OpenAI Gym tasks—HalfCheetah-v2, Hopper-v2, Walker2D-v2. Results are reported as success rate versus environment timesteps for RoboMimic and DexMimicGen, and return versus timesteps for Gym, averaged over 3 random seeds with 100 evaluation rollouts per checkpoint.

The baselines are vanilla finetuning, Max-Q candidate selection, and model-based exploration baselines adapted to the same candidate sets. Vanilla finetuning corresponds to DSRL for diffusion steering and ResFiT for residual finetuning. Max-Q uses the same candidate generator as DF-ExpEnse but selects

$\pi^{dp}_\theta(a\mid s)$9

with no disagreement term and no fleet normalization; it is equivalent to DF-ExpEnse with $(s_t,a_t,r_t,s_{t+1})$0. The model-based baselines are E3B, in state-feature and DINO-embedding variants, and Plan2Explore.

The principal finding is that DF-ExpEnse consistently improves sample efficiency across manipulation and locomotion tasks relative to default finetuning and alternative action-selection schemes (Luo et al., 17 Jun 2026). The gains are described as especially pronounced on harder tasks, including RoboMimic Can, Square, Tool Hang, the locomotion tasks, and the DexMimicGen suite. By contrast, Max-Q often underperforms even vanilla DSRL or ResFiT, which the paper attributes to exploitative collapse onto suboptimal modes.

Ablation studies further isolate the contributing factors. Ensemble size matters: $(s_t,a_t,r_t,s_{t+1})$1 is reported as insufficient, while $(s_t,a_t,r_t,s_{t+1})$2–10 yields clearly better performance, with saturation around 5–10. Candidate set size $(s_t,a_t,r_t,s_{t+1})$3 gives similar results, with mild gains for 5 and diminishing returns beyond that. Fleet size helps up to roughly 4–16 agents, after which gains saturate. Mean-Q performs similarly to min-Q, and weighted sampling by exploration interest is comparable early in training but slightly less stable later. These findings collectively suggest that DF-ExpEnse is not driven by a single heuristic but by the interaction of multimodal candidate generation, critic disagreement, and—in parallel settings—fleet-level normalization.

The paper also identifies several limitations. Exploration filtering is applied at every timestep even though some tasks may require exploration only in specific phases; real-robot fleets may face latency or bandwidth constraints for sharing normalized statistics; performance depends on the quality and multimodality of the pretrained generative controller; and evaluating $(s_t,a_t,r_t,s_{t+1})$4 critics on $(s_t,a_t,r_t,s_{t+1})$5 candidates per step is materially more expensive than a single-critic query (Luo et al., 17 Jun 2026). No formal convergence guarantees are provided; the conceptual justification is instead tied to standard UCB and ensemble-uncertainty ideas.

Within the broader literature, the paper positions DF-ExpEnse at the intersection of diffusion-policy finetuning, ensemble-based exploration, action-candidate selection, and coordinated fleet learning. Its specific contribution is the combination of a multimodal generative prior for continuous-action candidate generation, an ensemble-based UCB-style ranking rule, BC-SR for preserving candidate diversity during finetuning, and fleet-normalized collaborative exploration (Luo et al., 17 Jun 2026). This suggests a general interpretation of DF-ExpEnse: not as a new RL objective, but as an action-selection layer that makes pretrained generative control policies substantially more sample-efficient during online adaptation.