Density-Ratio Weighted Behavioral Cloning

• Density-Ratio Weighted BC is an offline imitation method that reweights contaminated trajectories via importance weighting to mimic clean expert behavior.
• It employs a small, verified clean reference set and a discriminator to estimate trajectory-level density ratios that down-weight low-quality samples.
• Empirical evidence demonstrates that Weighted BC significantly improves performance and robustness across various contamination scenarios compared to traditional BC and offline RL baselines.

Density-Ratio Weighted Behavioral Cloning (Weighted BC) denotes a family of offline imitation-style methods that replace uniform behavioral cloning with sample weights intended to correct a mismatch between the data distribution used for training and the target behavior distribution. In the most direct formulation, the target is the clean expert trajectory distribution $p_{\mathrm{clean}}(\tau)$, the available data come from a contaminated mixture $p(\tau)$, and the learning objective is the clean behavioral cloning risk rewritten by importance weighting with the trajectory-level ratio $p_{\mathrm{clean}}(\tau)/p(\tau)$ (Pandian et al., 1 Oct 2025). Closely related work uses discriminator outputs, expert-versus-suboptimal density contrasts, return-space reweighting, or posterior smoothing to alter the effective behavioral cloning objective, but these mechanisms are not equivalent; some are best understood as proxies or adjacent formulations rather than canonical density-ratio weighted BC (Xu et al., 2022, Zhang et al., 2024, Nguyen et al., 2022, Wagenmaker et al., 18 Dec 2025).

1. Formal setting and objective

The direct Weighted BC formulation is posed in an offline Markov decision process

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$

with a fixed dataset of trajectories

$\mathcal{D} = \{\tau_i\}_{i=1}^N,$

where each trajectory is

$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$

The policy is

$\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$

The goal is to match clean expert behavior even when the training data are contaminated (Pandian et al., 1 Oct 2025).

The contamination model is defined at the trajectory level: $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$ Here $p_{\text{clean}}$ is the clean expert trajectory distribution, $p_{\text{bad}}$ is an arbitrary contamination distribution, and $p(\tau)$0 is the contamination fraction. The targeted corruption modes include adversarial poisoning, system or sensor errors, annotation or logging errors, and low-quality or anomalous samples; the reported experimental protocols instantiate reward poisoning, state poisoning, transition poisoning, and action poisoning (Pandian et al., 1 Oct 2025).

Under standard BC, all data are treated equally through an average negative log-likelihood objective. In the paper’s notation, the contaminated-data loss is

$p(\tau)$1

Because $p(\tau)$2 under contamination, this objective is biased away from the desired clean expert objective. The same paper further argues that offline RL baselines such as BCQ and BRAC also degrade under contaminated data because they optimize using the corrupted replay distribution and therefore “implicitly treat all trajectories as equally reliable” (Pandian et al., 1 Oct 2025).

2. Density-ratio construction and weighted BC loss

The central Weighted BC mechanism introduces a small verified clean reference set

$p(\tau)$3

This reference set is trusted, drawn from the clean expert distribution, strictly disjoint from the main training set, and used only to guide weighting rather than to directly train the policy (Pandian et al., 1 Oct 2025).

The ideal clean BC objective is

$p(\tau)$4

With the trajectory imitation loss

$p(\tau)$5

the clean risk is

$p(\tau)$6

Weighted BC uses the identity

$p(\tau)$7

so the required weight is the trajectory-level density ratio

$p(\tau)$8

The formulation is explicitly trajectory-based rather than state-wise or transition-wise; every transition inside a trajectory receives the same trust score (Pandian et al., 1 Oct 2025).

Because direct density estimation is intractable, the method trains a binary discriminator

$p(\tau)$9

to distinguish class-1 clean reference trajectories from class-0 trajectories in the contaminated main dataset, using

$p_{\mathrm{clean}}(\tau)/p(\tau)$0

Under balanced sampling, the ideal discriminator is

$p_{\mathrm{clean}}(\tau)/p(\tau)$1

which yields the odds-ratio estimator

$p_{\mathrm{clean}}(\tau)/p(\tau)$2

In the ideal case,

$p_{\mathrm{clean}}(\tau)/p(\tau)$3

Thus the density ratio is recovered from discriminator outputs (Pandian et al., 1 Oct 2025).

For numerical stability, the raw ratio is clipped: $p_{\mathrm{clean}}(\tau)/p(\tau)$4 with reported default constants

$p_{\mathrm{clean}}(\tau)/p(\tau)$5

The paper explicitly states that the weights are not renormalized, that clipping is used for numerical stability, that low-density trajectories are down-weighted, that highly suspicious trajectories can be almost discarded through the lower clip, and that very large weights are capped to avoid variance explosion. The weights are precomputed after discriminator training and then frozen during policy optimization (Pandian et al., 1 Oct 2025).

The final Weighted BC objective is

$p_{\mathrm{clean}}(\tau)/p(\tau)$6

The reported algorithmic pipeline has three stages: train the discriminator on balanced batches from $p_{\mathrm{clean}}(\tau)/p(\tau)$7 and $p_{\mathrm{clean}}(\tau)/p(\tau)$8, compute and clip density-ratio estimates for every trajectory in $p_{\mathrm{clean}}(\tau)/p(\tau)$9, and then train the policy by weighted behavioral cloning with the frozen trajectory weights (Pandian et al., 1 Oct 2025).

3. Assumptions and finite-sample guarantees

The theory for Weighted BC is built on three assumptions. First, the imitation loss is bounded: $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$0 Second, there is absolute continuity,

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$1

so the true density ratio is well defined; under the contamination model, the paper notes that

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$2

Third, discriminator accuracy is quantified through

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$3

The function class is

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$4

with Rademacher complexity $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$5. Clipping bias is defined as

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$6

All of these quantities appear explicitly in the generalization bounds (Pandian et al., 1 Oct 2025).

The first main guarantee is a uniform clean-risk approximation result. For

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$7

the paper states that for any $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$8, with probability at least $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$9, uniformly over all $\mathcal{D} = \{\tau_i\}_{i=1}^N,$0,

$\mathcal{D} = \{\tau_i\}_{i=1}^N,$1

The decomposition is explicit: a finite-sample complexity term, a discriminator estimation term, and a clipping-bias term (Pandian et al., 1 Oct 2025).

The second theorem bounds the excess clean risk of the learned policy. Let $\mathcal{D} = \{\tau_i\}_{i=1}^N,$2 be an $\mathcal{D} = \{\tau_i\}_{i=1}^N,$3-approximate minimizer of $\mathcal{D} = \{\tau_i\}_{i=1}^N,$4. Then, with probability at least $\mathcal{D} = \{\tau_i\}_{i=1}^N,$5,

$\mathcal{D} = \{\tau_i\}_{i=1}^N,$6

This is a policy-class-relative guarantee: the learned policy approaches the best clean-risk solution available in $\mathcal{D} = \{\tau_i\}_{i=1}^N,$7 when the weighted empirical problem is well solved, the discriminator is accurate, and clipping is not too aggressive (Pandian et al., 1 Oct 2025).

A distinctive feature of this analysis is the contamination-rate statement. The paper explicitly remarks that if

$\mathcal{D} = \{\tau_i\}_{i=1}^N,$8

then $\mathcal{D} = \{\tau_i\}_{i=1}^N,$9, so the bound in $$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$0 is independent of $$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$1. This does not remove the need for accurate ratio estimation, but it isolates the contamination fraction from the final bound once the clipping interval fully contains the true ratio range (Pandian et al., 1 Oct 2025).

4. Related formulations and category boundaries

Several nearby methods modify BC by nonuniform weighting, but they operate on different objects and do not all estimate density ratios in the same sense.

Method	Weighted object	Relation to density-ratio weighted BC
Weighted BC (Pandian et al., 1 Oct 2025)	$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$2	Direct trajectory-level density-ratio weighting
DWBC (Xu et al., 2022)	$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$3, $$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$4	Discriminator-derived proxy with odds-ratio structure
ADR / DWR (Zhang et al., 2024)	$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$5	State-action conditional-density weighting
CWBC (Nguyen et al., 2022)	$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$6 in return space	Adjacent return-distribution reweighting, not occupancy-ratio weighting
PostBC (Wagenmaker et al., 18 Dec 2025)	Posterior smoothing and policy mixture	Not a density-ratio method

Discriminator-Weighted Behavioral Cloning (DWBC) addresses offline imitation from expert data plus a larger mixed offline dataset. Its final policy loss is exactly a weighted log-likelihood objective,

$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$7

with piecewise weights

$$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$8

Because odds-like terms $$\tau = \{s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_T, a_T, r_T, s_{T+1}\}.$$9 appear, the method has a density-ratio flavor, but the paper does not derive these weights as an expert-to-behavior occupancy ratio; it is best categorized as classifier-probability-derived weighting with odds-ratio structure rather than canonical density-ratio estimation (Xu et al., 2022).

ADR-BC, framed as Adversarial Density Weighted Regression, uses separate expert and sub-optimal conditional action densities $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$0 and $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$1, then optimizes a weighted regression objective

$\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$2

This is a state-action conditional-density contrast rather than a trajectory-ratio method. The manuscript also contains a sign inconsistency between the theorem statement and the definition of the weight $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$3, and the final step replacing $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$4 by an $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$5 regression term is not a standard mathematical equivalence; accordingly, the paper’s own presentation is more heuristic than a clean density-ratio derivation (Zhang et al., 2024).

CWBC is directly relevant to weighted BC in offline RL, but its reweighting is over returns rather than over state-action or trajectory occupancies. In its appendix, the reweighted objective takes the form

$\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$6

and the practical implementation samples trajectories from a return-reweighted distribution that emphasizes high-return bins. The paper explicitly distinguishes this from state-action density-ratio estimation; the closest characterization is importance-like weighting in return space (Nguyen et al., 2022).

Posterior Behavioral Cloning (PostBC) addresses a different pathology: standard BC may fail to ensure demonstrator action coverage during later RL finetuning. Its core mechanism is a mixture

$\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$7

where $\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$8 is the posterior mean demonstrator policy. The method increases support on low-count actions through posterior smoothing, not through importance ratios, occupancy ratios, or weighted empirical risk in the usual sense. A plausible implication is that it is best viewed as complementary to weighted BC rather than a member of the same technical category (Wagenmaker et al., 18 Dec 2025).

5. Empirical evidence

The direct Weighted BC paper evaluates on D4RL continuous-control benchmarks HalfCheetah-Medium, Ant-Medium, Hopper-Medium, and Walker2d-Medium, with contamination ratios

$\pi_\theta : \mathcal{S} \to \Delta(\mathcal{A}).$9

The clean reference set comprises $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$0 of expert trajectories and is strictly disjoint from the training set. Evaluation is performed on clean environments using 50 rollouts per configuration, with results averaged over 5 random seeds and error bars reported as standard error over seeds (Pandian et al., 1 Oct 2025).

The headline empirical findings are stated in contamination-robustness terms. Weighted BC is reported to be consistently robust across all four environments and all four poisoning types; it shows positive relative improvement in $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$1 of scenarios; gains are largest at high contamination, with up to $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$2 improvement over the best baseline; and it maintains over $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$3 performance retention up to $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$4 contamination across poisoning types. Under action poisoning in HalfCheetah at $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$5, Weighted BC achieves above $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$6 return, while Traditional BC is approximately $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$7 and BCQ and BRAC collapse below $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$8. Under state poisoning in Walker2d, Weighted BC maintains roughly $$p(\tau) = (1-\alpha)\,p_{\text{clean}}(\tau) + \alpha\,p_{\text{bad}}(\tau), \qquad \alpha \in [0,1].$$9 return at all contamination levels, while Traditional BC degrades approximately linearly from $p_{\text{clean}}$0 to $p_{\text{clean}}$1. The reported computational overhead on Ant is small: Standard BC uses $p_{\text{clean}}$2 h and $p_{\text{clean}}$3 GB, whereas Weighted BC uses $p_{\text{clean}}$4 h and $p_{\text{clean}}$5 GB, corresponding to $p_{\text{clean}}$6 relative time (Pandian et al., 1 Oct 2025).

DWBC provides complementary evidence for discriminator-derived weighting in mixed-quality offline imitation. Across D4RL MuJoCo and Adroit settings, the paper reports that DWBC outperforms baselines on 27 of 36 tasks, and especially on 21 of 24 tasks in its expert-plus-random and expert-plus-cloned settings. Representative numbers include hopper$p_{\text{clean}}$7: DWBC $p_{\text{clean}}$8 versus BC-exp $p_{\text{clean}}$9, BC-all $p_{\text{bad}}$0, ORIL $p_{\text{bad}}$1; walker2d$p_{\text{bad}}$2: DWBC $p_{\text{bad}}$3 versus BC-exp $p_{\text{bad}}$4, BC-all $p_{\text{bad}}$5; and pen$p_{\text{bad}}$6: DWBC $p_{\text{bad}}$7 versus BC-exp $p_{\text{bad}}$8, BC-all $p_{\text{bad}}$9. The same paper also reports 500k-step training times of 52m for BC, 1h 44m for DWBC, 3h 40m for DemoDICE, 7h 52m for BCND, and 8h 40m for ORIL (Xu et al., 2022).

ADR-BC supplies evidence for state-action density weighting in the few-expert, many-imperfect-data regime. On Gym-Mujoco, it reports a total score of $p(\tau)$00 compared with CEIL at $p(\tau)$01, described as a $p(\tau)$02 improvement. On Adroit and Kitchen, it reports a total of $p(\tau)$03, compared with IQL (oracle) at $p(\tau)$04 and CQL (oracle) at $p(\tau)$05, which the paper highlights as an $p(\tau)$06 improvement over IQL (oracle) and $p(\tau)$07 over CQL (oracle). Its key ablation further reports that the full ADR-BC objective is $p(\tau)$08 times higher than Max-ADE and $p(\tau)$09 times higher than ADE-divergence, supporting the claim that weighting the regression loss itself matters (Zhang et al., 2024).

CWBC contributes a distinct offline RL perspective on weighted conditional BC. On D4RL locomotion, adding CWBC improves RvS by average $p(\tau)$10 and DT by average $p(\tau)$11; reported examples include walker2d-med-replay, where RvS improves from $p(\tau)$12 to $p(\tau)$13, and halfcheetah-med-replay, where RvS improves from $p(\tau)$14 to $p(\tau)$15. The same study shows that weighting alone improves performance under high-return conditioning but does not eliminate catastrophic collapse beyond the dataset maximum, whereas weighting plus conservative regularization improves both performance and OOD reliability (Nguyen et al., 2022).

6. Limitations, misconceptions, and practical scope

The direct Weighted BC formulation has several explicit requirements. It assumes access to a small verified clean reference set drawn from the same clean expert distribution, and the reference set must be disjoint from the main training dataset. It also assumes

$p(\tau)$16

so if some clean expert trajectories have zero support in the contaminated dataset, the ratio is not well defined and the method cannot recover them. Performance depends on discriminator quality through $p(\tau)$17, and clipping can introduce bias through $p(\tau)$18 when the true ratio falls outside $p(\tau)$19. Because the weight is assigned per trajectory rather than per transition, a mostly clean trajectory containing localized corruption is not modeled separately from a uniformly clean or uniformly corrupted trajectory. The paper itself does not report ablations on reference-set size, clipping thresholds, discriminator quality, or trajectory-level versus finer-grained weighting (Pandian et al., 1 Oct 2025).

The experimental record is also not uniformly monotone. The Weighted BC paper notes slight underperformance in a few low-contamination state-poisoning settings, around $p(\tau)$20 worse in some Ant and HalfCheetah cases, and attributes this to the overhead of density-ratio estimation when contamination is minimal. DWBC likewise has failure cases, including ant$p(\tau)$21, where DWBC $p(\tau)$22 is below BC-all $p(\tau)$23 and ORIL $p(\tau)$24, and hammer$p(\tau)$25, where DWBC $p(\tau)$26 is below BC-exp $p(\tau)$27 (Pandian et al., 1 Oct 2025, Xu et al., 2022).

A recurrent misconception is to treat all nonuniform BC schemes as density-ratio weighted BC. The literature summarized here does not support that collapse of categories. CWBC is explicitly framed as return-conditioned trajectory reweighting rather than occupancy-ratio estimation, and PostBC is explicitly framed around posterior uncertainty and coverage rather than importance weighting (Nguyen et al., 2022, Wagenmaker et al., 18 Dec 2025). A plausible implication is that the label “density-ratio weighted behavioral cloning” is most precise when the weighting object is an explicit ratio such as $p(\tau)$28 or an expert-versus-suboptimal density contrast, and less precise when the mechanism is classifier-derived expert-likeness, return-space resampling, or posterior support expansion.

In practical terms, the direct Weighted BC method is most appropriate when learning must remain offline, the dataset may contain corrupted, poisoned, or low-quality trajectories, a small trusted clean reference set is available, and a simpler alternative to full offline RL is preferred. Within that scope, its defining contribution is conceptually narrow but technically clear: use a clean-vs-mixed discriminator to estimate a trajectory-level density ratio, clip and freeze the ratio, and train the policy by weighted behavioral cloning on the contaminated dataset (Pandian et al., 1 Oct 2025).