Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation

Abstract: This work investigates multi-objective imitation learning: the problem of recovering policies that lie on the Pareto front given demonstrations from multiple Pareto-optimal experts in a Multi-Objective Markov Decision Process (MOMDP). Standard imitation approaches are ill-equipped for this regime, as naively aggregating conflicting expert trajectories can result in dominated policies. To address this, we introduce Multi-Output Augmented Behavioral Cloning (MA-BC), an algorithm that systematically partitions divergent expert data while pooling state-action pairs where no behavior conflict is observed. Theoretically, we prove that MA-BC converges to Pareto-optimal policies at a faster statistical rate than any learner that considers each expert dataset independently. Furthermore, we establish a novel lower bound for multi-objective imitation learning, demonstrating that MA-BC is minimax optimal. Finally, we empirically validate our algorithm across diverse discrete environments and, guided by our theoretical insights, extend and evaluate MA-BC on a continuous Linear Quadratic Regulator (LQR) control task.

• The paper presents MA-BC, a novel algorithm that partitions expert data to efficiently recover ε-approximate Pareto optimal policies.
• It demonstrates that naive data pooling and isolated behavioral cloning fail due to sample inefficiency and geometric errors, emphasizing the benefit of structured partitioning.
• Empirical and theoretical evaluations confirm that MA-BC improves sample complexity by focusing on divergent states while leveraging overlap in common states.

Provably Efficient Multi-Objective Imitation via Data Partitioning

Background and Motivation

Multi-objective sequential decision-making arises in practical imitation learning scenarios, where demonstration datasets are curated from multiple experts with divergent implicit utility functions. This includes fine-tuning LLMs with diverse annotators or robotic learning from heterogeneous operators, each optimizing for distinct, sometimes conflicting objectives. Such data yields highly multimodal and potentially conflicting demonstrations, challenging conventional imitation learning (IL) that either pools all data together—risking dominated, suboptimal policies far from the Pareto frontier—or isolates each expert dataset—resulting in severe sample inefficiency by forgoing structural overlap among expert behaviors.

Problem Formulation: MOMDPs and Pareto Front

The paper formalizes the multi-objective imitation learning problem in the Multi-Objective Markov Decision Process (MOMDP) framework, where rewards are vector-valued ($r(x, a) \in \mathbb{R}^d$). The solution concept is to recover $\varepsilon$-approximate Pareto front (PF) policies, namely, policies for which there exists no other policy yielding uniformly higher vector returns. Crucially, the space of expected returns forms a convex polytope, with the Pareto front lying on boundary faces characterized by outward normal vectors in the non-negative orthant.

Figure 2: The DST Return Polytope—the Pareto front (green vertices) bounds the set of attainable returns; aliased policies (purple circles) expose structural redundancy at the vertex level.

An important structural result—the Pareto Path Guarantee—proves that any pair of neighboring PF-policies can be connected by a sequence of deterministic PF-policies, each differing in the action at only a single state. This guarantees a dense, lattice-like structure on the Pareto front, precluding pathological cases where optimal policies are mutually disjoint and suggesting substantial overlap in state-action space among PF experts.

Failure Modes of Standard Imitation Learning Approaches

The work carefully delineates two dominant failure modes in current approaches:

1. Failure I (Isolated BC): Treating each expert dataset independently as a single-expert IL problem requires each dataset size to scale linearly with the state space cardinality. This is fundamentally sample-inefficient and precludes leveraging state-action overlap among experts defined by the Pareto Path structural property.
2. Failure II (Naive Data Pooling): Aggregating all demonstrations (from multiple experts) into a single dataset and performing behavioral cloning can severely degrade performance. The learned policy can end up dominated in all objectives, falling off the Pareto front regardless of dataset size. This is a geometric failure—mixing deterministic, conflicting strategies in state-action space produces policies whose expected returns collapse into the non-optimal interior of the return polytope.

Figure 4: The geometric failure of Naive BC—aggregated imitation policies sag into the suboptimal interior (green dot), missing the Pareto boundary entirely.

Multi-Output Augmented Behavioral Cloning (MA-BC)

The authors introduce Multi-Output Augmented Behavioral Cloning (MA-BC), a principled algorithm leveraging the Pareto Path structural insight. Its core mechanism partitions the dataset based on observed behavioral divergence among experts:

• Divergent Set: States where at least two experts select distinct actions. These states are isolated for expert-specific policy training.
• Common Set: States lacking observed divergence; in these, all expert data is pooled, irrespective of origin.

Each output policy for expert $\ell$ is trained on the union of (i) the pooled common set, and (ii) the expert-specific data at divergent states. This allows maximal data sharing compatible with Pareto faithfulness, directly transferring support among policies where behaviors are empirically identical while respecting divergence.

Theoretical Guarantees

The paper provides a finite-sample, high-probability rate for the suboptimality of each recovered policy:

• The $\varepsilon$-optimality of each policy depends on two rates:
  • $1/N$ scaling on the cardinality of shared states in the pooled dataset.
  • $1/N_\ell$ scaling on the cardinality of divergent states specific to expert $\ell$.
• The sample complexity for each expert scales only with the number of "divergent" states, $K$, which is typically much smaller than the full state space $|\mathcal{X}|$.
• The dependency on a "concentrability" constant $C_*$ (ratio between expert and pooled occupancy distributions in common states) is shown to be minimax-optimal and unavoidable.

These guarantees demonstrate that MA-BC outperforms both naive pooling and isolated BC approaches, and achieves the best-known statistical rate for multi-objective imitation.

Scaling MA-BC to Continuous State-Action Domains

The authors further provide a heuristic extension of MA-BC to continuous domains by pooling across nearby state-action pairs—based on spatial proximity and Lipschitz continuity—with a tunable tolerance $\varepsilon$0. This induces a tradeoff between variance-reduction (through more aggressive pooling) and bias-introduction (via mismatched targets in divergent regions).

Figure 6: Drone LQR task—(Left) pooling threshold $\varepsilon$1 boosts efficiency at low data; (Middle, Right) the bias-data tradeoff shifts as $\varepsilon$2 increases, showing optimal pooling shrinks with more data.

Empirically, this tradeoff is demonstrated on a 6-DOF LQR quadcopter domain: in extremely low-data regimes, larger pooling ($\varepsilon$3) yields faster convergence at the cost of bias; as the dataset grows, the optimal $\varepsilon$4 narrows, emphasizing unbiasedness.

Empirical Validation and Observations

Across classic MOMDP benchmarks (Deep Sea Treasure, Resource Gathering, Slippery Y-Maze) and the LQR drone domain:

• Naive BC fails categorically to recover Pareto-optimal policies in all regimes, with suboptimality gaps plateauing even as $\varepsilon$5 increases—in full agreement with the geometric analysis of Failure II.
• Isolated BC converges to the Pareto front with sufficient data per expert, but requires sample sizes linear in $\varepsilon$6 per expert, showing extreme inefficiency (Failure I).
• MA-BC matches or exceeds the efficiency of both, requiring only sufficient coverage in divergent states and exploiting state-action overlap for rapid convergence. The sample efficiency improvement is especially pronounced as the overlap among experts' occupancy distributions (i.e., smaller $\varepsilon$7) increases.

Figure 5: Deep Sea Treasure domain—a sequential navigation with spatially distinct reward sources and divergent policies for time vs. treasure maximization.

Notably, analysis confirms the practical self-correction mechanism in MA-BC: as sample sizes grow, the initial bias (from pooled but soon-to-be-divergent states) is progressively purged as the algorithm detects previously unseen expert disagreement. This property is essential for consistency in the continuous-domain extension, where perfect state trajectory overlap is effectively unattainable.

Implications, Relations, and Future Work

MA-BC provides a concrete, theoretically optimal strategy for imitation in MOMDPs where multiple experts each operate on the Pareto front, resolving fundamental statistical bottlenecks in multi-objective IL regimes. Practically, this advances LLM fine-tuning and robotic IL from multimodal sources, preventing both severe sample overhead and the risk of deploying policies that are strictly dominated on all objectives.

Theoretically, the Pareto Path structural result opens avenues for traversal algorithms in multi-objective RL, enabling the systematic construction and exploration of connected PF policies—potentially extending beyond the settings guaranteed in prior works predicated on unique policy-return correspondences.

Further, the practical extension to continuous domains—though currently heuristic—identifies critical directions for deriving provable convergence guarantees under smoothness or Lipschitz regularity (see [jadbabaie2024multi]).

A connection to recent results in multi-agent imitation learning [freihaut2025learningequilibriadataprovably, viano2026multi] is also evident: whereas those analyze equilibria in Markov games with coupled rewards, the MOMDP setting here offers independence across vector objectives, allowing full exploitation of per-objective overlap absent in Nash arrangements.

Conclusion

This work rigorously demonstrates the necessity and optimality of structured data partitioning for imitation learning in multi-objective settings with Pareto-front experts. The Multi-Output Augmented Behavioral Cloning framework achieves provably efficient sample complexity by leveraging structural overlap among expert policies, with theoretical and empirical support across discrete and continuous domains. The results critically inform the design of robust, sample-efficient IL for emerging applications in RLHF, robotics, and beyond, and motivate further exploration into representation learning architectures for implicit, structural pooling and specialization.