---
title: 'MA-BC: Multi-Output Augmented Behavioral Cloning'
url: https://www.emergentmind.com/topics/multi-output-augmented-behavioral-cloning-ma-bc
type: topic
---

# MA-BC: Multi-Output Augmented Behavioral Cloning

Searching arXiv for the primary MA-BC paper and closely related behavioral-cloning augmentation work.

Multi-Output Augmented Behavioral Cloning (MA-BC) is an offline imitation-learning algorithm for demonstrations generated by multiple Pareto-optimal experts in a Multi-Objective Markov Decision Process (MOMDP). It is designed for the regime in which different experts optimize different tradeoffs among a shared vector-valued reward, so naive aggregation of demonstrations can produce a dominated policy, while isolated per-expert cloning wastes shared information. MA-BC addresses this by learning multiple output policies, splitting expert data where cross-expert behavioral conflict is observed and pooling state-action pairs where no such conflict appears. In the tabular discounted setting, the method is proved to converge to Pareto-optimal policies at a faster statistical rate than learners that treat each expert independently, and the paper establishes a matching lower bound showing minimax optimality. A heuristic continuous-state/action extension is also proposed and evaluated on a continuous LQR control problem [2605.12000].

## 1. Multi-objective imitation setting

MA-BC is formulated in a MOMDP
\[
\mathcal{M}=(X,\mathcal{A},P,\gamma,\rho,r),
\]
where \(X\) is the state space, \(\mathcal A\) the action space, \(P\) the transition kernel, \(\gamma \in [0,1)\) the discount factor, \(\rho\) the initial-state distribution, and the reward is vector-valued,
\[
r:X\times \mathcal A \to [0,1]^d.
\]
A stationary Markov policy \(\pi:X\to \Delta_{\mathcal A}\) has vector return
\[
J(\pi)=\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^t r(X_t,A_t)\right]\in\mathbb{R}^d,
\]
with occupancy measure
\[
\mu^\pi(x,a)=(1-\gamma)\,\mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty}\gamma^t \mathbf{1}\{X_t=x,A_t=a\}\right],
\qquad
\nu^\pi(x)=\sum_{a\in\mathcal A}\mu^\pi(x,a).
\]

Because the reward is vector-valued, the solution concept is Pareto optimality rather than scalar optimality. The paper defines an \(\varepsilon\)-approximate Pareto-front policy (\(\varepsilon\)-PF-policy) as a randomized output policy that is not dominated, up to \(\varepsilon\), by any other policy in every objective while being strictly worse in at least one coordinate. When \(\varepsilon=0\) and the policy is deterministic, this reduces to an ordinary Pareto-front policy.

The demonstration data are split by expert:
\[
\{\mathcal D_\ell\}_{\ell=1}^L,
\qquad
\mathcal D_\ell=\{(X_n^\ell,A_n^\ell)\}_{n=1}^{N_\ell},
\]
with each pair sampled from expert \(\pi_\ell\), and each expert assumed to be Pareto-optimal. The learning objective is to output one or more policies \(\{\hat\pi_\ell\}_{\ell=1}^{L'}\) that lie on, or approximate, the Pareto front. The main exposition usually sets \(L'=L\), but the number of outputs need not equal the number of experts [2605.12000].

## 2. Split-and-pool construction

The central principle of MA-BC is summarized by the phrase “split the differences, pool the rest.” If experts disagree at a state, that disagreement is treated as evidence of a genuine preference tradeoff and is preserved. If they do not disagree, the corresponding data are shared across outputs.

The paper motivates this with a structural theorem: neighboring Pareto-front deterministic policies can be connected by a path of Pareto-front policies that differ in one state at a time. This “Pareto path” result suggests that Pareto-optimal experts in a common MOMDP often agree on many states and differ only on a relatively small subset. A plausible implication is that sample sharing should be concentrated on those large agreement regions.

In the idealized tabular analysis, the true common-state set is
\[
X_{\mathrm{common}} := \{x\in X:\pi_1(x)=\pi_2(x)=\cdots=\pi_L(x)\}.
\]
Operationally, MA-BC identifies empirical divergence by scanning the expert-indexed datasets for states that appear with different observed actions across experts. It then defines a shared pool \(\mathcal D_{\mathrm{common}}\) by removing all occurrences of empirically diverging states from the union of expert datasets.

Each output policy is then estimated by behavioral cloning on an augmented dataset containing the expert’s own full data plus the pooled common data:
\[
\hat{\pi}_\ell = \arg\max_{\pi\in\Pi} \sum_{(X,A)\in \mathcal D_{\mathrm{common}}\cup \mathcal D_\ell} \log \pi(A\mid X).
\]
This is the sense in which the method is “multi-output”: it does not train one compromise policy, but a family of output policies, typically one per expert. Each output remains specialized on that expert’s conflicting regions while benefiting from the pooled signal on non-conflicting regions.

The paper positions MA-BC between two extremes. Ordinary pooled behavioral cloning merges all data into one output and can destroy Pareto optimality. Independent cloning keeps one output per expert but forgoes cross-expert sample sharing. MA-BC preserves the multi-output structure while sharing what is safely shareable [2605.12000].

## 3. Statistical guarantees and impossibility results

A central negative result in the paper is that naive single-output pooling can fail even asymptotically. If one learns
\[
\hat\pi = \arg\max_{\pi\in\Pi}\sum_{(X,A)\in \cup_{\ell=1}^L \mathcal D_\ell}\log \pi(A\mid X),
\]
then there exists an MOMDP such that there is a policy \(\pi^\star\) with
\[
\forall i\in[d],\qquad J_i(\pi^\star)\ge J_i(\hat\pi)+\frac{1}{3(1-\gamma)}.
\]
Thus pooled single-output behavioral cloning can converge to a dominated policy.

The positive result for MA-BC decomposes error into shared-state and divergent-state terms. To state that decomposition, the paper defines the pooled data occupancy
\[
\nu^{\pi_{\mathrm{data}}}(x)=\frac{\sum_{\ell=1}^L N_\ell \nu^{\pi_\ell}(x)}{N},
\qquad
N=\sum_{\ell=1}^L N_\ell,
\]
and a concentrability factor
\[
C_{\pi_{\mathrm{data}}}
:=
\max_{\ell\in[L]}
\left\|
\frac{\nu^{\pi_\ell}}{\nu^{\pi_{\mathrm{data}}}}
\right\|_\infty.
\]
The main theorem states that each output \(\hat\pi_\ell\) is \(\varepsilon_\ell\)-PF, with one term scaling with the total sample size \(N\) over common states and another term scaling with the expert-specific sample size \(N_\ell\) over divergent states. In prose, the shared-states term depends on \(|X_{\mathrm{common}}|\), \(N\), and the occupancy mismatch between expert \(\ell\) and the pooled data, while the divergent-states term depends on \(|X\setminus X_{\mathrm{common}}|\) and \(N_\ell\). This formalizes exactly why MA-BC pools only non-conflicting data.

A corollary makes the efficiency gain explicit. If
\[
|X\setminus X_{\mathrm{common}}|=K
\quad\text{and}\quad
|X|\ge 2K,
\]
then MA-BC outputs an \(\varepsilon\)-PF set provided
\[
N = \mathcal O\!\left(\frac{C_{\pi_{\mathrm{data}}}\,|X|}{(1-\gamma)^2\varepsilon}\right),
\qquad
N_\ell = \mathcal O\!\left(\frac{K}{(1-\gamma)^2\varepsilon}\right).
\]
Independent per-expert imitation would require each \(N_\ell\) to scale with \(|X|\); MA-BC replaces that with scaling in the number \(K\) of genuinely divergent states.

The lower bound mirrors the upper bound. For any offline multi-output imitation algorithm returning \(\varepsilon_\ell\)-PF outputs, the paper constructs a two-objective MDP in which every learner must pay both a common-state estimation term and a divergent-state expert-specific term. Up to constants and the precise \((1-\gamma)\)-dependence in the divergent component, this matches the MA-BC upper bound. The result is the basis for the minimax-optimality claim [2605.12000].

## 4. Continuous-state and continuous-action extension

The main theory is tabular. For continuous state and action spaces, the paper proposes a heuristic extension because exact state collisions are negligible and conflict detection must be approximate.

For expert \(\ell\), the augmented shared set is
\[
\mathcal D_\ell^+ =
\left\{
(X',A')\in \bigcup_{\ell'\neq \ell}\mathcal D_{\ell'}
~:~
\exists (X,A)\in \mathcal D_\ell
\text{ such that }
\|X-X'\|\le \delta
\ \text{and}\
\|A-A'\|\le L^{\pi_\ell}\|X-X'\|
\right\},
\]
where \(L^{\pi_\ell}\) is the Lipschitz constant of expert \(\pi_\ell\). Another expert’s sample is pooled only if it is compatible with what expert \(\ell\) could plausibly do in a nearby state under that Lipschitz criterion.

Learning then becomes squared-loss regression:
\[
\hat{\pi}_\ell \in \arg\min_{\pi:X\to\mathcal A} \sum_{(X,A)\in \mathcal D_\ell \cup \mathcal D_\ell^+} \|A-\pi(X)\|^2.
\]
The paper is explicit that this continuous MA-BC is a heuristic extension without formal guarantees.

The new radius parameter \(\delta\) induces a bias–data tradeoff: larger \(\delta\) increases the amount of pooled data but also increases target bias. This suggests that continuous MA-BC is governed by approximate compatibility rather than exact conflict separation, and that the discrete split/pool principle is preserved only approximately in continuous domains [2605.12000].

## 5. Empirical evidence

The empirical study evaluates MA-BC on three discrete MOMDP benchmarks: Deep Sea Treasure, Slippery Y-Maze, and Resource Gathering. Demonstrations come from two extreme Pareto experts in each environment. Performance is measured by the normalized \(L_\infty\) distance to the continuous Pareto frontier,
\[
\min_{J^*\in PF}\|J(\hat\pi)-J^*\|_\infty,
\]
with coordinate-wise normalization so that different reward scales are comparable.

Across these environments, the reported pattern is consistent: naive pooled BC fails to converge to the Pareto front, isolated BC converges but more slowly, and MA-BC converges faster and reaches the Pareto front reliably. In the Deep Sea Treasure appendix analysis, naive pooled BC produces stochasticity diffused across multiple conflicting states, causing the return to “sag” into the interior of the return polytope, whereas Pareto-optimal stochastic mixtures randomize only in specific places. MA-BC avoids that destructive averaging because conflicting states are isolated rather than pooled.

The paper also studies the predicted effect of concentrability. In a lower-bound-inspired MDP, \(C_{\pi_{\mathrm{data}}}\) is controlled directly. In Deep Sea Treasure and Resource Gathering, overlap is varied by changing the stochasticity of the initial state distribution. The empirical finding is that increasing overlap improves concentrability and widens the performance gap between MA-BC and isolated BC, which matches the theory.

The continuous experiment considers a linearized 6-DOF quadcopter control task with an LQR formulation. Two experts are generated from different LQR \(Q\)-matrices: an Agile expert emphasizing fast tracking and an Economic expert emphasizing energy conservation. The continuous MA-BC variant is evaluated by pooling nearby compatible state-action pairs under the \(\delta\)-criterion. In the very low-data regime, larger \(\delta\) helps because it aggressively increases pooled samples; with more data, smaller \(\delta\) becomes preferable because less pooling bias is needed. The experiment also reports both the amount of pooled data and the empirical action bias
\[
\|A_{\text{pooled}}-A_{\text{true}}\|_2,
\]
exhibiting the expected bias–variance tradeoff [2605.12000].

## 6. Relation to adjacent behavioral-cloning methods and scope

MA-BC belongs to a broader family of behavioral-cloning methods that augment plain action imitation with additional structure, but its defining structure is specific to multi-objective imitation. The paper explicitly distinguishes it from ordinary pooled BC, isolated per-expert BC, and broader multi-task or multi-agent imitation: the experts are assumed to be Pareto-front policies in a common MOMDP, and the algorithm’s key operation is a statewise partition of demonstrations into conflicting and non-conflicting subsets [2605.12000].

This distinguishes MA-BC from procedure-oriented multi-output imitation. Procedure cloning learns not only an action but also intermediate procedure observations \(x_0,\ldots,x_L\), using factorized objectives such as
\[
p(a, x\mid s) = p(a\mid x,s)\cdot \prod_{\ell=1}^{L} p(x_\ell\mid \mathbf{x}_{<\ell}, s)\cdot p(x_0\mid s),
\]
so that extra outputs are expert computational traces rather than Pareto-specialized policies [2205.10816]. It also differs from diffusion-augmented BC, where a policy is trained by
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{BC}} + \lambda \mathcal{L}_{\text{DM}},
\]
and the augmentation comes from a diffusion model of the joint state-action distribution rather than from statewise split/pool structure across multiple experts [2302.13335].

Inverse-dynamics augmentation provides another contrast. Concurrently trained BCO\(^*\) improves a policy estimate and an inverse dynamics model in tandem, using pseudo-labels for state-only demonstrations; its augmentation is semi-supervised and modular, not Pareto-structured [2008.01205]. ABCO similarly uses inverse dynamics, self-attention, and success-conditioned sampling to stabilize imitation from observation, again in a setting where the core issue is missing action labels rather than conflicting Pareto behaviors [2004.13529]. EBGAN-MDN addresses one-to-many action mappings by learning a conditional mixture distribution over actions with an energy model, which is directly relevant to multimodal BC but tackles mode averaging and mode collapse rather than multi-objective Pareto recovery [2510.07562].

Several limitations follow directly from the MA-BC formulation. The main guarantees are tabular; the continuous extension is heuristic and currently lacks guarantees. The main algorithm assumes expert identities are known. Conflict detection is observational, so finite data may fail to expose true disagreements immediately. Finally, dependence on concentrability is unavoidable: if experts’ occupancies differ too much even on common states, pooling offers little benefit. A common misconception is therefore to treat MA-BC as merely “multi-head BC” or as a generic multi-output architecture. Its defining ingredient is not the presence of several prediction channels per se, but the selective sharing of demonstration data according to observed cross-expert compatibility.

Source: https://www.emergentmind.com/topics/multi-output-augmented-behavioral-cloning-ma-bc