Flow Matching Adversarial Imitation Learning (FAIL)

• The paper introduces FAIL, a framework that recasts post-training of flow-matching generators as an imitation learning problem to match a high-quality expert distribution.
• It presents two algorithmic variants—FAIL-PD for differentiable, continuous settings and FAIL-PG for black-box, discrete applications—with distinct gradient and reward mechanisms.
• FAIL mitigates distribution shift and reward hacking by replacing explicit rewards with a dynamic adversarial discriminator signal that fine-tunes model performance.

Searching arXiv for FAIL and closely related flow-based imitation learning papers. Flow Matching Adversarial Imitation Learning (FAIL) is a post-training framework for flow-matching generative models that casts alignment to a smaller, higher-quality expert distribution as an imitation-learning problem. In this formulation, the generative model is the policy, expert samples are demonstrations conditioned on the same prompts, and training minimizes policy–expert divergence through adversarial training without explicit rewards or pairwise comparisons. The framework introduces two optimization variants, FAIL-PD and FAIL-PG, and is reported to generalize from continuous image generation to discrete image and video generation while also functioning as a regularizer to mitigate reward hacking in reward-based optimization (Ma et al., 12 Feb 2026). A persistent source of ambiguity is that the acronym FAIL was previously used for Forward Adversarial Imitation Learning in imitation learning from observations alone (Sun et al., 2019).

1. Terminology and historical usage

The acronym FAIL has two distinct uses in the literature. In the 2019 paper on imitation learning from observations alone, FAIL denotes Forward Adversarial Imitation Learning, a model-free algorithm for ILFO in large-scale episodic finite-horizon MDPs. That method learns a sequence of time-dependent policies by minimizing an Integral Probability Metric between expert and learner observation distributions, step by step, and is described as the first provably efficient algorithm in the ILFO setting (Sun et al., 2019).

The 2026 usage instead denotes Flow Matching Adversarial Imitation Learning. Here, “flow matching” refers to post-training of pretrained flow-matching generators rather than the earlier ILFO setting. The underlying conceptual continuity is distribution matching: both uses of FAIL are organized around matching learner-induced and expert-induced distributions, but the objects being matched, the optimization machinery, and the application domains differ substantially. In the later usage, the target domain is generative modeling—initially image generation, then extended to discrete image and video generation—rather than control from observation-only expert trajectories (Ma et al., 12 Feb 2026).

This terminological split matters because “flow” in Flow Matching Adversarial Imitation Learning does not denote the same object as in normalizing-flow occupancy modeling or in forward-training ILFO. The 2026 FAIL paper is specifically about post-training flow-matching generators through an adversarial imitation-learning lens, not about the 2019 provably efficient ILFO algorithm (Sun et al., 2019).

2. Imitation-learning formulation of post-training

FAIL is motivated by the claim that post-training of flow-matching models is mathematically equivalent to imitation learning. After pretraining, the objective is no longer to learn generic visual structure, but to make the model’s output distribution match a smaller, higher-quality expert distribution of preferred samples. Under this view, supervised fine-tuning is the analogue of behavioral cloning: it mimics expert demonstrations directly, but cannot correct policy drift in unseen states. Preference-optimization approaches such as RLHF or DPO address distribution shift, but require costly preference pairs or reward modeling. FAIL replaces those ingredients with adversarial imitation learning (Ma et al., 12 Feb 2026).

The core minimax objective is

$$\min_\theta \max_\omega \; \mathbb{E}_{x \sim p_E}[\log D_\omega(x)] + \mathbb{E}_{x \sim \pi_\theta}[\log(1 - D_\omega(x))].$$

Here $p_E(x)$ is the expert distribution and $\pi_\theta(x)$ is the current flow model’s distribution. The discriminator is trained to assign high scores to expert samples and low scores to policy samples, while the policy is trained to generate samples indistinguishable from the expert. The paper explicitly describes this as the same adversarial structure as GAIL, adapted to flow-matching generative models, with the discriminator supplying the learned training signal in place of an explicit reward model (Ma et al., 12 Feb 2026).

A central motivation is the distribution-shift problem. Supervised fine-tuning works when training-time and test-time distributions are close, but once the model drifts away from expert-like states during generation it no longer receives the right corrective signal. FAIL addresses that failure mode through adversarial distribution matching rather than one-shot regression onto demonstrations. This places the framework between pure supervised imitation and explicit reward optimization: it remains demonstration-driven, but the learning signal is dynamic and policy-dependent (Ma et al., 12 Feb 2026).

3. Algorithmic variants: FAIL-PD and FAIL-PG

The framework defines two optimization algorithms, chosen according to whether the generator is differentiable end-to-end or must be treated as a black box (Ma et al., 12 Feb 2026).

Variant	Intended setting	Core mechanism
FAIL-PD	White-box, continuous flow matching	Pathwise gradients through a differentiable ODE solver
FAIL-PG	Black-box, discrete or computationally constrained settings	Policy-gradient updates from discriminator-derived rewards

FAIL-PD is the white-box formulation. It exploits the fact that a flow-matching generator is a continuous-time dynamical system solved by a differentiable ODE solver. Generation starts from noise $\epsilon \sim \mathcal{N}(0,I)$ and integrates a learned vector field $v_\theta$ to produce a sample $x_0$. Because the solver is differentiable, the discriminator score on the final sample can be backpropagated through the generation trajectory to update $\theta$. The paper explicitly connects this mechanism to DPG/DDPG: the ODE solver plays the role of a known deterministic environment, and the discriminator acts as the reward (Ma et al., 12 Feb 2026).

For computational practicality, FAIL-PD may replace full ODE unrolling with a single-step denoising approximation. Sampling $t \sim \mathcal{U}(0,1)$, one forms $x_t = (1-t)x_0 + t\epsilon$ and then estimates a local clean sample by

$$x_0' = \frac{x_t + \Delta t \cdot v_\theta(x_t, t) - (t + \Delta t)\epsilon}{1 - (t + \Delta t)}.$$

The discriminator is then applied to $p_E(x)$0, and the policy is updated by backpropagating through that estimate. The stated purpose is to obtain a dense gradient signal indicating how the current flow field should move to become more expert-like (Ma et al., 12 Feb 2026).

FAIL-PG is the black-box alternative. It is intended for settings where differentiating through the generator or discriminator is too expensive or impossible, especially discrete image generation or autoregressive token generation. Instead of using pathwise derivatives, FAIL-PG treats the discriminator output as a scalar reward,

$p_E(x)$1

The policy update is built on Flow Policy Optimization, with a PPO-like clipped surrogate. The ratio is approximated from Conditional Flow Matching losses,

$p_E(x)$2

and the surrogate objective is

$p_E(x)$3

The advantage is normalized within a group of sampled outputs using a GRPO-style group-relative normalization,

$p_E(x)$4

A KL penalty to a reference policy $p_E(x)$5, in practice the pretrained model, is added to prevent excessive drift:

$p_E(x)$6

The paper emphasizes a practical tradeoff: FAIL-PG often converges faster but can be less stable over long training horizons, whereas FAIL-PD is slower initially but more stable and better at preserving the generator’s manifold structure (Ma et al., 12 Feb 2026).

4. Empirical results and application scope

The main image-generation experiment fine-tunes FLUX.1-dev using only 13,000 demonstrations from Gemini 3 Pro / Nano Banana Pro. The dataset is intentionally restrictive: for each prompt, only a single expert image is kept. Prompts are drawn from a mixture of HPDv3 and UniGen-Bench training splits. The main discriminator is a VLM-based Qwen3-VL-2B-Instruct model, with additional experiments using DINOv3 and the FLUX backbone itself as discriminator variants. Training uses AdamW, a global batch size of 128, and 32 NVIDIA H20 GPUs; the main setup trains at $p_E(x)$7 resolution for one epoch with discriminator warmup (Ma et al., 12 Feb 2026).

On UniGen-Bench, FAIL-PD improves FLUX.1-dev from 61.61 to 73.70 overall. On DPG-Bench, FAIL-PD reaches 87.32. On HPDv3, FAIL-PD achieves 11.28. FAIL-PG is reported as competitive, sometimes achieving faster gains and strong benchmark scores, though not always matching the long-horizon stability of FAIL-PD (Ma et al., 12 Feb 2026).

The paper extends the framework beyond continuous image generation. For discrete image generation, FAIL-PG is applied to Xomni, precisely because discrete token generation is difficult to differentiate through. For video generation, the method fine-tunes Wan2.1-1.3B using expert videos from Wan2.2-A14B and prompts from VidProM. Both FAIL-PD and FAIL-PG improve the model, especially on the semantic dimension of VBench. The paper presents this as evidence that the framework is modality-agnostic so long as expert and policy samples can be defined and a discriminator can be trained (Ma et al., 12 Feb 2026).

The empirical section also uses ablations to sharpen interpretation. Compared with RLHF baselines, FAIL is described as much more robust when reward hacking is present: RLHF can increase the reward-model score while making true benchmark performance worse. Compared with Online DPO, FAIL-PG tends to do better because it replaces binary chosen/rejected simplification with a continuous discriminator score. In discriminator ablations, FAIL-PD is more sensitive to discriminator architecture than FAIL-PG, and the best overall results come from the FM backbone discriminator, likely because it shares latent-space knowledge with the generator. Training dynamics further distinguish the two variants: FAIL-PG improves very fast but can collapse after longer training, while FAIL-PD improves more gradually yet remains stable and keeps improving over long horizons (Ma et al., 12 Feb 2026).

5. Relation to adjacent flow-based imitation-learning methods

FAIL belongs to a broader family of methods that combine adversarial or density-ratio learning with flow-based generative structure, but several neighboring methods use different flow formalisms and different control loops.

A close predecessor in spirit is Coupled Flow Imitation Learning (CFIL), which studies standard reinforcement learning and imitation learning rather than generative-model post-training. CFIL models state or state-action distributions with a pair of coupled normalizing flows, linked through the Donsker–Varadhan representation of KL divergence, and uses the critic-like quantity

$p_E(x)$8

to define the imitation reward $p_E(x)$9. Its central claim is that jointly coupling the flows through the DV objective avoids the out-of-distribution instability of independently trained density models. The paper explicitly notes that CFIL is not FAIL itself, but is a closely related predecessor or alternative formulation in the same design space of explicit density estimation, flow-based ratio modeling, and adversarial policy optimization (Freund et al., 2023).

A later video-generation method, Adversarial Flow Distillation (AFD), is presented as instantiating the broader FAIL idea for black-box heterogeneous autoregressive video distillation. AFD trains a prompt-conditioned Bradley–Terry discriminator on teacher–student video pairs, converts the resulting signal into an advantage, and then propagates that advantage through the student’s forward noising process to obtain dense flow-matching supervision on noised states. The paper summarizes the core move as learning an adversarial advantage on samples and then converting it into dense flow-matching supervision on the student’s own forward noising trajectory (Luo et al., 25 May 2026).

In embodied control, FM-IRL and FA-OPD adapt the general pattern to online reinforcement learning with demonstrations. FM-IRL uses a teacher flow-matching model to define both an FM-enhanced discriminator-based reward and a policy regularizer for a separate MLP student, thereby avoiding direct policy-gradient optimization through the flow model while still leveraging the teacher’s distributional structure (Wan et al., 10 Oct 2025). FA-OPD describes itself as an adversarial dual on-policy distillation method in which a flow-matching teacher is learned from demonstrations and co-trained with a lightweight student. Its two supervision channels are a reward channel over state-action pairs and an action channel that provides dense local targets on student-visited states (Wan et al., 26 May 2026).

These relations indicate that FAIL is best understood not as an isolated algorithmic motif but as one branch of a larger flow-based imitation-learning landscape. The branches differ in whether the “flow” is a normalizing flow over occupancy-like distributions, a flow-matching generator over images or videos, or a training-time teacher for online control; they also differ in whether the learned signal is used as a discriminator score, a reward, an advantage, a regularizer, or a dense action target.

6. Limitations, boundary conditions, and common misconceptions

FAIL inherits the usual liabilities of adversarial training. The paper explicitly notes instability and sensitivity to hyperparameters and architecture choices. It was validated only at a modest 13K demonstration scale, so scaling behavior remains open. The method is also framed as a refinement and alignment mechanism rather than a capability-creation mechanism: it cannot create abilities that the base model never had. The paper states that text rendering improves, but that a significant gap remains relative to models whose pretraining already contains stronger OCR or spelling capability (Ma et al., 12 Feb 2026).

A common misconception is to equate FAIL with supervised fine-tuning. The framework is motivated precisely by the claim that SFT behaves like behavioral cloning: it can mimic expert demonstrations effectively but cannot correct policy drift in unseen states. Another misconception is to treat FAIL as merely reward optimization with a different reward model. The paper instead emphasizes adversarial imitation learning, in which the discriminator is dynamic and distribution-matching, and this dynamism is part of why the framework is presented as a robust regularizer against reward hacking when combined with reward-based optimization (Ma et al., 12 Feb 2026).

A second source of confusion concerns the word “flow.” In FAIL, the object being post-trained is a flow-matching generator. In CFIL, by contrast, the relevant machinery is a pair of normalizing flows used to estimate a Donsker–Varadhan critic and density ratio for state-action distribution matching in imitation learning. The conceptual overlap is strong, but the mathematical objects and training pipelines are different (Freund et al., 2023).

Finally, the acronym itself is overloaded. In 2019, FAIL meant Forward Adversarial Imitation Learning for ILFO, with a provable sample-complexity guarantee independent of the number of unique observations under realizability and finite-capacity assumptions (Sun et al., 2019). In 2026, FAIL denotes Flow Matching Adversarial Imitation Learning for post-training generative models (Ma et al., 12 Feb 2026). Any technical discussion of FAIL therefore requires explicit disambiguation of the intended expansion.