Multimodal Flow-Matching Generative Models (FMIP)

• Multimodal Flow-Matching Generative Models (FMIP) are generative models that blend continuous and discrete data modalities using optimal transport and flow-matching to achieve structured generation.
• They fuse optimal transport-inspired path coupling, neural vector/rate field learning, and explicit guidance to enable scalable, efficient inference across diverse domains.
• FMIP architectures, including graph-based and U-Net models, facilitate one-shot or few-step multimodal sampling in applications ranging from MILP to image fusion and molecular design.

Multimodal Flow-Matching Generative Models (FMIP) are a class of generative models that generalize the flow-matching paradigm to settings involving both continuous and discrete data modalities, supporting direct probabilistic transport in mixed or structured domains. FMIP methods blend optimal-transport-inspired conditional path coupling, neural vector/rate field learning, and explicit guidance mechanisms for structured generation, demonstrating improved inference efficiency, cross-modality integration, and performance on challenging benchmarks. These architectures enable scalable, theoretically grounded generative modeling in domains ranging from image fusion and multimodal structured optimization to scientific and molecular design.

1. Probabilistic Formulation for Mixed Modalities

FMIP generalizes flow matching to the joint modeling of mixed continuous and discrete variables, crucial for domains such as mixed-integer linear programming (MILP) and multimodal scientific data. Let $d \in \mathbb{Z}^q$ denote integer variables, $c \in \mathbb{R}^{n-q}$ continuous variables, with $x = (d, c)$ sampled from an empirical data law $p_{\mathrm{data}}(d, c)$ over feasible solution sets $\mathcal{X}$ (e.g., in MILP) (Li et al., 31 Jul 2025). FMIP defines a time-indexed mixed stochastic process $(d_t, c_t)$, equipped with dedicated forward noising or mixing kernels for each modality:

• Continuous: $c_t|c_1 \sim \mathcal{N}(t c_1, (1-t)^2 I)$
• Discrete: $$d_t^{(i)}|d_1^{(i)} \sim \mathrm{Cat}(t \delta_{d_1^{(i)}} + (1-t)/K)$$

This framework leverages a joint process $(G_t) = (\text{problem-graph}, d_t, c_t)$ encoding structure, inputs, and state.

2. Flow-Matching Objective in the Multimodal Setting

FMIP trains a joint neural parameterization (typically a GNN for structured problems, or a U-Net for imaging) to recover the true vector and rate fields governing the multimodal process. The flow-matching loss couples the continuous and integer parts:

$$\mathcal{L}_\omega(\theta) = \mathbb{E}_{t, G_1} \left[\frac{\|\hat{c}_1^\theta(G_t) - c_1\|^2}{1-t} - \omega \sum_i \log p_\theta(d_1^i|G_t)\right]$$

where $\hat{c}_1^\theta$ is the predicted denoised continuous solution, $p_\theta$ denotes logits over integer variables, and $G_t$ encodes the structured problem. Each variable's target velocity and rate are set by analytic coupling kernels (e.g., $\nu_{t|1}(c_t|c_1) = \frac{c_1 - c_t}{1-t}$ for continuous, rate matrices for discrete). The same formulation naturally extends to image fusion and other modalities (Zhu et al., 17 Nov 2025, Li et al., 31 Jul 2025).

3. Model Architectures and Conditioning Mechanisms

FMIP architectures are strongly problem-dependent but share modular structure:

• Graph-based Models: For MILP, a tripartite GNN with TriConv (variable-to-constraint) and BiConv (constraint-to-variable) modules encodes integer variables, continuous variables, constraints, and their interactions. 12-layer deep GCNs with residual connections, LayerNorm, and GELU activations provide stability and scalability (Li et al., 31 Jul 2025).
• U-Net for Fusion: In image fusion, a U-Net with four down-/up-sampling stages and early-channel concatenation of modality-specific inputs models the fused distribution (Zhu et al., 17 Nov 2025).
• Mixed Inputs: Architectural conditioning involves concatenation or embedding of continuous, discrete, and context/problem representations (e.g., instance graphs or paired modality data) into input tensors or node features.
• No explicit cross-attention is required in the core FMIP fusion architectures; early fusion and unified representations suffice for optimal transport in most cases (Zhu et al., 17 Nov 2025).

4. Efficient Multimodal Sampling and Guidance

Flow-matching enables significantly more efficient sampling than multi-step score-based or diffusion models, often reducing ODE/SDE integration to one shot or a few steps. In FMIP:

• Euler/RK2 Integration: Few (1–3) fixed-step Euler or RK2 steps suffice to trace the joint vector/rate field, exploiting the near-constant transport paths in linear-interpolation couplings (Zhu et al., 17 Nov 2025, Li et al., 31 Jul 2025).
• Guidance: FMIP incorporates explicit guidance functionals $f(d,c)$ tailored to domain-specific objectives (e.g., MILP objective + constraint penalties):

$$f(d, c) = w^\top (d, c) + \gamma \sum_j [A_d d + A_c c - b]_{j,+}^2$$

Guidance is integrated by re-weighting candidate discrete transitions (importance sampling on rate matrices), gradient-based adjustment of continuous variables, and interaction with the learned vector field. The approach leverages the TFG-Flow paradigm for unbiased, efficient training-free guidance in multimodal flows (Lin et al., 24 Jan 2025, Li et al., 31 Jul 2025).

5. Continual and Multi-Task Learning

FMIP models are readily extended to multi-task or continual learning scenarios by augmenting flow-matching objectives with regularization for stability:

• Elastic Weight Consolidation (EWC): After task $k$ training, a diagonal Fisher information matrix over model parameters is estimated. An EWC penalty

$$L_{\mathrm{EWC}}(\theta) = \sum_{k<T} \sum_{i} \lambda F_{k,i} (\theta_i - \theta_{k,i}^*)^2$$

retains past task performance (Zhu et al., 17 Nov 2025).

• Experience Replay (ER): Mini-batches from previous tasks are replayed to maintain diversity and stability across tasks, leveraging a unified multi-task loss (Zhu et al., 17 Nov 2025).

6. Empirical Performance and Benchmarks

FMIP achieves state-of-the-art performance and practicality across several domains:

Benchmark	Setting	Metric/Outcome
MILP Benchmarks	7 canonical tasks	50.04% avg. primal gap reduction vs. GNN baselines (Li et al., 31 Jul 2025)
Image Fusion	TNO, LLVIP, RoadScene	Competitive PSNR, SSIM, etc. with 100–1000$\times$ faster inference (Zhu et al., 17 Nov 2025)
Sampling Steps	Mixed Modalities	1–3 steps (hundreds $\times$ faster) via one-shot/few-shot ODE integration (Zhu et al., 17 Nov 2025, Li et al., 31 Jul 2025)

Generalization to variable problem sizes, structural regularity, and domain constraints is supported by the architecture and training schemes.

7. Theoretical Guarantees and Extensions

FMIP inherits theoretical transport guarantees from continuous-time flow-matching: with infinite model capacity and vanishing integration step, the neural flow recovers the target distribution exactly. Extensions leveraging the Generator Matching (GM) framework further unify hybrid continuous/discrete (jumps, SDEs), manifold, and multimodal flows under a common loss and process algebra (Holderrieth et al., 2024).

Potential research directions include:

• Expansion to combinatorial and sequence-based optimization (e.g., SAT, non-linear programming, (Li et al., 31 Jul 2025)).
• Integration of learned initialization or cross-modal prior networks for even faster convergence.
• Application to new domains (e.g., multimodal scientific and molecular design; protein co-design with jump processes, (Holderrieth et al., 2024)).
• Joint training across more complex graph or cross-modal structures, leveraging advances in multimodal discrete flow matching and reference-based preference alignment (Susladkar et al., 12 Feb 2026).

• FMIP: Multimodal Flow Matching for Mixed Integer Linear Programming (Li et al., 31 Jul 2025)
• FusionFM: All-in-One Multi-Modal Image Fusion with Flow Matching (Zhu et al., 17 Nov 2025)
• TFG-Flow: Training-free Guidance in Multimodal Generative Flow (Lin et al., 24 Jan 2025)
• Generator Matching: Generative modeling with arbitrary Markov processes (Holderrieth et al., 2024)
• Best of Both Worlds: Multimodal Reasoning and Generation via Unified Discrete Flow Matching (Susladkar et al., 12 Feb 2026)