FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation

Abstract: Modern text-to-image diffusion models encode rich visual priors, but expose them only through one-way text-conditioned generation. Existing unified vision--LLMs derived from them recover bidirectional capability through large-scale joint pretraining or substantial retraining of the text pathway, discarding the strong image prior the text-to-image backbone already encodes. We introduce \emph{FullFlow}, a parameter-efficient recipe that upgrades a pretrained rectified-flow text-to-image model into a bidirectional vision--language generator by training only LoRA adapters and lightweight text heads. FullFlow keeps images in their native continuous flow and adds a discrete insertion process for text. Separate image and text timesteps turn inference into trajectory selection in a two-dimensional generative space, enabling text$\rightarrow$image, image$\rightarrow$text, joint sampling, and partial-text prediction with a single backbone. On Stable Diffusion 3 (SD3) under an identical trainable-parameter count and matched LoRA rank, FullFlow improves text$\rightarrow$image FID from $62.7$ to $31.6$ and image$\rightarrow$text CIDEr from $2.0$ to $99.4$ over a LoRA equivalent following the previous SOTA formulation (Dual Diffusion) at matched wall-clock training time, while reducing peak VRAM from ${\sim}84$\,GB to ${\sim}38$\,GB and raising throughput by ${\sim}8\times$ on two RTX A5000 GPUs in under 24 hours, training only ${\sim}5\%$ of the backbone parameters. The same recipe transfers to FLUX.1-dev and supports downstream VQA through partial-text generation. These results show that strong bidirectional vision--language capability can be unlocked from pretrained text-to-image flow models without full multimodal pretraining.

• The paper introduces FullFlow, a parameter-efficient approach that upgrades rectified-flow text-to-image models to support bidirectional vision–language generation with minimal adaptation.
• It employs lightweight LoRA adapters and small text-specific heads to maintain pretrained image priors while achieving significant improvements in FID and captioning metrics.
• The method facilitates diverse tasks—captioning, joint sampling, VQA—with reduced training time and VRAM requirements, offering a pragmatic baseline for multimodal generation.

FullFlow: Parameter-Efficient Bidirectional Upgrading of Rectified-Flow Text-to-Image Models

Overview and Motivation

FullFlow introduces a parameter-efficient approach for augmenting pretrained rectified-flow text-to-image (T2I) diffusion models with strong bidirectional vision-language generative capabilities. Existing state-of-the-art text-to-image diffusion models (e.g., SD3, FLUX.1-dev) encode highly structured visual priors, but operate exclusively in the text-to-image direction. Recent efforts toward unified vision-LLMs typically mandate expensive joint multimodal pretraining, significant retraining of the text pathway, or introduction of additional complex modeling components—sacrificing the pretrained visual prior and often prohibiting lightweight research iteration.

FullFlow addresses this gap by demonstrating that a pretrained T2I rectified-flow model can be efficiently "uplifted" to a unified, bidirectional generator by training only lightweight LoRA adapters and small text-specific heads, leaving the pretrained image process entirely intact. The approach leverages asymmetrical but compatible generative trajectories for text (discrete insertion process) and image (continuous flow), delivering text-to-image, image-to-text (captioning), joint image-text sampling, and partial-text (VQA/structured inference) within a single, non-autoregressive generative backbone.

Figure 1: Continuous flow matching (rectified flow) governing the native latent space evolution for images within FullFlow, highlighting path-based transport.

Methodology

Asymmetric Modality Formulation

• Images: Remain in the rectified-flow continuous latent space, utilizing the pretrained backbone's generative trajectory parameterized by an MLP-predictive velocity head.
• Text: Modeled as a discrete state evolving via an insertion process (Edit Flows). Text sequences are progressively constructed by inserting missing tokens into corrupted subsequences, parameterized by count and identity heads.
• Dual-Timestep Space: Decoupled image and text "times" ($t$ for image, $\tau$ for text) allow for mixed or conditional trajectories. For instance, text-to-image equates to $t:0\rightarrow1$, $\tau=1$; image-to-text to $t=1$, $\tau:0\rightarrow1$; joint sampling to $t,\tau:0\rightarrow1$ simultaneously.

Lightweight Parameter-Efficient Adaptation

FullFlow modifies a pretrained backbone only via:

• LoRA adapters (e.g., SD3: 136M params, $r=32$; $\sim$5% total params updated)
• Lightweight heads for text insertion and count estimation
• Text-timestep specific pathway for $\tau$ conditioning

All text encoders (T5, CLIP) are frozen; the discrete insertion process operates in T5 token space, decoding partially observed sequences to natural fragments for auxiliary encoder conditioning. This design maintains VRAM use and computational cost at commodity GPU-scale (<40GB VRAM, 2 GPUs), enabling rapid research iteration.

Dual-Timestep Training and Stabilization

Training is orchestrated via a scheduling distribution $\tau$0:

• Mixed ($\tau$1): Pretrains on all partial (image,text) states, enforcing tight cross-modal correspondence.
• Alternating clean: Specialized refinement with either image or text clean at each step—matching real deployment tasks (captioning or generation).
• An adaptive gradient balancing mechanism rescales the text loss using observed shared-parameter gradient norms (EMA), removing hyperparameter sensitivity due to severe scale disparity between image and text heads.
• Teacher Matching (Same-noise): To prevent drift of the image prior, a teacher-student objective matches the updated backbone's image velocity field to that of the frozen base model, conditioned on noise-corrupted text.

Empirical Results

Quantitative evaluation demonstrates that FullFlow delivers a strong upgrade in bidirectional generative performance and sample quality relative to prior LoRA-parameter-matched approaches (notably Dual Diffusion). All results are obtained under tightly controlled training budgets, matched wall-clock, and matched architectures.

• Text-to-Image Generation: On Stable Diffusion 3, FID is reduced from 62.7 ($\tau$20.7) to 31.6 with FullFlow, at matched training time and parameter update count.
• Image-to-Text Generation (Captioning): CIDEr sky-rockets from 2.0 to 99.4, with BERTScore-F1 from -0.27 to 0.44, compared to the strongest LoRA baseline under identical training compute.
• Sample Efficiency: FullFlow trains $\tau$38x faster and at less than half the peak VRAM ($\tau$438GB vs. 84GB), enabling 24hr training on two RTX A5000s.
• Transferability: The FullFlow recipe ported directly to FLUX.1-dev without architecture redesign, yielding FID 24.7 and CIDEr 73.6 at similar compute cost.
• Downstream Structured Tasks: Partial-text infilling enables VQA via image+question$\tau$5answer in the same backbone, attaining best-in-table VizWiz accuracy (50.7 vs. 29.9 for Dual Diffusion), with competitive MS-COCO CIDEr (54.9 vs. 61.6 for much larger models).

Analysis of Generative Trajectories

FullFlow's core innovation is the decoupling of text and image timesteps. The generative interface emerges as "trajectory selection" over a 2D time plane: any path between $\tau$6 and $\tau$7 produces a coherent multimodal pair, supporting flexible tasks including infilling, joint synthesis, and structured query answering. Empirically, a balanced diagonal path maintains maximal cross-modal agreement (CLIP similarity), while $\tau$8-sweeps (altering the order of modality denoising) trade off between semantic commitment and visual detail.

Implications and Future Directions

The results substantiate two major claims:

• Strong multimodal bidirectionality can be extracted from pretrained T2I rectified-flow priors with minimal parameter adaptation. This challenges the assumption that large-scale end-to-end multimodal retraining is requisite for unified generation.
• T2I diffusion models encode latent structure sufficient for upstream tasks such as captioning, VQA, and joint sampling even when trained solely for generation.

This finding opens potential for:

• Iterative, low-cost research and customization on powerful T2I backbones for diverse multimodal tasks.
• Lightweight downstream adaptation of frozen visual-LLMs in cost-sensitive settings (e.g., edge deployment, iterative prototyping).
• Extension of the parameter-efficient uplift protocol to larger generative backbones (e.g., SDXL, multi-modal LLMs), where compute savings will be multiplicatively more significant.

Limitations

• Experiments focus on $\tau$9 resolution due to hardware constraints; scaling to higher resolutions or paired dataset scales is presumed feasible but unverified.
• End-to-end instruction and dialog adaptation, necessary for SOTA structured understanding or world knowledge tasks, is out of scope but conceptually compatible with the FullFlow formulation.
• The theoretical optimality and limits of the insertion-based discrete process vis-à-vis mask-based diffusion or autoregressive text heads remain open.

Conclusion

FullFlow operationalizes a unified, bidirectional, and highly compute-efficient vision-language generator by asymmetrically augmenting a frozen T2I rectified-flow backbone with minimal trainable components. The observed emergence of strong captioning, VQA, and joint synthesis is a testament to the hidden versatility of T2I model priors and establishes a pragmatic new baseline for resource-conscious multimodal generative modeling.