FlowOVD: Learning Generative Latent Flows for Zero-shot Open-vocabulary Detection

Abstract: Open-vocabulary object detection (OVD) has achieved remarkable progress through large-scale vision-language pre-training. Existing methods, however, typically formulate OVD as a discriminative prediction problem, where decoder queries are either static or initialized from encoder features, thus limiting their diversity and flexibility. In this paper, we introduce a generative perspective by modeling decoder query generation as a continuous transport process in latent space. We propose FlowOVD, a text-conditioned query generation framework based on rectified flow that progressively transforms text-agnostic queries into text-guided queries. By introducing continuous latent query dynamics into a vision-LLM (VLM) based detector, our method avoids heuristic discrete query construction and enables more expressive semantic alignment for open-vocabulary detection. Without requiring additional training data, FlowOVD achieves 49.5 AP on COCO and 31.5 AP on LVIS, outperforming GroundingDINO by +1.2 AP (+2.5 %) and +4.1 AP (+15.0 %), respectively. The larger gain on the challenging long-tailed LVIS benchmark further highlights the effectiveness of continuous query generation for open-vocabulary generalization.

• The paper introduces generative latent flows to transform static, text-agnostic queries into rich, text-guided queries for improved open-vocabulary object detection.
• The paper demonstrates superior benchmark performance with up to 49.5 AP on COCO and significant gains on LVIS, highlighting enhanced accuracy over discriminative methods.
• The paper’s methodology reduces decoder complexity while achieving robust semantic alignment, paving the way for scalable, real-time, vision-language detection.

FlowOVD: Generative Latent Flows for Zero-shot Open-vocabulary Detection

Motivation and Background

Open-vocabulary object detection (OVD) targets the localization of objects without restricting recognition to a fixed set of categories. The landscape has advanced via vision-LLMs (VLMs), leveraging alignment between visual and textual domains inside Transformer architectures. Despite their success, OVD methods rely on discriminative prediction paradigms where decoder queries are either statically constructed (e.g., learnable embeddings) or heuristically selected from encoder output, resulting in constrained diversity and limited expressive capacity for handling open-ended semantic variation.

Generative modeling, particularly flows and diffusion processes, has gained traction for capturing complex data distributions, but its application to vision-language tasks typically focuses on geometric localization (box generation) rather than rich semantic generalization. Prior works such as DiffusionDet and FlowDet incorporate generative transport for bounding box distributions, but lack direct integration of cross-modal semantic conditioning necessary for OVD.

Methodology: Text-conditioned Query Flow

FlowOVD introduces a generative approach for query initialization in Transformer-based OVD frameworks, modeling the transformation of text-agnostic queries into text-guided queries as a continuous transport process in latent space, conditioned on textual inputs.

Figure 1: Query initialization as a continuous, text-conditioned flow, transporting text-agnostic queries to text-guided queries, thereby enhancing both diversity and efficiency in open-vocabulary detection.

Framework Overview

The system architecture consists of:

• Vision-language encoder extracting visual and textual features,
• Query Flow module operating in latent space,
• Transformer decoder consuming refined queries for detection.

Given an image $I$ and text prompt $P$, visual features $\mathbf{f}_I$ and textual features $\mathbf{f}_P$ are extracted. Initial text-agnostic queries $\mathbf{q}_0$ and text-guided queries $\mathbf{q}_1$ are constructed; the former are learnable, the latter are Top-$K$ selections from visual features based on similarity to text. FlowOVD learns a velocity field governing the transformation from $\mathbf{q}_0$ to $\mathbf{q}_1$ via rectified flow, forming more semantically aligned queries.

Figure 2: Overall pipeline—vision-language encoder, conditional query flow, and decoder integration; query flow involves matching, interpolation, and integration of velocity field.

Conditional Rectified Flow in Latent Space

• Matching: Alignment between unordered sets $\mathbf{q}_0$ and $P$0 using cosine similarity for set-level pairing.
• Interpolation: Linear interpolation defines intermediate states $P$1, parameterized by timestep $P$2.
• Velocity Network: Transformer-based network $P$3, with cross-attention to text features and FiLM-based time conditioning, predicts the velocity $P$4, implementing the ODE $P$5.

  Figure 3: Architecture of the velocity network $P$6 with flow layers, cross-attention, and FiLM modulations for time and condition injection.

Training combines standard detection losses (bounding box regression: L1 + GIOU; classification: focal loss) with flow matching (MSE between predicted and ground truth velocities).

Inference integrates the learned velocity field (via discretized Euler steps) from $P$7, optionally blending with the source by $P$8 to strike a balance between expressive and stable queries.

Comparative Analysis and Empirical Results

Figure 4: Evolution of query initialization—static, vanilla, mixed, and flow-based strategies illustrating progression towards generative latent modeling.

Performance Metrics

• On COCO, FlowOVD with Swin-T backbone achieves up to 49.5 AP, outperforming GroundingDINO by +1.2 AP (+2.5%), especially under large-scale vision-language pretraining.
• On LVIS, FlowOVD reaches 31.5 AP, yielding +4.1 AP (+15.0%) advantage over GroundingDINO, with marked improvements on rare categories ($P$9).

Efficiency ablation demonstrates that FlowOVD maintains AP with substantially fewer decoder layers, outperforming GroundingDINO even in lightweight configurations (2 layers, 170M params, faster inference).

Qualitative Insights

Figure 5: FlowOVD vs. GroundingDINO on COCO—FlowOVD excels in fine-grained localization and compositional understanding.

Figure 6: FlowOVD generalizes to rare categories (wok, sandals); reduces false/repeated detection; improves attribute-level discrimination in multi-instance scenes.

Figure 7: Additional FlowOVD examples emphasize robust open-vocabulary detection.

Failure cases remain: FlowOVD is sensitive to visual distractors (reflections), struggles under fine-grained compositional prompts with visually similar instances, and occasionally hallucinates cross-modal features (e.g., "dog face" on a cat).

Figure 8: Challenging scenarios including distraction by reflections, inability to disambiguate fine categories, and compositional hallucination highlight limitations.

Practical and Theoretical Implications

FlowOVD moves OVD query initialization from ad-hoc, discrete frameworks to a formal generative transport model in latent space, fundamentally enhancing query diversity and semantic alignment. This methodology enables richer cross-modal generalization, efficient parameter utilization, and improved grounding on challenging benchmarks—especially in long-tailed distributions.

From a theoretical standpoint, latent query flows serve as a bridge between generative modeling and discriminative detection, moving towards unified vision-language paradigms amenable to open-world, multimodal perception. The separation between content and positional queries reflects the balance required between semantic flexibility and spatial stability in Transformer-based detectors.

Practically, FlowOVD's reduced reliance on deep decoders and ease of integration into prevailing OVD pipelines unlocks scalability for real-time applications, extreme-class generalization, and complex multimodal scenarios including video understanding and fine-grained grounding.

Future Directions

Advancing generative latent query modeling requires addressing robustness to complex backgrounds, occlusions, and compositional language; integrating structured generative objectives and richer linguistic supervision may further improve grounding precision. Extending flow-based frameworks to broader multimodal tasks (e.g., video, referring expression segmentation) offers promising avenues for general-purpose perception.

Conclusion

FlowOVD establishes a principled generative framework for open-vocabulary detection, modeling decoder query generation as a continuous text-conditioned latent flow. Empirical results validate its superiority in both performance and efficiency over discriminative alternatives. By reframing query initialization, FlowOVD expands the operational semantics of vision-language detection and guides the field towards more expressive and adaptive cross-modal perception architectures.