VLMFP: Vision-Language Fusion Frameworks

Updated 7 July 2026

VLMFP is a polysemous umbrella term for vision-language fusion pipelines that integrate staged perception, verification, and downstream decision-making.
It has been applied to formal visual planning, 6D pose estimation, visual retrieval, and fairness-aware prompting with demonstrable performance gains.
Evaluations across grid-worlds, robotics, and autonomous systems show improved planning validity, geometric precision, and computational efficiency.

VLMFP is a context-dependent acronym in recent arXiv literature rather than a single standardized method. Across the listed papers, it denotes several distinct but related research programs centered on vision-LLMs: dual-VLM generation of PDDL for formal visual planning, fusion pipelines for pose estimation and visual place recognition, false-positive reasoning detection, federated fair prompting, embodied navigation and inspection systems, multi-frame autonomous-driving VQA, and training-free forward-pass optimization for video VLMs (Hao et al., 3 Oct 2025). A plausible implication is that “VLMFP” has evolved into a family label for vision-language-model–centric pipelines whose common structure is staged perception, fusion, verification, and downstream decision-making, rather than a uniquely defined architecture.

1. Acronym scope and naming conventions

In the surveyed literature, the same string is attached to substantially different technical objects. One line of work uses VLMFP for a “Dual-VLM-guided framework” that generates both PDDL domain and problem files from a visual instance and a natural-language rule description (Hao et al., 3 Oct 2025). Another uses VLMFP for selecting and fusing CLIP and DINOv2 for semantically grounded 6D object pose estimation in hand–object grasping (Sarowar et al., 8 Dec 2025). A third describes a “vision-LLM fusion pipeline” for visual place recognition built around LVLM captioning, DINOv2, BERT, token-wise attention recalibration, and agent-based cross-attention (Wang et al., 2024). Other usages include “Visual-LLM False Positive” reasoning in VLM reliability studies (Zhang et al., 6 Aug 2025), the federated fair-prompting framework FVL-FP (Chen et al., 3 May 2025), a VLM-informed multi-mode planetary navigation framework (Cheng et al., 20 Jun 2025), a multi-frame processing/prompting interpretation in autonomous-driving VQA (Gopalkrishnan et al., 2024), and a training-free forward-pass optimization protocol for video VLMs (Bastien et al., 5 May 2026).

Usage of VLMFP	Core task	Representative paper
Dual-VLM formal planning	PDDL domain/problem generation from images and rules	(Hao et al., 3 Oct 2025)
CLIP–DINOv2 fusion	6D object pose estimation	(Sarowar et al., 8 Dec 2025)
Multimodal retrieval fusion	Visual place recognition	(Wang et al., 2024)
False-positive reasoning analysis	VLM reasoning reliability	(Zhang et al., 6 Aug 2025)
Federated fair prompting	Group fairness in federated VLMs	(Chen et al., 3 May 2025)
Forward-pass optimization	Training-free anti-recomputation in video VLMs	(Bastien et al., 5 May 2026)

The ambiguity extends beyond VLMs proper. In one medical-imaging context, the query string is mapped to the Visual-Motion-Focus module, or VMF, for heart localization and segmentation in 4D CMR; that module exploits motion energy and a Gaussian RBF focus map, and reported 99.69% ROI coverage in the “train all” setting, a +1.7 Dice-point gain in the “train all, test all” condition, and about 2.5× faster training (Lima et al., 2021). This suggests that acronym-level retrieval for “VLMFP” can conflate vision-language work with older vision-only ROI modules.

2. Formal visual planning and symbolic model induction

The most explicit named formulation is “VLMFP: A Dual-VLM Framework for Formal Visual Planning,” which targets the bottleneck that VLMs can often generate PDDL problem files but struggle to generate correct PDDL domain files (Hao et al., 3 Oct 2025). The framework receives a natural-language domain description $n_d$ and an image $i_p$ , and seeks a PDDL domain $f_d$ , a PDDL problem $f_p$ , and a valid plan $\pi = [a_1, a_2, \dots, a_T]$ .

Its architecture separates roles between two VLMs. SimVLM is Qwen2-VL-7B fine-tuned on 430k grid-world datapoints; it produces a scenario description $n_p$ , simulates step-by-step action consequences, and judges whether an action sequence reaches the goal. GenVLM is GPT-4o API; it generates candidate PDDL domain and problem files and iteratively refines them from discrepancy feedback generated by cross-checking SimVLM execution against PDDL execution (Hao et al., 3 Oct 2025). The loop includes prescreening, bidirectional executability checks, discrepancy summarization, and constrained refinement through interfaces such as predicate and action-schema modification.

The framework is evaluated on six grid-world domains—FrozenLake, Maze, Sokoban, Package, Printer, and Overcooked—with generalization to unseen instances, unseen appearances, and altered rules. Averaged across domains, SimVLM achieved Task Desc accuracy of 95.5% and 82.6%, Exec Result accuracy of 85.5% and 87.8%, and Goal Reach accuracy of 82.4% and 85.6% for seen and unseen appearances, respectively. With SimVLM guidance, VLMFP generated PDDL files yielding 70.0% and 54.1% valid plans for unseen instances in seen and unseen appearances, outperforming CodePDDL’s 30.7% and 32.3% averages (Hao et al., 3 Oct 2025). Ablations further showed that removing prescreening reduced average valid plan rate to 47.5, removing feedback to 61.1, and removing iterative updates to 30.7, indicating that refinement is structurally central rather than auxiliary.

3. Fusion protocols for pose, retrieval, and visual reasoning

A second major meaning of VLMFP is a fusion template for combining complementary model families. In 6D object pose estimation for hand–object grasping, the VLMFP formulation compares and fuses CLIP as a VLM and DINOv2 as a VFM (Sarowar et al., 8 Dec 2025). The CLIP-based branch uses semantic grounding, attention-based cross-modal fusion, and an MLP head predicting quaternion and translation, while the DINOv2-based branch uses dense geometric feature extraction, keypoint localization, cosine-similarity matching, PnP/RANSAC, and differentiable ICP-like refinement. On the driller object, DINOv2 outperformed CLIP with ADD 28.45 mm versus 32.17 mm, ADD-S 29.12 mm versus 32.17 mm, rotation error 9.34° versus 11.68°, and translation error 17.52 mm versus 20.00 mm. The proposed hybrid strategy is explicitly staged: CLIP provides semantic grounding and coarse hypotheses, DINOv2 supplies geometric precision, and a refinement stage minimizes $L_{icp}$ and $L_{reproj}$ (Sarowar et al., 8 Dec 2025).

In visual place recognition, VLMFP appears as a multimodal representation-learning pipeline that fuses LVLM-generated text with image features (Wang et al., 2024). The image branch uses DINOv2 ViT-B/14, with last four layers fine-tuned and multi-level features from layers 2, 6, and 12 summed. The text branch uses frozen BERT-base-uncased with a learnable linear layer. Captions are generated by LLaMA-Adapter V2 and filtered by Attention-based Text Recalibration (AT-REC), then fused through Cross-Attention Multi-Modal Fusion (CA-MMF). The final descriptor is a concatenation of three fused query tokens, yielding a 2304-D descriptor. Quantitatively, the method reported 95.1/98.7/99.5 Recall@1/5/10 on Pitts250k-test, 92.0/96.4/96.9 on MSLS-val, 73.8/85.9/88.1 on MSLS-challenge, and 89.8/94.7/95.4 on SPED (Wang et al., 2024).

A third fusion usage is V3Fusion, which formalizes VLMFP as a multi-VLM ensemble protocol for visual reasoning (Tekin et al., 13 Mar 2026). Its key constructs are focal error diversity, CKA-based focal diversity, a GA-based pruning stage over the ensemble surface, task-specific fusion heads, and epistemic-uncertainty verification. On MMMU, V3Fusion-Rectify achieved 56.09% accuracy, an 8.09% gain over the best single VLM; on MMMU-Pro, it achieved 49.27%, a 4.87% gain. On OCR-VQA, V3Fusion-LED reached F1 86.82, BLEU-1 86.24, and EM 71.91 (Tekin et al., 13 Mar 2026). This suggests that, in one prominent strand of the literature, VLMFP denotes not just multimodal fusion but explicitly diversity-aware, uncertainty-aware fusion.

4. Reliability, fairness, and trustworthy inference

In reliability research, VLMFP denotes “Visual-LLM False Positive,” a failure mode in which the final answer is correct but the reasoning path is flawed (Zhang et al., 6 Aug 2025). The ViFP framework treats this as a distinct category from both correct reasoning and false negatives. It is training-free and organizes reasoning by first classifying each question into one of 11 question types, then constructing concise sub-question chains from a bank of ten templates such as object discovery, existence verification, object localization, characteristic description, scene description, relationship description, temporal information discovery, and spatial information discovery. Detection relies on dynamic consistency analysis between direct reasoning and multi-step reasoning, including the TDFM condition “True in Direct, False in Multi-step.” The framework also introduces the reliability metric

$\mathrm{VoC} = (P - Q)\, P \left(\frac{TN}{FP + TN}\right).$

On A-OKVQA, it improved Gemini-2.5 from 86.6 to 92.0, surpassing Prophet++ at 87.7; on GPT-4o, it improved A-OKVQA from 88.8 to 90.7 and OKVQA from 49.0 to 55.2 (Zhang et al., 6 Aug 2025).

In federated learning, VLMFP maps to FVL-FP, a parameter-efficient fairness framework for federated VLMs (Chen et al., 3 May 2025). Its three modules are Cross-Layer Demographic Fair Prompting (CDFP), Demographic Subspace Orthogonal Projection (DSOP), and Fair-aware Prompt Fusion (FPF). The vision encoder is prompted while the CLIP ViT-B/32 backbone remains frozen, DSOP removes demographic components by projecting image embeddings into the orthogonal complement of the estimated demographic subspace, and FPF aggregates prompts on the server with weights proportional to Accuracy $\times (1-\mathrm{Bias})$ . The paper reports an average reduction in demographic disparity of about 45% compared to standard FL baselines while keeping task performance within about 6% of state of the art (Chen et al., 3 May 2025). Here VLMFP denotes fairness-aware prompt tuning rather than fusion or planning, but the structural pattern remains similar: a frozen backbone, lightweight adapters, and explicit control signals for a desired system property.

5. Embodied autonomy and task execution

Several papers use VLMFP to denote embodied pipelines in which a VLM mediates between perception and control. In planetary navigation, the VLM-informed planning framework uses GPT-4V to classify terrain as flat, rocky, or challenging from forward-looking RGB images at 0.2 Hz, then switches between three navigation modes: Efficient, Safe, and Conservative (Cheng et al., 20 Jun 2025). Efficient mode disables perception and mapping and uses B-spline plus pure pursuit at 2.0 m/s; Safe mode uses occupancy mapping and A* at 0.8 m/s; Conservative mode uses 2.5D elevation mapping, costmaps, and A* at 0.5 m/s. In mixed-terrain simulation, multi-mode traversal reduced time from 1081.7 s to 602.6 s for roughly equal distance, yielding a reported efficiency improvement of about 79.5% while maintaining collision-free behavior (Cheng et al., 20 Jun 2025).

In manipulation, VLMFP appears as “VLMs for Fine-grained Planning/Manipulation” in the VL-MP framework that bridges GPT-4o, GroundingDINO, semantic 3D keypoints, and Kernelized Movement Primitives (Zhu et al., 4 Mar 2025). The bridge layer estimates keypoints with HRNet, calibrates them by topological constraints, and derives precise task parameters for Local Feature Enhanced KMP. In real-world pouring, VL-MP achieved 9/10 success in a normal environment and 8/10 in an ambiguous environment, compared with 5/10 and 2/10 for VLM+Prim and 4/10 and 0/10 for Diffusion (Zhu et al., 4 Mar 2025). The same paper reported improved shape preservation in Handwriting GShape, with $i_p$ 0 and $i_p$ 1 for VL-MP versus $i_p$ 2 and $i_p$ 3 for KMP.

A maritime interpretation uses an LLM–VLM fusion framework for cooperative UAV–USV port inspection (Din et al., 19 Jan 2026). The LLM generates symbolic plans with dependency graphs, while the VLM performs semantic inspection and compliance assessment. Among tested VLMs, Qwen2-VL achieved semantic correctness around 82.7–84.5 on real data with about 0.59 s inference, and Moondream2 achieved about 80.4–83.3 with about 0.29–0.31 s inference (Din et al., 19 Jan 2026). In autonomous driving, a multi-frame processing interpretation is realized by EM-VLM4AD, which ingests six synchronized camera views, fuses them through gated attention pooling, and answers DriveLM questions with lightweight T5 backbones (Gopalkrishnan et al., 2024). EM-VLM4AD-Base reported ROUGE-L 71.98 and CIDEr 3.20 while using 235M parameters, 9.47B FLOPs, and 0.94 GB memory, compared with the BLIP-2 baseline’s 3.96B parameters, 439B FLOPs, and 14.43 GB memory (Gopalkrishnan et al., 2024).

6. Runtime optimization and forward-pass efficiency

A further meaning of VLMFP concerns the forward pass itself rather than task semantics. “VLMaxxing through FrameMogging Training-Free Anti-Recomputation for Video Vision-LLMs” defines VLMFP as training-free anti-recomputation: reusing previously computed visual state when validation says it survives, and buying fresh evidence only when scene change, query change, or cache-topology issues require it (Bastien et al., 5 May 2026). The framework distinguishes C-PERSIST after-ingest follow-up reuse, C-VISION first-pass vision-tower pruning, candidate C-STREAM streaming reuse, and C-CEILING as the Amdahl-style accounting constraint on end-to-end gains.

The strongest measured effect occurs in after-ingest follow-up reuse on Qwen2.5-VL-7B-Instruct-4bit. Adaptive repaired-cache inheritance reduced follow-up latency by 14.90–35.92× across 93 paired queries with 0/93 paired choice drift and 0/93 correctness drift. Fixed $i_p$ 4 selective re-prefill yielded 9.48–20.37× speedup with the same 0/93 paired drift in the breadth setting (Bastien et al., 5 May 2026). In repeated-question stress over 50 turns, adaptive repair and scheduled refresh-10 showed 0/343 paired choice drift and 0/343 correctness drift, while fixed $i_p$ 5 showed 3/343 choice drift and 2/343 correctness drift. For first-pass pruning, Gemma 4-E4B-4bit achieved a clean 1.316× first-query speedup in the 32f short cell with no paired drift or parse failures on 20 items (Bastien et al., 5 May 2026). This strand treats VLMFP not as a task-level planner or fusion model but as an inference protocol for reducing redundant compute.

Across these usages, VLMFP consistently denotes a structured interface between vision-language modeling and an auxiliary control objective: symbolic validity, geometric precision, retrieval robustness, reasoning reliability, demographic fairness, navigation safety, manipulation accuracy, or computational efficiency. The specific expansion varies by paper, but the recurring design pattern is modular and staged—perception or representation extraction, selective fusion or prompting, explicit verification or calibration, and a downstream action or prediction head. This suggests that, within current arXiv usage, VLMFP is best understood as a polysemous umbrella term for VLM-centered pipeline design rather than a single canonical method.