F4-ITS: Fine-grained Food Image-Text Search

• F4-ITS is a training-free, vision-language framework that targets fine-grained food image-to-text retrieval by fusing image and VLM-generated text features.
• The method employs a uni-directional fusion strategy and ingredient-level re-ranking to address challenges in dense and sparse caption retrieval.
• Empirical results show marked improvements in Recall@1 and mAP, demonstrating that enhancing textual cues can make smaller models competitive in food retrieval tasks.

Searching arXiv for the exact topic and closely related retrieval work. {"^^^^^^^^3query3 OR abs:\3^^^^^^^^"Fine-grained Feature Fusion for Food Image-Text Search\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} Searching for related food image-text retrieval / VLM fusion papers for context. {"^^^^^^^^3query3 image-text retrieval VLM fusion CLIP SigLIP arXiv","max_results":^^^^3ti:\3query3^^^^,"sort_by":"submittedDate","sort_order":"descending"} F4-ITS, short for Fine-grained Feature Fusion for Food Image-Text Search, is a training-free, vision-LLM (VLM)-guided framework for food image-to-text retrieval that targets fine-grained semantic matching between a ^^^^^^^^3query3^^^^^^^^ food image and a caption corpus (&&&^^^^3query3^^^^&&&). The framework is designed for the food domain, where visually similar dishes often differ in subtle ingredient composition, cooking method, or plating, and where general-purpose image-text encoders such as CLIP and SigLIP can underperform because captions are often too coarse to capture ingredient-level distinctions. In this usage, the term refers to image-text search, not to intelligent transportation systems; its central problem is retrieving either a single best dense caption or a top-PRESERVED_PLACEHOLDER_^^^^3query3^^^^ set of sparse ingredient captions for a given food image.

^^^^3ti:\3^^^^. Retrieval setting and domain-specific difficulty

The paper formulates F4-ITS around food image-to-text retrieval: given a ^^^^^^^^3query3 food image, the system retrieves the most semantically relevant text caption from a caption corpus (&&&^^^^3query3^^^^&&&). The application settings explicitly listed are dietary monitoring, smart kitchens, nutritional analysis, restaurant automation, and recipe understanding and recommendation. The task is decomposed into two retrieval scenarios.

The first scenario is single image-text retrieval (dense caption retrieval), where the goal is to retrieve the single best dense caption describing the dish. The second is top-PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^ retrieval (sparse ingredient retrieval), where the objective is to retrieve the top-PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^ ingredient captions corresponding to visible food components. The paper evaluates the first via top-^^^^3ti:\3^^^^ / top-5 caption retrieval and the second via ingredient-level ranking precision.

The food domain is treated as unusually difficult because dishes may differ only in fine-grained details. The paper identifies three concrete failure modes for general-purpose image-text models in this setting: captions in food datasets are often too coarse or do not capture fine-grained ingredients; manual creation of rich food descriptions is expensive; and even with VLMs, precise ingredient-level search and ranking remain difficult. This framing situates F4-ITS as a retrieval-layer method that enriches existing pretrained encoders rather than replacing them.

^^^^3 OR abs:\3^^^^. Framework architecture and representational pipeline

F4-ITS is presented as a training-free, zero-shot, VLM-guided retrieval system built on top of pretrained image-text encoders (&&&^^^^3query3^^^^&&&). Its pipeline has three stages:

^^^^3ti:\3^^^^. Embedding extraction ^^^^3 OR abs:\3^^^^. Multi-modal feature fusion

1. Feature re-ranking

The paper explicitly identifies two principal contributions. The first is a uni-directional (and bi-directional) multi-modal fusion strategy that combines image embeddings with VLM-generated textual descriptions. The second is a feature-based top-$k$ re-ranking mechanism that uses predicted food ingredients to refine retrieval results.

For a ^^^^^^^^3query3^^^^^^^^ image $IQ$, the framework computes the image embedding

$E_{img} = ImageEncoder(IQ).$

For VLM-generated text, it computes either

$E_{densecaption} = TextEncoder(DenseCaptionText)$

or

$$E_{sparsecaption} = TextEncoder(SparseCaptionText).$$

This architecture is intentionally layered over pretrained encoders rather than dependent on end-to-end fine-tuning. The paper benchmarks multiple pretrained open-source image-text encoders, including ViT-B-3^^^^3 OR abs:\3^^^^, ViT-L-^^^^3ti:\3^^^^4, ViT-H-^^^^3ti:\3^^^^4, ViT-g-^^^^3ti:\3^^^^4, ViT-bigG-^^^^3ti:\3^^^^4, ViT-H-^^^^3ti:\3^^^^4-378-quickgelu, and ViT-L-^^^^3ti:\3^^^^6-SigLIP^^^^3 OR abs:\3^^^^-5^^^^3ti:\3 OR abs:\3^^^^, and augments them with VLM outputs generated using Gemma-3n or Gemini ^^^^3 OR abs:\3^^^^.5 Flash. A plausible implication is that the method is intended as a modular retrieval-time augmentation layer for existing VLM and CLIP/SigLIP-style stacks, rather than as a new pretrained backbone.

3. Multi-modal fusion mechanisms

The central representation step in F4-ITS is weighted fusion of the ^^^^^^^^3query3^^^^^^^^ image embedding with a VLM-generated textual embedding (&&&^^^^3query3^^^^&&&). In the uni-directional formulation, only the ^^^^^^^^3query3^^^^^^^^ representation is enhanced. For dense caption retrieval, the paper defines

$$E_{fused} = w_{img} * E_{img} + w_{text} * E_{densecaption},$$

and for sparse caption retrieval,

$$E_{fused} = w_{img} * E_{img} + w_{text} * E_{sparsecaption}.$$

The reported default fusion weights are:

• PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^
• PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^

These values were selected through experiments, and the paper states that optimal text contribution is about ^^^^3 OR abs:\3query3^^^^–3^^^^3query3^^^^% text weight. It also reports that too much text hurts and no text also hurts, so the preferred operating regime is moderate textual augmentation rather than text-dominant fusion.

The paper also describes a bi-directional variant. In this setting, not only the ^^^^^^^^3query3^^^^^^^^ embedding but also the index caption embeddings are fused with the ^^^^^^^^3query3^^^^^^^^ image embedding at runtime. The paper gives the formula in abbreviated / slightly malformed form as

PRESERVED_PLACEHOLDER_^^^^3ti:\3 OR abs:\3^^^^

and states the weights

• PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^
• PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4

Retrieval is performed by cosine similarity: PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5

Empirically, the paper finds that bi-directional fusion does not meaningfully improve over uni-directional fusion and therefore prefers the uni-directional variant for practical use. This is a notable design conclusion because it narrows the effective contribution of F4-ITS to ^^^^^^^^3query3 enrichment plus optional second-stage refinement, rather than full symmetric re-embedding of both ^^^^^^^^3query3^^^^^^^^ and index.

4. Ingredient-aware top-PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6 re-ranking

The second major component of F4-ITS is a feature-based top-PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7 re-ranking stage tailored to sparse ingredient retrieval (&&&^^^^3query3^^^^&&&). The process begins with a VLM-generated sparse food description, such as a short ingredient list, which is embedded and fused with the image embedding: PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8 again using

• PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9
• PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3query3^^^^

An initial candidate set is then retrieved: PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3ti:\3^^^^

The sparse description is next split into individual ingredient phrases PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3 OR abs:\3^^^^, and each phrase is embedded separately: PRESERVED_PLACEHOLDER_^^^^3 OR abs:\33^^^^ Each retrieved candidate PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^4 is then re-scored by maximum similarity against the parsed ingredient embeddings: PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^5

The paper characterizes this as transforming retrieval into a high-recall initial retrieval followed by high-precision re-ranking. The intended effect is to improve discrimination among semantically close candidates by emphasizing exact ingredient matches. In the paper’s interpretation, this is especially useful for visually similar dishes that differ by one salient ingredient. The mechanism is therefore not a generic re-ranker over full captions, but a phrase-level ingredient-matching step that injects itemized semantic evidence into ranking.

5. Experimental protocol and quantitative behavior

The paper evaluates F4-ITS on the VLM Metafood Challenge / Dishcovery datasets (&&&^^^^3query3^^^^&&&). The full challenge datasets are listed as:

• MTF^^^^3 OR abs:\3^^^^5-VLM-Challenge-Web: ^^^^3ti:\33^^^^9K image-caption pairs
• MTF^^^^3 OR abs:\3^^^^5-VLM-Challenge-Synth: ^^^^3 OR abs:\3^^^^58K image-caption pairs

For evaluation, the paper uses smaller subsets:

• MTF^^^^3 OR abs:\3^^^^5-VLM-Challenge-Dataset-Web-^^^^3ti:\33: ^^^^3ti:\3 OR abs:\3^^^^,68^^^^3query3^^^^ image-caption pairs
• MTF^^^^3 OR abs:\3^^^^5-VLM-Challenge-Dataset-Synth-^^^^3ti:\3: ^^^^3ti:\3^^^^5, OR abs:\3^^^^7 image-caption pairs

Each image has both a dense caption and a sparse caption, and the authors state that they use synthetically generated captions as ground truth to reduce noise and improve source-target alignment.

Component	Dataset / metric	Reported details
Dense and sparse retrieval	Web-^^^^3ti:\33^^^^K, Synth-^^^^3ti:\3^^^^5K	Each image has dense and sparse captions
Single-caption evaluation	Recall@^^^^3ti:\3^^^^, Recall@5	Used for dense-caption retrieval
Top-PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^6 ingredient retrieval	mAP	PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^7 varies by image and matches the number of ground-truth items

The dense-caption results show consistent gains over baselines. Examples reported in the paper include:

• ViT-B-3^^^^3 OR abs:\3^^^^ on Web-^^^^3ti:\33^^^^K: baseline R@^^^^3ti:\3^^^^ = ^^^^3query3^^^^.335, F4-ITS + Gemini = ^^^^3query3^^^^.5 OR abs:\3^^^^
• ViT-H-^^^^3ti:\3^^^^4-378-quickgelu on Web-^^^^3ti:\33^^^^K: baseline R@^^^^3ti:\3^^^^ = ^^^^3query3^^^^.5, F4-ITS + Gemini = ^^^^3query3^^^^.6
• ViT-g-^^^^3ti:\3^^^^4 on Web-^^^^3ti:\33^^^^K: baseline R@^^^^3ti:\3^^^^ = ^^^^3query3^^^^.45, F4-ITS + Gemini = ^^^^3query3^^^^.587

The paper summarizes dense-caption gains as roughly ~^^^^3ti:\3query3^^^^% top-^^^^3ti:\3^^^^ improvement.

For sparse-caption retrieval, the reported improvements are also substantial. Examples include:

• ViT-B-3^^^^3 OR abs:\3^^^^ on Web-^^^^3ti:\33^^^^K: baseline R@^^^^3ti:\3^^^^ = ^^^^3query3^^^^. OR abs:\3query3^^^^5, F4-ITS + Gemini = ^^^^3query3^^^^.346
• ViT-H-^^^^3ti:\3^^^^4-378-quickgelu on Web-^^^^3ti:\33^^^^K: baseline R@^^^^3ti:\3^^^^ = ^^^^3query3^^^^.35, F4-ITS + Gemini = ^^^^3query3^^^^.4 OR abs:\3^^^^8

The paper summarizes sparse-caption gains as about ~7.7% improvement in top-^^^^3ti:\3^^^^ retrieval.

For ingredient-level top-PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^8 retrieval, the re-ranking stage is the crucial driver. Reported examples include:

• ViT-B-3^^^^3 OR abs:\3^^^^ on Web-^^^^3ti:\33^^^^K: baseline mAP = ^^^^3query3^^^^., F4-ITS + Gemini = ^^^^3query3^^^^.379
• ViT-L-^^^^3ti:\3^^^^6-SigLIP OR abs:\3^^^^-5^^^^3ti:\3 OR abs:\3^^^^ on Synth-^^^^3ti:\3^^^^5K: baseline mAP = ^^^^3query3^^^^., F4-ITS + Gemini = ^^^^3query3^^^^.4

The paper states that re-ranking yields up to ~^^^^3 OR abs:\3^^^^8.6% mAP gain.

Several ablation-style observations further specify the empirical profile. Dense-caption fusion outperforms sparse-caption fusion by about 8% on average, with an improvement range of about 3.7% to ^^^^3ti:\3^^^^4.3% in Recall@^^^^3ti:\3^^^^. Heterogeneous matching provides only a small boost, about ^^^^3ti:\3^^^^.^^^^3ti:\3 to 3.5%. The paper also reports that smaller models can become highly competitive: in one dense-retrieval example, ViT-B-3^^^^3 OR abs:\3^^^^ + Gemini reaches ^^^^3query3^^^^.5^^^^3ti:\3 OR abs:\3^^^^ R@^^^^3ti:\3^^^^, whereas ViT-bigG-^^^^3ti:\3^^^^4 + Gemini reaches ^^^^3query3^^^^.499 R@^^^^3ti:\3^^^^. This suggests that the fusion mechanism can compensate, at least partially, for smaller encoder capacity in this domain.

6. Interpretation, limitations, and scope

The paper positions F4-ITS as a general-purpose image search framework, while demonstrating it specifically on food retrieval (&&&^^^^3query3^^^^&&&). It explicitly suggests possible relevance to retail, fine-grained object search, and extensions to text-to-image or image-to-image retrieval, though these broader use cases are presented as suggested rather than empirically established.

Its practical value is described in terms of four properties already established by the reported experiments: it requires no training; it can improve retrieval using pretrained models + VLM prompting; it can make smaller models much more competitive; and it improves both semantic caption retrieval and ingredient-level precision. The paper’s strongest empirical message is that high-quality VLM-generated text can steer image-text embeddings toward fine-grained food semantics, and that phrase-level ingredient re-ranking can materially improve the quality of the top retrieved captions.

The limitations are equally explicit. Performance depends on VLM caption quality; when dense or sparse descriptions are noisy, gains shrink. The fusion weights are manually chosen, and the paper mentions adaptive fusion as future work. Bi-directional fusion is introduced but is not clearly beneficial. Gains are smaller on synthetic datasets than on real images, which the paper attributes to distribution mismatch and the fact that OpenCLIP models are trained mainly on real-world web images. On the original noisy captions, fusion still helps only slightly—around ^^^^3ti:\3^^^^–^^^^3 OR abs:\3^^^^%—and can be close to baseline or slightly worse.

A final point of terminology is important for disambiguation. Within the arXiv literature, the string “F4” appears in multiple unrelated mathematical and algorithmic contexts, but F4-ITS as a paper title refers specifically to Fine-grained Feature Fusion for Food Image-Text Search (&&&^^^^3query3^^^^&&&). The abstract states that code and test datasets will be made publicly available at https://github.com/mailcorahul/f4-its, which locates the framework within the emerging body of retrieval-oriented, training-free VLM augmentation methods rather than within the unrelated literature on Coxeter type PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^9, Brauer algebras, or intelligent transportation localization.