---
title: Compositional Zero-Shot Food Recognition
url: https://www.emergentmind.com/topics/compositional-zero-shot-food-recognition-czsfr
type: topic
---

# Compositional Zero-Shot Food Recognition

to=arxiv_search.query  天天中彩票是  银航: json
{"query":"id:2509.03873 OR ti:\"SalientFusion: Context-Aware Compositional Zero-Shot Food Recognition\"","max_results":5,"sort_by":"relevance"}
to=arxiv_search.query 彩票直属 _天天啪: json
{"query":"id:2510.11106 OR ti:\"Compositional Zero-Shot Learning: A Survey\"","max_results":5,"sort_by":"relevance"}
Compositional Zero-Shot Food Recognition (CZSFR) is a food-domain instantiation of Compositional Zero-Shot Learning (CZSL) in which a dish is not treated as a monolithic class, but as a composition of two primitives: a cuisine and an ingredient. Under this formulation, food recognition becomes the problem of recognizing unseen cuisine–ingredient combinations by recombining primitives observed during training, rather than enumerating every dish class in advance. The formulation was introduced together with the SalientFusion method, which addresses food-specific failure modes through context-aware visual fusion and text adaptation [2509.03873]. Within the broader CZSL literature, CZSFR inherits the central premise that novel combinations of known primitives should be recognizable at inference time, while adapting that premise to the particular compositionality, contextuality, and ambiguity of food categories [2510.11106].

## 1. Relation to compositional zero-shot learning

CZSL is defined as the recognition of unseen combinations of known attributes and objects during inference, and its central difficulty is that the visual appearance of primitives is highly contextual. The survey literature characterizes this as a combinatorial generalization problem: training data are available only for a subset of possible compositions, but inference must extend to novel combinations [2510.11106]. Food is a natural application domain because dishes are inherently compositional and can be described through ingredients, preparation methods, flavors, textures, cuisines or regions, and dish types.

In a food-domain reinterpretation of CZSL, the primitive space can include ingredients or food objects such as chicken, tofu, rice, pasta, or mango; preparation methods or states such as grilled, fried, baked, boiled, or fermented; attributes such as spicy, sweet, crispy, creamy, or sour; cuisines or regions such as Italian, Thai, Japanese, or Mexican; textures or forms such as chopped, shredded, layered, or whole; and dish types such as soup, salad, curry, or dessert [2510.11106]. This suggests that food recognition is especially well matched to compositional modeling because many plausible dishes can be formed from primitives that are already known individually, even when their combinations are absent from the training set.

A common misconception is that zero-shot food recognition is adequately modeled by assigning a single unseen class label to each dish. The CZSFR formulation rejects that view by explicitly factorizing dish labels into reusable primitives. A plausible implication is that this factorization improves generalization precisely because the space of possible dishes is open-ended, while the underlying primitive inventory is more stable.

## 2. Formalization of CZSFR

The SalientFusion paper defines the compositional label space through the Cartesian product of two primitive sets,
\[
\mathcal{A} = \{a_1, a_2, \dots, a_m\}, \qquad
\mathcal{O} = \{o_1, o_2, \dots, o_n\},
\]
where \(\mathcal{A}\) is the set of states or cuisines and \(\mathcal{O}\) is the set of objects or ingredients. The full compositional space is
\[
\mathcal{C} = \mathcal{A} \times \mathcal{O},
\]
with \(p = |\mathcal{C}|\) [2509.03873].

Seen and unseen compositions are disjoint subsets. Training uses only seen compositions,
\[
\mathcal{T} = \{(\bm{x}_i, c_i) \mid \bm{x} \in \mathcal{X}, c \in \mathcal{C}_{\text{se}}\},
\]
while test-time prediction is performed over the target set
\[
\mathcal{C}_{\text{tgt}} = \mathcal{C}_{\text{se}} \cup \mathcal{C}_{\text{us}}.
\]
Operationally, labels such as “braise pork” or “stir-fry chicken” are represented as compositions \(\langle a_i, o_j\rangle\), with cuisine mapped to the attribute or state role and ingredient mapped to the object role. The task is then to recognize unseen compositions such as “braise chicken” or “stir-fry pork” by recombining previously observed primitives [2509.03873].

This pairwise formalization is consistent with the general CZSL setup surveyed in the broader literature, where each class label is a pair \(c = (a,o)\) drawn from a composition space \(\mathbf{C} = \mathbf{A} \times \mathbf{O}\). The survey distinguishes only-unseen closed-world evaluation, generalized closed-world evaluation, and open-world evaluation, the last of which includes all possible compositions, including unfeasible ones [2510.11106]. In the food domain, this implies that benchmark design is inseparable from ontology design: one must decide whether primitives denote cuisines, cooking methods, flavors, textures, or other factors, and whether implausible food pairs should be scored or excluded.

## 3. Food-specific challenges

The CZSFR formulation is motivated not only by the compositional nature of food labels but also by three specific challenges that are especially severe in food imagery [2509.03873].

The first is **background redundancy**. Food photographs often contain plates, bowls, tables, chopsticks, garnishes, and other incidental context. The paper notes that a vision-language model may describe a grilled beef image as “grilled beef on a round plate placed on a table,” indicating attention to non-food cues. If representations are dominated by such clutter, the model may learn spurious correlations rather than food-specific evidence, impairing generalization to unseen cuisine–ingredient compositions.

The second is **role confusion between staple and side dishes**. Food images frequently contain multiple objects, and the visually dominant region is not always the primary ingredient. Noodles, vegetables, or accompaniments may be mistaken for the main ingredient, or the cuisine may be associated with the wrong object. This problem is structurally important because CZSL assumes a meaningful factorization into primitives; if the underlying ingredient assignment is wrong, the predicted composition is wrong even when parts of the scene are recognized correctly.

The third is **semantic bias in a single attribute**. A cuisine term can shift meaning depending on the ingredient. The example given is “stew”: in **stew sparerib**, it aligns more with **braised**, whereas in **stew seafood**, it aligns more with **boil**. A fixed text prompt for a cuisine is therefore semantically ambiguous across compositions. In the food domain, this is a particularly acute instance of the contextuality emphasized by the CZSL survey, which argues that primitives are not globally invariant concepts but change in appearance and interpretation as a function of the object they modify [2510.11106].

These challenges also clarify what CZSFR is not. It is not merely standard fine-grained food classification with unseen labels, and it is not merely prompt engineering over dish names. Its central problem is the interaction between compositional generalization and food-specific contextuality.

## 4. SalientFusion: context-aware architecture

SalientFusion is a two-stage context-aware CZSL pipeline consisting of **SalientFormer** and **DebiasAT**. The first stage improves visual representation by suppressing irrelevant background and incorporating depth; the second stage refines text prompts using salient visual features to reduce semantic bias. The method uses CLIP-style encoders but does not rely on the raw image alone [2509.03873].

### SalientFormer

SalientFormer takes an input image \(\bm{x}\) and applies two auxiliary transforms,
\[
T_s : \bm{x} \mapsto \bm{x}_f, \qquad
T_d : \bm{x} \mapsto \bm{x}_d,
\]
where \(\bm{x}_f\) is the foreground or segmented image and \(\bm{x}_d\) is the depth map. The original image, foreground image, and depth image are tokenized into patches and encoded by a CLIP-based ViT image encoder \(E_v\). A projection layer with parameters \(\mathbf{W}_g\) maps token embeddings into a shared latent space,
\[
\bm{\hat{x}}_i = E_v(\bm{x}_i)\mathbf{W}_g^\top, \qquad i \in \{\text{o}, \text{f}, \text{d}\}.
\]

The three branches correspond to the original image (\(\text{o}\)), foreground image (\(\text{f}\)), and depth image (\(\text{d}\)). Query, key, and value tensors are formed by mixing foreground and depth while using the original image as value:
\[
\mathbf{Q} \leftarrow \alpha \bm{\hat{x}}_d + (1-\alpha)\bm{\hat{x}}_f, \qquad
\mathbf{K} \leftarrow \alpha \bm{\hat{x}}_f + (1-\alpha)\bm{\hat{x}}_d, \qquad
\mathbf{V} \leftarrow \bm{\hat{x}}_o,
\]
with \(\alpha \in [0,1]\). Multi-head attention then fuses them:
\[
\bm{x}_s = M(\mathbf{Q}, \mathbf{K}, \mathbf{V}).
\]

The stated purposes are explicit. Foreground segmentation removes plates, tables, and other irrelevant regions. Depth features capture object volume and relative distance, helping distinguish staple dish from side dish. The fusion mechanism encourages attention to meaningful food regions rather than full-scene clutter [2509.03873].

### DebiasAT

DebiasAT addresses the text-side ambiguity of cuisine prompts. For a text embedding \(\bm{t} \in \mathbb{R}^d\) and salient visual tokens \(\bm{x}_s\), DebiasAT applies multi-head attention and a feed-forward block:
\[
\bm{\tilde{t}} = \mathcal{N}\big(\bm{t} + M(\bm{t}, \bm{x}_s, \bm{x}_s)\big),
\]
\[
\bm{t}' = \mathcal{N}\big(\bm{\tilde{t}} + \mathcal{F}_{\text{FFN}}(\bm{\tilde{t}})\big),
\]
followed by
\[
\bm{t}_s = \bm{t} + \lambda \cdot \bm{t}',
\]
where \(\lambda \in [0,1]\) controls the amount of adaptation.

The purpose is to make text prompts composition-aware rather than static. Instead of using a fixed cuisine embedding that may conflate different senses of a term such as “stew,” the method nudges the text representation toward the current visual context [2509.03873]. Within the taxonomy of the CZSL survey, this places SalientFusion close to the cross-modal disentanglement family, which combines visual grounding with semantic language structure and is presented as an especially promising direction for domains rich in textual supervision such as food [2510.11106].

### Prompt learning and objective

The method learns separate prompt prefixes for attribute, object, and composition branches. For a composition \(c_{i,j}=\langle a_i,o_j\rangle\),
\[
\bm{P}^{a}_i = [\bm{p}^{a}_1, \dots, \bm{p}^{a}_r, \bm{v}^{a}_i], \qquad
\bm{P}^{o}_j = [\bm{p}^{o}_1, \dots, \bm{p}^{o}_r, \bm{v}^{o}_j], \qquad
\bm{P}^{c}_{i,j} = [\bm{p}^{c}_1, \dots, \bm{p}^{c}_r, \bm{v}^{c}_i, \bm{v}^{c}_j].
\]
Here, \(\bm{p}_1,\dots,\bm{p}_r\) are learnable prefix tokens, \(\bm{v}\) are vocabulary tokens, and \(r\) is the prefix length. These prompts are encoded by the text encoder \(E_t\) into attribute, object, and composition embeddings [2509.03873].

Classification probabilities are defined for attribute, object, and composition prediction using CLIP-style temperature-scaled similarities:
\[
p(a_i \mid \bm{x}) =
\frac{\exp(\bm{x}^a \cdot \bm{t}_i^a/\tau)}
{\sum_{k=1}^{m} \exp(\bm{x}^a \cdot \bm{t}_k^a/\tau)},
\]
\[
p(o_i \mid \bm{x}) =
\frac{\exp(\bm{x}^{o} \cdot \bm{t}_i^{o}/\tau)}
{\sum_{k=1}^{n} \exp(\bm{x}^{o} \cdot \bm{t}_k^{o}/\tau)},
\]
\[
p(c_{i,j} \mid \bm{x}) =
\frac{\exp(\bm{x}^{c} \cdot \bm{t}_{i,j}^{c}/\tau)}
{\sum_{k=1}^{p} \exp(\bm{x}^{c} \cdot \bm{t}_k^{c}/\tau)}.
\]

The branch-wise cross-entropy losses are
\[
\mathcal{L}^k = -\frac{1}{|\mathcal{X}|} \sum_{x \in \mathcal{X}} \log p(k \mid \bm{x}), \qquad k \in \{\text{a}, \text{o}, \text{c}\},
\]
and the final objective is
\[
\mathcal{L} = \beta^a\mathcal{L}^a + \beta^o\mathcal{L}^o + \beta^c\mathcal{L}^c.
\]

## 5. Benchmarks and evaluation protocol

The SalientFusion work introduces two benchmarks by re-annotating existing datasets: **CZSFood-90** from **ETH Food-101** and **CZSFood-164** from **VireoFood-172** [2509.03873]. Each food category is redefined as a \((\text{cuisine}, \text{ingredient})\) pair. Cuisines are derived using a large language model to generate common cooking methods, ingredients are manually selected as the one-to-two most salient ingredients per dish, and each composition is verified against culinary reference guides. The resulting benchmarks contain 90 compositions for CZSFood-90 and 164 compositions for CZSFood-164.

The split protocol follows CZSL-style evaluation with training, validation, and closed-world test sets, plus an additional **real-world test** split containing unseen compositions that reflect practical novelty, such as new ingredient combinations or recombinations not present in training [2509.03873]. This protocol is consistent with the survey’s emphasis on closed-world and generalized closed-world settings, while the explicit inclusion of real-world novelty suggests a closer approximation to practical deployment conditions [2510.11106].

The reported statistics are as follows.

| Benchmark | Primitive counts | Split statistics |
|---|---:|---|
| CZSFood-90 | 11 cuisines, 62 ingredients | 32 train compositions / 28.8k images; 16 validation compositions / 9.6k images; 16 closed-world test compositions / 9.6k images; 58 real-world test compositions / 51.6k images |
| CZSFood-164 | 16 cuisines, 118 ingredients | 54 train compositions / 32.1k images; 20 validation compositions / 8.1k images; 20 closed-world test compositions / 8.6k images; 110 real-world test compositions / 64.3k images |

Evaluation follows prior CZSL practice using **S** for best seen accuracy, **U** for best unseen accuracy, **HM** for the best harmonic mean of seen and unseen performance, and **AUC** for the area under the seen–unseen tradeoff curve. The paper also mentions **calibration bias** during testing to balance seen and unseen predictions [2509.03873]. The survey explains the same logic as bias correction by adding a scalar bias to novel composition scores and sweeping that bias to obtain the tradeoff curve summarized by AUC [2510.11106].

## 6. Empirical performance, transfer, and unresolved issues

On the two food benchmarks, SalientFusion is reported to achieve the best results under both closed-world and real-world testing [2509.03873].

For **CZSFood-90**, the reported closed-world scores are \(S = 97.8\), \(U = 69.5\), \(HM = 68.0\), and \(AUC = 61.7\). On the real-world split, the scores are \(S = 96.9\), \(U = 27.5\), \(HM = 39.0\), and \(AUC = 25.4\). For **CZSFood-164**, the closed-world scores are \(S = 96.4\), \(U = 78.3\), \(HM = 74.4\), and \(AUC = 70.9\), while the real-world scores are \(S = 97.6\), \(U = 26.5\), \(HM = 38.0\), and \(AUC = 24.8\). Against **Troika**, described as the strongest baseline, the reported improvements are HM gains of **+1.8, +3.8, +4.7, +0.7** and AUC gains of **+5.1, +7.2, +4.4, +0.9**, with particular emphasis on unseen accuracy [2509.03873].

Ablation studies are aligned with the method’s stated design rationale. For SalientFormer, original-only input is weakest; adding depth improves performance; foreground plus depth is better; and all three branches are best. For DebiasAT, removing the module reduces performance on both datasets. On **CZSFood-90 closed-world**, performance changes from \(HM = 67.5, AUC = 60.9\) without DebiasAT to \(HM = 68.0, AUC = 61.7\) with DebiasAT. On **CZSFood-164 closed-world**, it changes from \(HM = 72.2, AUC = 65.6\) to \(HM = 74.4, AUC = 70.9\) [2509.03873]. The hyperparameter study reports the best \(\alpha\) as \(0.7\) on CZSFood-90 and \(0.4\) on CZSFood-164; the paper interprets this as reflecting stronger staple–side separation in the former and weaker separation in the latter.

The method also transfers to standard CZSL datasets. Reported scores are \(HM = 39.4\) and \(AUC = 22.2\) on **MIT-States**, and \(HM = 51.6\) and \(AUC = 38.1\) on **UT-Zappos** [2509.03873]. This is consistent with the survey’s broader conclusion that CLIP-based cross-modal methods, prompt-based methods, conditional attribute modeling, prototype learning, and retrieval augmentation are among

Source: https://www.emergentmind.com/topics/compositional-zero-shot-food-recognition-czsfr