MultiFoodChat: Interactive Food Recognition

• MultiFoodChat is a dialogue-driven, multi-agent framework for intelligent food recognition that employs zero-shot generalization and multi-modal reasoning.
• It leverages an Object Perception Token mechanism with YOLOX-based detection to focus on salient food image regions, enhancing interpretability.
• The framework iteratively refines predictions through expert agent collaboration, achieving high accuracy on standard food recognition benchmarks.

MultiFoodChat refers to a dialogue-driven, multi-agent reasoning framework for intelligent food recognition and analysis, with a particular emphasis on zero-shot generalization, interpretability, and multi-modal reasoning. The system is conceptualized as an integration of state-of-the-art vision–LLMs (VLMs), LLMs, and interactive multi-agent protocols tailored for food image classification and food quality inspection. Unlike traditional approaches that require extensive supervised data or task-specific model retraining, MultiFoodChat employs collaborative dialogue among specialized agents to flexibly interpret complex food scenes, leveraging explicit object localization and iterative reasoning for improved accuracy and explainability (Hu et al., 14 Oct 2025).

1. System Architecture and Dialogue-Driven Reasoning

The MultiFoodChat architecture is characterized by the interaction between VLM-based visual analysis and LLM-driven multi-round dialogue. The core input structure is a composite triplet:

• $I$: the input image (e.g., a food photograph)
• $C$: coordinate information (bounding box outputs from a detection module)
• $L$: the current dialogue history

The image is first processed by a visual encoder (ViT-L/14 or equivalent), producing high-dimensional features that are later projected into the embedding space of the LLM. The framework then initiates a dialogue involving multiple specialized agents—such as a Food Scientist, Vision Analyst, and Decision Maker—each sequentially contributing domain-specific expertise. These agents iteratively reason over the visual and semantic cues received thus far, updating the dialogue state and ultimately synthesizing a final food category prediction.

Mathematically, the dialogue-driven reasoning process is represented as:

$T(y_n) = \Phi(H_v, \{H_q^{(t)}\}_{t=1}^n)$

where $H_v$ describes the aligned visual features, $\{H_q^{(t)}\}$ denotes the accumulation of query/response pairs up to round $n$, and $\Phi$ is the reasoning function realized by the LLM over the multi-modal feature set.

2. Object Perception Token and Localization

A key innovation is the Object Perception Token (OPT), derived through explicit visual localization of salient regions within food images. The OPT is produced by a YOLOX-based object detector that processes the image $I$ using a backbone network ($\mathcal{B}(I)$) and feature pyramid network, yielding detection outputs $Z = \mathcal{D}(F)$. Each detection consists of bounding box coordinates $(x_i, y_i, w_i, h_i)$ and class confidence $p_i$:

$z_i = [ (x_i, y_i, w_i, h_i), p_i ]$

Redundant detections are culled by non-maximum suppression:

$$C = \mathrm{NMS}(Z, \tau) = \{ (x_j, y_j, w_j, h_j) \}_{j=1}^n$$

These high-confidence regions become OPTs, which serve as explicit, localized visual anchors. The OPTs are then injected into the dialogue prompt stream, ensuring subsequent agent deliberations are spatially constrained to the most diagnostically relevant regions.

3. Interactive Reasoning Agent (IRA) and Multi-Agent Dialogue

The Interactive Reasoning Agent (IRA) module implements the multi-agent dialogue protocol with distinct expert roles:

• Food Scientist: Makes an initial, knowledge-driven hypothesis using the OPT and background nutrition taxonomy; provides an evidence-based rationale.
• Vision Analyst: Audits the hypothesis by examining detailed visual characteristics (texture, color, morphology), cross-verifies with the Food Scientist, and refines the candidate label.
• Decision Maker: Aggregates preceding agent outputs and resolves disagreements, synthesizing all available cues to produce the final food label.

Each agent’s response is a tuple of (prediction, rationale), for example,

$$(\hat{y}_{\text{food}}, r_{\text{food}}) = \mathrm{Agent}_{\text{food}}(I, C, L)$$

This agent-based, multi-turn dialogue allows for iterative hypothesis revision, disambiguation of visually similar food items, and explicit tracing of the model’s decision path, leading to improved interpretability in challenging scenarios.

4. Zero-Shot Recognition and Performance Evaluation

MultiFoodChat operates in a zero-shot learning regime, requiring no further model retraining or labeled supervision for new categories. The system was evaluated on several public benchmarks:

• Fruit-10 ($N=3374$) with 10 fruit categories
• Fruit and Vegetable Disease (FVD, $N=30,000$; 14 classes)
• Food11 ($N=16,643$; 11 food categories)
• Food101 ($N=101,000$; 101 categories)

Recognition accuracy on Fruit-10 reached 90.19% (c.f. 95.22% for MobileNetV2 supervised), and on Food101, MultiFoodChat achieved 87.70%, outperforming conventional unsupervised alternatives (e.g., DINO at 61.40%) (Hu et al., 14 Oct 2025). Ablation studies confirmed gains are attributable to each architectural module (OPT, IRA, and multi-turn dialogue).

5. Interpretability, Robustness, and Application Domains

By structuring the reasoning as an explicit, multi-agent dialogue, MultiFoodChat directly exposes the rationale behind classification decisions, addressing the “black box” problem prevalent in neural food recognition systems. Each agent’s textual output clarifies the role of domain expertise, visual evidence, and consensus building.

The iterative protocol also improves robustness in ambiguous or visually confounded cases (e.g., distinguishing between closely related fruit or composite dishes), with dynamic revisiting of prior beliefs leading to higher final classification fidelity.

Potential application domains articulated in the data include:

• Automated, training-free quality inspection in food production and retail supply chains
• Food safety and surveillance platforms to flag anomalous items or spoilage
• Dietary assessment/counseling and nutritional analysis tools in mobile health applications—especially where collection of task-specific training data is infeasible
• Research in food science via high-throughput, interpretable compositional analysis integrated with existing nutrient databases

6. Mathematical Formalism and Information Flow

The end-to-end information flow is specified by a series of LaTeX-formulated relationships:

• Visual encoding: $F = \mathcal{B}(I)$
• Detection: $Z = \mathcal{D}(F)$
• Object tokenization: $C = \mathrm{NMS}(Z, \tau)$
• Multi-agent dialogue: for each agent role $r$, output as

$$(\hat{y}_r, r_r) = \mathrm{Agent}_r(I, C, L, \ldots)$$

• Final category prediction: $y = \mathrm{Agent}_{\mathrm{decider}}(\ldots)$

Cumulative text and visual features at each turn are dynamically fused:

$T(y_n) = \Phi(H_v, \{H_q^{(t)}\}_{t=1}^n)$

ensuring that both current and historical reasoning context inform subsequent decisions.

7. Prospects and Position in the Research Landscape

MultiFoodChat embodies a shift from monolithic, retraining-dependent CNNs/transformers to highly interactive, agent-based multi-modal systems. Its zero-shot, dialogue-centered design is suited to dynamic environments and emergent food categories, as well as scenarios requiring transparent, evidence-traceable predictions. This distinguishes it from prior unsupervised and few-shot classification approaches by embedding human-like reasoning into the inference loop, with demonstrated performance gains and interpretability on standard benchmarks (Hu et al., 14 Oct 2025).

A plausible implication is that similar architectures could be extended beyond food quality inspection to other domains requiring fine-grained, explainable recognition (such as medical imaging or complex product monitoring), wherever task-specific ground truth may be scarce or continuously evolving.