PortionNet: Cross-Modal Food Nutrition Estimation

• PortionNet is a cross-modal knowledge distillation framework that transfers latent 3D geometric cues from point clouds to an RGB-only model for food nutrition estimation.
• It employs dual RGB encoders (ViT-B/16 and ResNet-18) with a lightweight RGB-to-Point-Cloud adapter and cross-attention fusion to predict food class, volume, and energy.
• Benchmark results on MetaFood3D and SimpleFood45 demonstrate significant reductions in volume and energy estimation errors compared to prior methods.

Searching arXiv for PortionNet and directly related food portion estimation work. PortionNet is a cross-modal knowledge distillation framework for food nutrition estimation from a single RGB image that is designed to recover some of the missing 3D reasoning that monocular images inherently lack. During training, it is allowed to observe point-cloud geometry extracted from 3D food data, but at test time it uses only a standard RGB image. In this way, it transfers 3D geometric knowledge into an RGB-only model and avoids the need for depth sensors or LiDAR-equipped phones at inference. The method targets the estimation of food portion size, volume, and energy, and its core mechanism is to teach an RGB network to imitate the latent geometric representation produced by a point-cloud encoder (Bright et al., 26 Dec 2025).

1. Problem setting and motivation

Food nutrition estimation from a single image is not only a semantic recognition problem but also a metric geometric reasoning problem. The central difficulty is that a monocular RGB image collapses 3D structure into a 2D projection, destroying explicit depth and scale information. Two visually similar plates can correspond to very different real-world volumes depending on camera distance, viewpoint, food shape, or pile height. Depth-based systems can reconstruct geometry more reliably, but they require specialized hardware unavailable on most consumer phones. RGB-only methods are therefore more deployable, but they generally perform poorly because absolute volume estimation is fundamentally underconstrained without 3D cues (Bright et al., 26 Dec 2025).

PortionNet is situated within a broader literature on image-based food portion estimation that repeatedly identifies 3D ambiguity from 2D images as the core bottleneck. A review of the area describes single-image portion estimation as mathematically ill-posed because real-world scale and shape cannot be uniquely recovered from pixels alone, and distinguishes portion estimation, volume estimation, mass estimation, and calorie estimation as related but distinct targets (Vinod et al., 4 Feb 2026). Earlier single-view work also emphasized that food portion and energy estimation are intrinsically difficult because appearance varies with preparation, arrangement, and occlusion, and proposed learning an image-to-energy mapping without explicit 3D reconstruction (Fang et al., 2018).

Within this landscape, PortionNet addresses a specific gap. Rather than requiring explicit depth at inference or bypassing geometry entirely through direct scalar regression, it learns a latent geometry-aware representation from point clouds during training and deploys that representation through RGB alone at test time (Bright et al., 26 Dec 2025).

2. Architectural design

PortionNet has four main components: dual RGB encoders, a PointNet-based geometry encoder that acts as the teacher during training, a lightweight RGB-to-point-cloud adapter, and a fusion and prediction module with heads for classification, volume, and energy (Bright et al., 26 Dec 2025).

For RGB feature extraction, the model uses two pretrained backbones in parallel: ViT-B/16 and ResNet-18. Their features are concatenated into a 1280-dimensional vector and then projected through a two-layer MLP into a 256-dimensional representation. The stated intent of this dual-encoder design is to combine transformer-style global and contextual features with CNN-style local texture and shape cues (Bright et al., 26 Dec 2025).

For geometric supervision, PortionNet uses a modified PointNet encoder operating on point clouds. Two modifications are specified. First, it appends a learned embedding of the object’s bounding-box dimensions, mapping $\mathbb{R}^3 \to \mathbb{R}^{64}$, to provide explicit scale information. Second, it applies multi-scale adaptive pooling at resolutions $\{64, 128, 256, 512\}$ before global aggregation. The final point-cloud representation is projected to a 256-dimensional feature vector so that it matches the RGB feature dimensionality (Bright et al., 26 Dec 2025).

The component that enables RGB-only deployment is the RGB-to-Point-Cloud Adapter. This is a lightweight two-layer MLP that maps the RGB feature vector into the geometry-aware latent space learned by PointNet. At inference, because no point cloud is available, the adapter serves as a stand-in for the missing 3D branch and produces a pseudo-3D feature from RGB alone (Bright et al., 26 Dec 2025).

After feature extraction, PortionNet performs cross-modal fusion using bidirectional cross-attention with residual connections and layer normalization. RGB and geometry features—either true PointNet features during multimodal training or the adapter’s pseudo-3D features during RGB-only mode and inference—are fused through attention, concatenated, and passed through a fusion MLP. The fused representation feeds three task heads:

1. a 3-layer MLP classifier for the food class label, with $C = 108$ classes on MetaFood3D;
2. a volume regression head using Softplus activation to ensure positive predictions;
3. an energy regression head using concatenated features to model the dependency between food appearance, geometry, and caloric content (Bright et al., 26 Dec 2025).

A concise summary of the architectural roles is given below.

Component	Function	Key specification
Dual RGB encoders	RGB feature extraction	ViT-B/16 + ResNet-18
Geometry teacher	Point-cloud encoding	Modified PointNet
RGB-to-PC adapter	Pseudo-3D feature synthesis	Two-layer MLP
Fusion and heads	Joint prediction	Cross-attention + classification, volume, energy

A common misconception is to treat PortionNet as a reconstruction model. It does not reconstruct an explicit 3D shape or depth map at test time. Instead, it learns a latent feature space in which RGB features become geometrically informed through teacher supervision (Bright et al., 26 Dec 2025).

3. Distillation mechanism and training objective

The method frames training as cross-modal feature distillation. Let $f_{\text{adapter}} \in \mathbb{R}^{256}$ denote the geometry-like feature produced by the RGB adapter and $f_{\text{PointNet}} \in \mathbb{R}^{256}$ denote the teacher feature produced from the real point cloud. The adapter is trained to mimic the teacher through magnitude, direction, and distribution matching, using the loss

$$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$

with weights $w_{\text{mse}} = 0.7$, $w_{\text{cos}} = 0.2$, and $w_{\text{KL}} = 0.1$ (Bright et al., 26 Dec 2025).

The three terms are

$$\mathcal{L}_{\text{MSE}} = \lVert f_{\text{adapter}} - f_{\text{PointNet}} \rVert_2^2$$

$\{64, 128, 256, 512\}$0

$\{64, 128, 256, 512\}$1

where $\{64, 128, 256, 512\}$2 denotes softmax and $\{64, 128, 256, 512\}$3 is the temperature (Bright et al., 26 Dec 2025).

Training alternates between two modes. In multimodal mode, the model uses both RGB features and true PointNet geometry features. In RGB-only mode, it uses RGB features together with adapter-generated pseudo-geometry. The paper uses 70% multimodal batches and 30% RGB-only batches, corresponding to $\{64, 128, 256, 512\}$4 as the RGB-only training ratio. The stated rationale is that training only with true point-cloud features would encourage over-reliance on information unavailable at test time, whereas training too frequently without real geometry would weaken the distillation target (Bright et al., 26 Dec 2025).

The full supervision combines food category, volume, and energy labels with the distillation term:

$\{64, 128, 256, 512\}$5

The classification loss is cross-entropy with label smoothing $\{64, 128, 256, 512\}$6:

$\{64, 128, 256, 512\}$7

where $\{64, 128, 256, 512\}$8, $\{64, 128, 256, 512\}$9 is the smoothed target, and $C = 108$0 is the predicted probability (Bright et al., 26 Dec 2025).

For regression, the model jointly supervises volume and energy. Let $C = 108$1 be ground-truth and predicted volume, and $C = 108$2 be ground-truth and predicted energy. The loss is

$C = 108$3

with $C = 108$4 (Bright et al., 26 Dec 2025).

Energy receives higher weight than volume, and the paper states that $C = 108$5 is batch-normalized for scale invariance. In multimodal mode, the authors additionally use GradNorm for balancing losses, with $C = 108$6 and $C = 108$7 (Bright et al., 26 Dec 2025).

This training strategy is what the paper calls pseudo-3D reasoning: the model learns how true 3D geometry corresponds to RGB appearance during training, then uses the adapter feature as a compact internal 3D prior at inference (Bright et al., 26 Dec 2025).

4. Inference pipeline and benchmark results

The inference pipeline is fully RGB-only. A single food image is passed through ViT-B/16 and ResNet-18, their features are fused and projected, the adapter generates a pseudo-point-cloud embedding, and the RGB plus pseudo-geometry features are fused via cross-attention and sent to the classification, volume, and energy heads. No depth map, point cloud, LiDAR, or calibrated sensor is required (Bright et al., 26 Dec 2025).

The primary benchmark is MetaFood3D, which contains 637 objects across 108 classes, with 12,752 RGB images and point clouds. The paper follows the 80/20 split used by MFP3D. It also evaluates cross-dataset transfer on SimpleFood45, which contains 513 images across 12 classes. Results are averaged over 3 seeds with standard deviations, except some ablations that use a single seed with tuned hyperparameters (Bright et al., 26 Dec 2025).

On MetaFood3D, PortionNet reports nearly identical performance in RGB-only and RGB+point-cloud modes.

Setting	Representative metrics	Result
RGB-only	$C = 108$8 / Accuracy	$C = 108$9 / $f_{\text{adapter}} \in \mathbb{R}^{256}$0
RGB-only	Volume MAE / Energy MAE	$f_{\text{adapter}} \in \mathbb{R}^{256}$1 mL / $f_{\text{adapter}} \in \mathbb{R}^{256}$2 kcal
RGB-only	Volume MAPE / Energy MAPE	$f_{\text{adapter}} \in \mathbb{R}^{256}$3 / $f_{\text{adapter}} \in \mathbb{R}^{256}$4
RGB+PC	$f_{\text{adapter}} \in \mathbb{R}^{256}$5 / Accuracy	$f_{\text{adapter}} \in \mathbb{R}^{256}$6 / $f_{\text{adapter}} \in \mathbb{R}^{256}$7
RGB+PC	Volume MAE / Energy MAE	$f_{\text{adapter}} \in \mathbb{R}^{256}$8 mL / $f_{\text{adapter}} \in \mathbb{R}^{256}$9 kcal
RGB+PC	Volume MAPE / Energy MAPE	$f_{\text{PointNet}} \in \mathbb{R}^{256}$0 / $f_{\text{PointNet}} \in \mathbb{R}^{256}$1

The paper identifies this small gap between RGB-only and RGB+PC operation as evidence that the adapter successfully substitutes for the true point-cloud branch (Bright et al., 26 Dec 2025).

Compared with reported prior methods on MetaFood3D, PortionNet is presented as outperforming all baselines overall. The reported values are: Density Map with energy MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$2; Stereo with volume MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$3; Voxel with volume MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$4; 3D Assisted with volume MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$5 and energy MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$6; MFP3D with volume MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$7 and energy MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$8; and PortionNet with volume MAPE $f_{\text{PointNet}} \in \mathbb{R}^{256}$9 and energy MAPE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$0 (Bright et al., 26 Dec 2025).

Relative to MFP3D, the paper reports a reduction from 41.43% to 17.43% in volume MAPE, corresponding to a 57.9% relative improvement, and from 68.05% to 15.36% in energy MAPE, corresponding to a 77.4% relative improvement (Bright et al., 26 Dec 2025).

5. Generalization, ablations, and optimization behavior

The cross-dataset experiment on SimpleFood45 provides a more differentiated picture. PortionNet reports volume MAPE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$1 and energy MAPE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$2. The energy result is described as state of the art and substantially better than MFP3D’s 24.03%, a 49.4% relative improvement. However, volume generalization is weaker than some prior methods: MFP3D reports 16.15% and 3D Assisted reports 14.01%, whereas PortionNet obtains 23.51% (Bright et al., 26 Dec 2025).

The authors explicitly discuss this asymmetry: energy estimation transfers better across datasets because it relies more on food composition and appearance features, while volume depends on absolute scale and camera parameters, which shift across datasets (Bright et al., 26 Dec 2025). This is one of the central empirical nuances of the method.

Ablations clarify that the gains are not attributable merely to adding a point-cloud branch. A baseline without adapter or distillation, described as “dual encoders + PointNet with concatenation,” gives Volume MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$3 mL and Energy MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$4 kcal. PortionNet improves this to Volume MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$5 and Energy MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$6 in RGB mode, and Volume MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$7 and Energy MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$8 in RGB+PC mode. The paper interprets this as roughly 25.5% improvement in volume and 20.2% in energy over the baseline (Bright et al., 26 Dec 2025).

A more targeted ablation compares direct RGB use against the adapter. No Adapter (direct RGB) yields Volume MAE $$\mathcal{L}_{\text{distill}} = w_{\text{mse}}\mathcal{L}_{\text{MSE}} + w_{\text{cos}}\mathcal{L}_{\text{cos}} + w_{\text{KL}}\mathcal{L}_{\text{KL}}$$9 and Energy MAE $w_{\text{mse}} = 0.7$0, whereas With Adapter (PortionNet) yields Volume MAE $w_{\text{mse}} = 0.7$1 and Energy MAE $w_{\text{mse}} = 0.7$2 (Bright et al., 26 Dec 2025). The paper also reports that removing distillation by setting $w_{\text{mse}} = 0.7$3 causes 33.2% and 28.4% increases in energy and volume MAE, respectively, indicating that the adapter does not learn useful pseudo-geometry without explicit supervision (Bright et al., 26 Dec 2025).

The dual-mode training ratio $w_{\text{mse}} = 0.7$4 is also ablated:

Setting	Volume MAE	Energy MAE
$w_{\text{mse}} = 0.7$5	27.57	34.82
$w_{\text{mse}} = 0.7$6	21.74	29.03
$w_{\text{mse}} = 0.7$7	27.13	39.69

These results support the stated rationale for alternating multimodal and RGB-only training (Bright et al., 26 Dec 2025).

The distillation weight is similarly sensitive:

Setting	Volume MAE	Energy MAE
$w_{\text{mse}} = 0.7$8	37.07	37.00
$w_{\text{mse}} = 0.7$9	21.74	29.03
$w_{\text{cos}} = 0.2$0	23.62	36.32

The paper interprets this as evidence that moderate feature alignment is preferable to either no alignment or over-constrained imitation (Bright et al., 26 Dec 2025).

Optimization details are conventional but clearly specified: AdamW with learning rate $w_{\text{cos}} = 0.2$1 for pretrained encoders and $w_{\text{cos}} = 0.2$2 for heads and adapter; OneCycleLR with 10% warmup over 25 epochs; effective batch size 64, realized as $w_{\text{cos}} = 0.2$3 gradient accumulation; cross-modal attention with 8 heads; and gradient clipping to norm 1.0 (Bright et al., 26 Dec 2025).

6. Interpretation, limitations, and relation to adjacent approaches

PortionNet’s principal contribution is to show that point-cloud geometry can be distilled into an RGB-only food nutrition estimator through a teacher-student style design in which a PointNet teacher supervises a lightweight RGB-to-geometry adapter (Bright et al., 26 Dec 2025). A plausible implication is that the method occupies an intermediate position between explicit geometry pipelines and purely appearance-based regressors: it uses 3D information during training but not as an inference-time input.

This distinguishes it from earlier single-view energy estimation approaches that avoid explicit 3D reasoning altogether. For example, a 2018 method proposed learning an image-to-energy mapping through a dense “energy distribution” image constructed from segmentation masks and dietitian-provided labels, and reported an average energy estimation error rate of $w_{\text{cos}} = 0.2$4 using a conditional GAN with $w_{\text{cos}} = 0.2$5 loss (Fang et al., 2018). That line of work is relevant because it demonstrated that calorie estimation can be learned directly from image appearance, but its target is not a physical portion volume or a geometry-aware latent representation.

PortionNet also aligns with a broader trend identified in a 2026 review: hybrid systems that combine learned monocular priors with geometric cues are increasingly treated as a practical compromise between deployability and physical grounding (Vinod et al., 4 Feb 2026). The review describes monocular methods as the most usable but still limited by scale ambiguity and hidden geometry, which is consistent with PortionNet’s cross-dataset behavior on volume estimation.

The limitations highlighted in the paper are correspondingly specific. First, the model shows cross-device and cross-dataset volume estimation sensitivity: even with distilled geometric priors, monocular volume remains dependent on camera intrinsics, viewpoint, and scale cues that may not transfer across datasets (Bright et al., 26 Dec 2025). Second, although the adapter itself is lightweight, the full RGB inference stack uses both ViT-B/16 and ResNet-18, so the system is not demonstrated to be ultra-tiny or mobile-real-time. The paper does not provide detailed FLOPs, latency, or mobile deployment benchmarks, and thus removes the hardware requirement of the depth sensor rather than proving low compute cost (Bright et al., 26 Dec 2025). Third, the training procedure still requires paired RGB and point-cloud data, which may be difficult to collect at scale (Bright et al., 26 Dec 2025).

The most important clarification is therefore negative: PortionNet does not solve monocular nutrition estimation by reconstructing explicit 3D geometry at test time, and it does not eliminate scale ambiguity in a general sense. Instead, it learns a geometrically informed latent prior that substantially improves monocular food volume and calorie estimation under the evaluated conditions (Bright et al., 26 Dec 2025).