Papers
Topics
Authors
Recent
Search
2000 character limit reached

MetaFood3D: 3D Food Dataset with Nutrition Values

Published 3 Sep 2024 in cs.CV | (2409.01966v2)

Abstract: Food computing is both important and challenging in computer vision (CV). It significantly contributes to the development of CV algorithms due to its frequent presence in datasets across various applications, ranging from classification and instance segmentation to 3D reconstruction. The polymorphic shapes and textures of food, coupled with high variation in forms and vast multimodal information, including language descriptions and nutritional data, make food computing a complex and demanding task for modern CV algorithms. 3D food modeling is a new frontier for addressing food related problems, due to its inherent capability to deal with random camera views and its straightforward representation for calculating food portion size. However, the primary hurdle in the development of algorithms for food object analysis is the lack of nutrition values in existing 3D datasets. Moreover, in the broader field of 3D research, there is a critical need for domain-specific test datasets. To bridge the gap between general 3D vision and food computing research, we introduce MetaFood3D. This dataset consists of 743 meticulously scanned and labeled 3D food objects across 131 categories, featuring detailed nutrition information, weight, and food codes linked to a comprehensive nutrition database. Our MetaFood3D dataset emphasizes intra-class diversity and includes rich modalities such as textured mesh files, RGB-D videos, and segmentation masks. Experimental results demonstrate our dataset's strong capabilities in enhancing food portion estimation algorithms, highlight the gap between video captures and 3D scanned data, and showcase the strengths of MetaFood3D in generating synthetic eating occasion data and 3D food objects.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Objectron: A large scale dataset of object-centric videos in the wild with pose annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7822–7831, 2021.
  2. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  3. Parametric correspondence and chamfer matching: Two new techniques for image matching. In International Joint Conference on Artificial Intelligence, 1977.
  4. Food-101–mining discriminative components with random forests. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 446–461. Springer, 2014.
  5. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference on advanced robotics (ICAR), pages 510–517. IEEE, 2015.
  6. Shapenet: An information-rich 3D model repository. arXiv preprint arXiv:1512.03012, 2015.
  7. Text2tex: Text-driven texture synthesis via diffusion models. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 18512–18522, Los Alamitos, CA, USA, Oct 2023.
  8. Putting the object back into video object segmentation. arXiv preprint arXiv:2310.12982, 2023.
  9. Two-view 3d reconstruction for food volume estimation. IEEE Transactions on Multimedia, 19(5):1090–1099, 2017.
  10. Objaverse: A universe of annotated 3D objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023.
  11. A comparison of food portion size estimation using geometric models and depth images. Proceedings of the 2016 IEEE International Conference on Image Processing, pages 26–30, 2016.
  12. Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items, 2022.
  13. Dynamic mixup for multi-label long-tailed food ingredient recognition. IEEE Transactions on Multimedia, 25:4764–4773, 2022.
  14. Get3D: A generative model of high quality 3D textured shapes learned from images. Proceedings of Advances In Neural Information Processing Systems, 35:31841–31854, 2022.
  15. Metagraspnetv2: All-in-one dataset enabling fast and reliable robotic bin picking via object relationship reasoning and dexterous grasping. IEEE Transactions on Automation Science and Engineering, pages 1–19, 2023.
  16. Revisiting point cloud shape classification with a simple and effective baseline. In International Conference on Machine Learning, pages 3809–3820. PMLR, 2021.
  17. Single image-based food volume estimation using monocular depth-prediction networks. In Universal Access in Human-Computer Interaction. Applications and Practice: 14th International Conference, UAHCI 2020, Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II 22, pages 532–543. Springer, 2020.
  18. Long-Tailed Food Classification. Nutrients, 15(12):2751, June 2023.
  19. Long-tailed food classification. Nutrients, 15(12):2751, 2023.
  20. An end-to-end food image analysis system. Electronic Imaging, 2021(8):285–1, 2021.
  21. Multi-task image-based dietary assessment for food recognition and portion size estimation. 2020 IEEE Conference on Multimedia Information Processing and Retrieval, pages 49–54, 2020.
  22. Online continual learning for visual food classification. Proceedings of the IEEE/CVF international conference on computer vision, pages 2337–2346, 2021.
  23. Food image analysis: Segmentation, identification and weight estimation. In 2013 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2013.
  24. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
  25. 2D gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403.17888, 2024.
  26. Accuracy of food portion size estimation from digital pictures acquired by a chest-worn camera. Public health nutrition, 17(8):1671–1681, 2014.
  27. Estimating amount of food in a circular dining bowl from a single image. In Proceedings of the 8th International Workshop on Multimedia Assisted Dietary Management, pages 1–9, 2023.
  28. Multi-scale multi-view deep feature aggregation for food recognition. IEEE Transactions on Image Processing, 29:265–276, 2019.
  29. 3D gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1–14, 2023.
  30. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015–4026, 2023.
  31. A novel approach to estimate the weight of food items based on features extracted from an image using boosting algorithms. Scientific Reports, 13(1):21040, 2023.
  32. A review of image-based food recognition and volume estimation artificial intelligence systems. IEEE Reviews in Biomedical Engineering, 17:136–152, 2024.
  33. 3d reconstruction and volume estimation of food using stereo vision techniques. In 2021 IEEE 21st International Conference on Bioinformatics and Bioengineering (BIBE), pages 1–4, 2021.
  34. Foodsam: Any food segmentation. IEEE Transactions on Multimedia, 2023.
  35. Neuralangelo: High-fidelity neural surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  36. Computer vision-based food calorie estimation: dataset, method, and experiment. arXiv preprint arXiv:1705.07632, 2017.
  37. Magic3D: High-resolution text-to-3D content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  38. Differences in dietary intake exist among us adults by diabetic status using nhanes 2009–2016. Nutrients, 14(16):3284, 2022.
  39. A comprehensive survey on 3d content generation. arXiv preprint arXiv:2402.01166, 2024.
  40. Akb-48: A real-world articulated object knowledge base. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14789–14798, Los Alamitos, CA, USA, June 2022.
  41. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
  42. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023.
  43. Depth estimation based on a single close-up image with volumetric annotations in the wild: A pilot study. Proceedings of the 2019 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pages 513–518, 2019.
  44. Image-based food classification and volume estimation for dietary assessment: A review. IEEE Journal of Biomedical and Health Informatics, 24(7):1926–1939, 2020.
  45. Wonder3D: Single image to 3D using cross-domain diffusion. arXiv preprint arXiv:2310.15008, 2023.
  46. An improved encoder-decoder framework for food energy estimation. Proceedings of the 8th International Workshop on Multimedia Assisted Dietary Management, page 53–59, 2023.
  47. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.07123, 2022.
  48. Visual aware hierarchy based food recognition. In International conference on pattern recognition, pages 571–598. Springer, 2021.
  49. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  50. Large scale visual food recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):9932–9949, 2023.
  51. Usda food and nutrient database for dietary studies (fndds), 5.0. Procedia Food Science, 2:99–112, 2013. 36th National Nutrient Databank Conference.
  52. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, July 2022.
  53. Extracting Triangular 3D Models, Materials, and Lighting From Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8280–8290, June 2022.
  54. Nutritionverse-synth: An open access synthetically generated 2d food scene dataset for dietary intake estimation. arXiv preprint arXiv:2312.06192, 2023.
  55. Personalized food image classification: Benchmark datasets and new baseline. 2023 57th Asilomar Conference on Signals, Systems, and Computers, pages 1095–1099, 2023.
  56. Fmifood: Multi-modal contrastive learning for food image classification. arXiv preprint arXiv:2408.03922, 2024.
  57. Recognition and volume estimation of food intake using a mobile device. Proceedings of the 2009 Workshop on Applications of Computer Vision, pages 1–8, 2009.
  58. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  59. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
  60. Magic123: One image to high-quality 3D object generation using both 2D and 3D diffusion priors. arXiv preprint arXiv:2306.17843, 2023.
  61. Online class-incremental learning for real-world food image classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8195–8204, January 2024.
  62. Common objects in 3D: Large-scale learning and evaluation of real-life 3D category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901–10911, 2021.
  63. Benchmarking and analyzing point cloud classification under corruptions. In International Conference on Machine Learning, pages 18559–18575. PMLR, 2022.
  64. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  65. Multi-task learning for calorie prediction on a novel large-scale recipe dataset enriched with nutritional information. In International Conference on Pattern Recognition (ICPR), pages 4001–4008. IEEE, 2021.
  66. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  67. Towards learning food portion from monocular images with cross-domain feature adaptation. Proceedings of 2021 IEEE 23rd International Workshop on Multimedia Signal Processing, pages 1–6, 2021.
  68. An end-to-end food portion estimation framework based on shape reconstruction from monocular image. Proceedings of 2023 IEEE International Conference on Multimedia and Expo, pages 942–947, 2023.
  69. Mvdream: Multi-view diffusion for 3D generation. arXiv preprint arXiv:2308.16512, 2023.
  70. Robustpointset: A dataset for benchmarking robustness of point cloud classifiers. arXiv preprint arXiv:2011.11572, 2020.
  71. A comprehensive survey of image-based food recognition and volume estimation methods for dietary assessment. Healthcare, 9:1676, 2021.
  72. Foodverse: A dataset of 3d food models for nutritional intake estimation. Journal of Computational Vision and Imaging Systems, 8(1):23–26, 2022.
  73. Nutritionverse-3D: A 3D food model dataset for nutritional intake estimation. arXiv preprint arXiv:2304.05619, 2023.
  74. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  75. Nutrition5k: Towards automatic nutritional understanding of generic food. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8903–8911, June 2021.
  76. Food portion estimation via 3d object scaling. arXiv preprint arXiv:2404.12257, 2024.
  77. Image based food energy estimation with depth domain adaptation. Proceedings of 2022 IEEE 5th International Conference on Multimedia Information Processing and Retrieval, pages 262–267, 2022.
  78. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689, 2021.
  79. A review on vision-based analysis for automatic dietary assessment. Trends in Food Science & Technology, 122:223–237, April 2022.
  80. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog), 38(5):1–12, 2019.
  81. Omniobject3D: Large-vocabulary 3D object dataset for realistic perception, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803–814, 2023.
  82. Walk in the cloud: Learning curves for point clouds shape analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 915–924, 2021.
  83. Model-based food volume estimation using 3d pose. Proceedings of the 2013 IEEE International Conference on Image Processing, pages 2534–2538, 2013.
  84. Image enhancement and quality measures for dietary assessment using mobile devices. Computational Imaging X, 8296:153–162, 2012.
  85. Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173–3182, 2021.
  86. Learning geometry-disentangled representation for complementary understanding of 3D object point cloud. Proceedings of the AAAI conference on artificial intelligence, 35:3056–3064, 2021.
  87. Human-Mimetic Estimation of Food Volume from a Single-View RGB Image Using an AI System. Electronics, 10(13):1556, June 2021.
  88. Point-bert: Pre-training 3D point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19313–19322, 2022.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up