Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
117 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Credibility-Aware Multi-Modal Fusion Using Probabilistic Circuits (2403.03281v2)

Published 5 Mar 2024 in cs.LG and cs.AI

Abstract: We consider the problem of late multi-modal fusion for discriminative learning. Motivated by noisy, multi-source domains that require understanding the reliability of each data source, we explore the notion of credibility in the context of multi-modal fusion. We propose a combination function that uses probabilistic circuits (PCs) to combine predictive distributions over individual modalities. We also define a probabilistic measure to evaluate the credibility of each modality via inference queries over the PC. Our experimental evaluation demonstrates that our fusion method can reliably infer credibility while maintaining competitive performance with the state-of-the-art.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Learning the structure of sum-product networks via an svd-based algorithm. In Conference on Uncertainty in Artificial Intelligence, 2015.
  2. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018.
  3. Neural network approaches versus statistical methods in classification of multisource remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 28(4):540–552, 1990. 10.1109/TGRS.1990.572944.
  4. Urref reliability versus credibility in information fusion (stanag 2511). In Proceedings of the 16th International Conference on Information Fusion, pages 1600–1607. IEEE, 2013.
  5. Urref self-confidence in information fusion trust. In 17th International Conference on Information Fusion (FUSION), pages 1–8. IEEE, 2014.
  6. Lecture notes: Probabilistic circuits: Representation and inference. February 2020.
  7. Strudel: Learning structured-decomposable probabilistic circuits. In Proceedings of the 10th International Conference on Probabilistic Graphical Models, volume 138, pages 137–148. PMLR, 23–25 Sep 2020.
  8. Uncertainty representation and evaluation for modeling and decision-making in information fusion. Journal for Advances in Information Fusion, 13(2):198–215, 2018.
  9. Assessing sensor reliability for multisensor data fusion within the transferable belief model. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1):782–787, 2004a. 10.1109/TSMCB.2003.817056.
  10. Assessing sensor reliability for multisensor data fusion within the transferable belief model. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(1):782–787, 2004b.
  11. Presentation and description of two classification methods using data fusion based on sensor management. Inf. Fusion, 2:49–71, 2001.
  12. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEEE 23rd international conference on information fusion (FUSION), pages 1–6. IEEE, 2020.
  13. Learning the structure of sum-product networks. In International conference on machine learning, pages 873–880. PMLR, 2013.
  14. Multiple classifier systems for the classification of audio-visual emotional states. In Affective Computing and Intelligent Interaction, pages 359–368. Springer, 2011.
  15. Trusted multi-view classification. In International Conference on Learning Representations, 2021.
  16. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2015.
  17. Mmtm: Multimodal transfer module for cnn fusion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13286–13296, 2019.
  18. Multisensor data fusion: A review of the state-of-the-art. Information fusion, 14(1):28–44, 2013.
  19. Multimodal machine learning in precision health: A scoping review. npj Digital Medicine, 5(1):171, 2022.
  20. Deepproblog: Neural probabilistic logic programming. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  21. Fast and accurate density estimation with extremely randomized cutset networks. In ECML/PKDD (1), volume 10534 of Lecture Notes in Computer Science, pages 203–219. Springer, 2017.
  22. Learning first-order probabilistic models with combining rules. In Proceedings of the 22nd international conference on Machine learning, pages 609–616, 2005.
  23. Vincent Nimier. Supervised multisensor tracking algorithm. In 9th European Signal Processing Conference (EUSIPCO 1998), pages 1–4, 1998.
  24. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In ICML, 2020a.
  25. Random sum-product networks: A simple and effective approach to probabilistic deep learning. In UAI, 2020b.
  26. Mfas: Multimodal fusion architecture search. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6959–6968, 2019.
  27. Sum-product networks: A new deep architecture. In UAI, 2011.
  28. Modeling latent discriminative dynamic of multi-dimensional affective signals. In Affective Computing and Intelligent Interaction: Fourth International Conference, ACII 2011, Memphis, TN, USA, October 9–12, 2011, Proceedings, Part II, pages 396–406. Springer, 2011.
  29. G Rogova and Jyotsna Kasturi. Reinforcement learning neural network for distributed decision making. In Proc. of the Forth Conf. on Information Fusion, 2001.
  30. Reliability in information fusion: literature survey. In Proceedings of the seventh international conference on information fusion, volume 2, pages 1158–1165, 2004.
  31. Learning sum-product networks with direct and indirect variable interactions. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pages 710–718, Bejing, China, 22–24 Jun 2014. PMLR.
  32. Multimodal multi-task financial risk forecasting. In Proceedings of the 28th ACM international conference on multimedia, pages 456–465, 2020.
  33. Aggregating predictions vs. aggregating features for relational classification. In 2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM), pages 121–128. IEEE, 2014.
  34. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 160–170, 2016.
  35. Probabilistic flow circuits: Towards unified deep models for tractable probabilistic inference. In The 39th Conference on Uncertainty in Artificial Intelligence, 2023.
  36. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision, 2012.
  37. Sun rgb-d: A rgb-d scene understanding benchmark suite. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567–576, 2015.
  38. Uno: Uncertainty-aware noisy-or multimodal fusion for unanticipated input degradation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5716–5723. IEEE, 2020.
  39. Centralnet: a multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0–0, 2018.
  40. The caltech-ucsd birds-200-2011 dataset, 2011.
  41. Credibility models for multi-source fusion. In 2006 9th International Conference on Information Fusion, pages 1–7. IEEE, 2006.
  42. Cpm-nets: Cross partial multi-view networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
  43. Provable dynamic fusion for low-quality multimodal data. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now