Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Double-Bounded Optimal Transport for Advanced Clustering and Classification (2401.11418v1)

Published 21 Jan 2024 in cs.LG, cs.AI, and math.OC

Abstract: Optimal transport (OT) is attracting increasing attention in machine learning. It aims to transport a source distribution to a target one at minimal cost. In its vanilla form, the source and target distributions are predetermined, which contracts to the real-world case involving undetermined targets. In this paper, we propose Doubly Bounded Optimal Transport (DB-OT), which assumes that the target distribution is restricted within two boundaries instead of a fixed one, thus giving more freedom for the transport to find solutions. Based on the entropic regularization of DB-OT, three scaling-based algorithms are devised for calculating the optimal solution. We also show that our DB-OT is helpful for barycenter-based clustering, which can avoid the excessive concentration of samples in a single cluster. Then we further develop DB-OT techniques for long-tailed classification which is an emerging and open problem. We first propose a connection between OT and classification, that is, in the classification task, training involves optimizing the Inverse OT to learn the representations, while testing involves optimizing the OT for predictions. With this OT perspective, we first apply DB-OT to improve the loss, and the Balanced Softmax is shown as a special case. Then we apply DB-OT for inference in the testing process. Even with vanilla Softmax trained features, our extensive experimental results show that our method can achieve good results with our improved inference scheme in the testing stage.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. Barycenters in the Wasserstein space. SIAM Journal on Mathematical Analysis, 43(2): 904–924.
  2. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9(8): 1295.
  3. Wasserstein GAN. ICML.
  4. Sliced optimal partial transport. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13681–13690.
  5. Iterative Bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 37(2): A1111–A1138.
  6. Bregman, L. M. 1967. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3): 200–217.
  7. Free boundaries in optimal transport and Monge-Ampere obstacle problems. Annals of mathematics, 673–730.
  8. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems, 32.
  9. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33: 9912–9924.
  10. Partial optimal tranport with applications on positive-unlabeled learning. Advances in Neural Information Processing Systems, 33: 2903–2913.
  11. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
  12. Discrete Probabilistic Inverse Optimal Transport. In International Conference on Machine Learning, 3925–3946. PMLR.
  13. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Computation, 87(314): 2563–2609.
  14. Spherical optimal transportation. Computer-Aided Design, 115: 181–193.
  15. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9268–9277.
  16. Cuturi, M. 2013. Sinkhorn distances: Lightspeed computation of optimal transportation distances. arXiv preprint arXiv:1306.0895.
  17. Figalli, A. 2010. The optimal partial transport problem. Archive for rational mechanics and analysis, 195(2): 533–560.
  18. Improved Training of Wasserstein GANs. In NIPS.
  19. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9): 1263–1284.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778.
  21. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33: 22118–22133.
  22. Otlda: A geometry-aware optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 33: 18573–18582.
  23. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.09217.
  24. Kantorovich, L. 1942. On the transfer of masses (in Russian). In Doklady Akademii Nauk, volume 37, 227–229.
  25. Supervised contrastive learning. Advances in neural information processing systems, 33: 18661–18673.
  26. M2m: Imbalanced classification via major-to-minor translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 13896–13905.
  27. Learning Multiple Layers of Features from Tiny Images. Technical report, Citeseer.
  28. Léonard, C. 2012. From the Schrödinger problem to the Monge–Kantorovich problem. Journal of Functional Analysis, 262(4): 1879–1920.
  29. Learning to match via inverse optimal transport. Journal of machine learning research, 20.
  30. Improving generative adversarial networks via adversarial learning in latent space. Advances in Neural Information Processing Systems, 35: 8868–8881.
  31. Optimal entropy-transport problems and a new Hellinger–Kantorovich distance between positive measures. Inventiones mathematicae, 211(3): 969–1117.
  32. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, 2980–2988.
  33. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2537–2546.
  34. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314.
  35. Influence-balanced loss for imbalanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 735–744.
  36. Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems, 33: 4175–4186.
  37. Graph matching via optimal transport. arXiv preprint arXiv:2111.05366.
  38. Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective. in ICML.
  39. Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification. In Thirty-seventh Conference on Neural Information Processing Systems.
  40. Inverse optimal transport. SIAM Journal on Applied Mathematics, 80(1): 599–619.
  41. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11662–11671.
  42. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2495–2504.
  43. An improved K-Means clustering algorithm. In 2011 IEEE 3rd international conference on communication software and networks, 44–46. IEEE.
  44. Learning Combinatorial Embedding Networks for Deep Graph Matching. In ICCV, 3056–3065.
  45. Neural graph matching network: Learning lawler’s quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  46. A linear optimal transportation framework for quantifying and visualizing variations in sets of images. International journal of computer vision, 101(2): 254–269.
  47. Wilson, A. G. 1969. The use of entropy maximising models, in the theory of trip distribution, mode split and route split. Journal of transport economics and policy, 108–126.
  48. Adversarial robustness under long-tailed distribution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8659–8668.
  49. Gromov-wasserstein learning for graph matching and node embedding. In International conference on machine learning, 6932–6941. PMLR.
  50. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596.
  51. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now