Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Autoencoder-based General Purpose Representation Learning for Customer Embedding (2402.18164v1)

Published 28 Feb 2024 in cs.LG and cs.AI

Abstract: In recent years, exploiting the domain-specific underlying structure of data and its generative factors for representation learning has shown success in various use-case agnostic applications. However, the diversity and complexity of tabular data have made it challenging to represent these structures in a latent space through multi-dimensional vectors. We design an autoencoder-based framework for building general purpose embeddings, we assess the performance of different autoencoder architectures, and show simpler models outperform complex ones in embedding highly complex tabular data. We apply our framework to produce plug-and-play, rich, and anonymized embeddings representing AWS customers for usage in any model, saving up to 45% of development time, and observe significant improvements in downstream models. Moreover, we propose a significant improvement to the calculation of reconstruction loss for multi-layer contractive autoencoders (CAE) by calculating the Jacobian of the entire encoder leading to a 15% improvement in reconstruction quality when compared to a stacked CAE.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. A deep contractive autoencoder for solving multiclass classification problems. Evolutionary Intelligence, 14:1619–1633, 2021.
  2. Robust variational autoencoder for tabular data with beta divergence. CoRR, abs/2006.08204, 2020. URL https://arxiv.org/abs/2006.08204.
  3. Seeing 3d chairs: exemplar part-based 2d-3d alignment using a large dataset of cad models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3762–3769, 2014.
  4. Neural networks and principal component analysis: Learning from examples without local minima. Neural Networks, 2(1):53–58, 1989. ISSN 0893-6080. doi: https://doi.org/10.1016/0893-6080(89)90014-2. URL https://www.sciencedirect.com/science/article/pii/0893608089900142.
  5. Understanding disentangling in β𝛽\betaitalic_β-vae, 2018.
  6. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020.
  7. Customer lifetime value prediction using embeddings. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pp.  1753–1762, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/3097983.3098123. URL https://doi.org/10.1145/3097983.3098123.
  8. Dynamic customer embeddings for financial service applications, 2021.
  9. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019.
  10. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  11. Dupont, E. Learning disentangled joint continuous and discrete representations. In Advances in Neural Information Processing Systems, pp. 707–717, 2018.
  12. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.
  13. The dimensional fact model: A conceptual model for data warehouses. International Journal of Cooperative Information Systems, 7(02n03):215–247, 1998.
  14. On embeddings for numerical features in tabular deep learning. Advances in Neural Information Processing Systems, 35:24991–25004, 2022.
  15. On embeddings for numerical features in tabular deep learning, 2023.
  16. Deep residual learning for image recognition, 2015.
  17. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  18. Early visual concept learning with unsupervised deep learning, 2016.
  19. Tabtransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
  20. Auto-encoding variational bayes, 2013.
  21. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  22. dsprites: Disentanglement testing sprites dataset, 2017.
  23. Umap: Uniform manifold approximation and projection for dimension reduction, 2020.
  24. Disentangling generative factors in natural language with discrete variational autoencoders, 2021.
  25. Efficient estimation of word representations in vector space, 2013.
  26. Munson, M. A. A study on the importance of and time spent on different modeling steps. SIGKDD Explor. Newsl., 13(2):65–71, may 2012. ISSN 1931-0145. doi: 10.1145/2207243.2207253. URL https://doi.org/10.1145/2207243.2207253.
  27. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018.
  28. OpenAI. Gpt-4 technical report, 2023.
  29. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  30. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp.  1532–1543, 2014.
  31. Press, G. Data preparation: Most time-consuming, least enjoyable data science task, survey says. Forbes, Mar 2016. URL https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/.
  32. Efficient learning of sparse representations with an energy-based model. In Schölkopf, B., Platt, J., and Hoffman, T. (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper_files/paper/2006/file/87f4d79e36d68c3031ccf6c55e9bbd39-Paper.pdf.
  33. Higher order contractive auto-encoder. In Gunopulos, D., Hofmann, T., Malerba, D., and Vazirgiannis, M. (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 645–660, Berlin, Heidelberg, 2011a. Springer Berlin Heidelberg.
  34. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th international conference on international conference on machine learning, pp.  833–840, 2011b.
  35. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181–1191, 2020. ISSN 0169-2070. doi: https://doi.org/10.1016/j.ijforecast.2019.07.001. URL https://www.sciencedirect.com/science/article/pii/S0169207019301888.
  36. Syne Tune: A library for large scale hyperparameter tuning and reproducible research. In International Conference on Automated Machine Learning, AutoML 2022, 2022. URL https://proceedings.mlr.press/v188/salinas22a.html.
  37. Dptvae: Data-driven prior-based tabular variational autoencoder for credit data synthesizing. Expert Systems with Applications, 241:122071, 2024. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2023.122071. URL https://www.sciencedirect.com/science/article/pii/S0957417423025733.
  38. Subtab: Subsetting features of tabular data for self-supervised representation learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp.  18853–18865. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/9c8661befae6dbcd08304dbf4dcaf0db-Paper.pdf.
  39. Causal disentangled variational auto-encoder for preference understanding in recommendation. arXiv preprint arXiv:2304.07922, 2023.
  40. Cloud intrusion detection method based on stacked contractive auto-encoder and support vector machine. IEEE transactions on cloud computing, 10(3):1634–1646, 2020.
  41. Pilots’ fatigue status recognition using deep contractive autoencoder network. IEEE Transactions on Instrumentation and Measurement, 68(10):3907–3919, 2019. doi: 10.1109/TIM.2018.2885608.
  42. Interpretation for variational autoencoder used to generate financial synthetic tabular data. Algorithms, 16(2), 2023. ISSN 1999-4893. doi: 10.3390/a16020121. URL https://www.mdpi.com/1999-4893/16/2/121.
  43. Image denoising and inpainting with deep neural networks. In Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/file/6cdd60ea0045eb7a6ec44c54d29ed402-Paper.pdf.
  44. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  9593–9602, June 2021.
  45. Overview of artificial neural networks. Artificial neural networks: methods and applications, pp. 14–22, 2009.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now