2000 character limit reached
High Dimensional Data Enrichment: Interpretable, Fast, and Data-Efficient (1806.04047v4)
Published 11 Jun 2018 in stat.ML and cs.LG
Abstract: We consider the problem of multi-task learning in the high dimensional setting. In particular, we introduce an estimator and investigate its statistical and computational properties for the problem of multiple connected linear regressions known as Data Enrichment/Sharing. The between-tasks connections are captured by a cross-tasks \emph{common parameter}, which gets refined by per-task \emph{individual parameters}. Any convex function, e.g., norm, can characterize the structure of both common and individual parameters. We delineate the sample complexity of our estimator and provide a high probability non-asymptotic bound for estimation error of all parameters under a geometric condition. We show that the recovery of the common parameter benefits from \emph{all} of the pooled samples. We propose an iterative estimation algorithm with a geometric convergence rate and supplement our theoretical analysis with experiments on synthetic data. Overall, we present a first thorough statistical and computational analysis of inference in the data-sharing model.
- E. J. Candès and T. Tao, “The power of convex relaxation: Near-optimal matrix completion,” IEEE Transactions on Information Theory, vol. 56, no. 5, pp. 2053–2080, 2010.
- D. L. Donoho, “Compressed sensing,” IEEE Transactions on information theory, vol. 52, no. 4, pp. 1289–1306, 2006.
- J. Friedman, T. Hastie, and R. Tibshirani, “Sparse inverse covariance estimation with the graphical lasso,” Biostatistics, vol. 9, no. 3, pp. 432–441, 2008.
- E. J. Candès and B. Recht, “Exact matrix completion via convex optimization,” Foundations of Computational mathematics, vol. 9, no. 6, p. 717, 2009.
- E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on information theory, vol. 52, no. 2, pp. 489–509, 2006.
- R. Tibshirani, “Regression shrinkage and selection via the lasso,” Journal of the Royal Statistical Society. Series B (Methodological), pp. 267–288, 1996.
- F. Bach, R. Jenatton, J. Mairal, G. Obozinski et al., “Optimization with sparsity-inducing penalties,” Foundations and Trends® in Machine Learning, vol. 4, no. 1, pp. 1–106, 2012.
- S. Negahban, B. Yu, M. J. Wainwright, and P. K. Ravikumar, “A unified framework for high-dimensional analysis of m𝑚mitalic_m-estimators with decomposable regularizers,” in Advances in Neural Information Processing Systems, 2009, pp. 1348–1356.
- P. T. Boufounos and R. G. Baraniuk, “1-bit compressive sensing,” in Information Sciences and Systems, 2008. CISS 2008. 42nd Annual Conference on. IEEE, 2008, pp. 16–21.
- Y. Plan, R. Vershynin, and E. Yudovina, “High-dimensional estimation with geometric constraints,” Information and Inference: A Journal of the IMA, vol. 6, no. 1, pp. 1–40, 2017.
- T. Blumensath and M. E. Davies, “Iterative hard thresholding for compressed sensing,” Applied and computational harmonic analysis, vol. 27, no. 3, pp. 265–274, 2009.
- P. Jain, P. Netrapalli, and S. Sanghavi, “Low-rank matrix completion using alternating minimization,” in Proceedings of the forty-fifth annual ACM symposium on Theory of computing. ACM, 2013, pp. 665–674.
- J. Barretina, G. Caponigro, N. Stransky, K. Venkatesan, A. A. Margolin, S. Kim, C. J. Wilson, J. Lehár, G. V. Kryukov, D. Sonkin et al., “The cancer cell line encyclopedia enables predictive modelling of anticancer drug sensitivity,” Nature, vol. 483, no. 7391, p. 603, 2012.
- F. Iorio, T. A. Knijnenburg, D. J. Vis, G. R. Bignell, M. P. Menden, M. Schubert, N. Aben, E. Goncalves, S. Barthorpe, H. Lightfoot et al., “A landscape of pharmacogenomic interactions in cancer,” Cell, vol. 166, no. 3, pp. 740–754, 2016.
- Q. Gu and A. Banerjee, “High dimensional structured superposition models,” in Advances In Neural Information Processing Systems, 2016, pp. 3684–3692.
- M. B. McCoy and J. A. Tropp, “The achievable performance of convex demixing,” Sep. 2013.
- E. Yang and P. K. Ravikumar, “Dirty statistical models,” in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2013, pp. 611–619.
- A. Jalali, P. Ravikumar, S. Sanghavi, and C. Ruan, “A Dirty Model for Multi-task Learning,” in Advances in Neural Information Processing Systems, 2010, pp. 964–972.
- Y. Zhang and Q. Yang, “A survey on Multi-Task learning,” 2017.
- S. M. Gross and R. Tibshirani, “Data shared lasso: A novel tool to discover uplift,” Computational Statistics & Data Analysis, vol. 101, pp. 226–235, 2016.
- A. Asiaee, S. Oymak, K. R. Coombes, and A. Banerjee, “High dimensional data enrichment: Interpretable, fast, and data-efficient,” arXiv preprint arXiv:1806.04047, 2018.
- ——, “Data enrichment: Multi-task learning in high dimension with theoretical guarantees,” in Adaptive and Multitask Learning Workshop at ICML, 2019.
- A. Chen, A. B. Owen, and M. Shi, “Data enriched linear regression,” Electronic journal of statistics, vol. 9, no. 1, pp. 1078–1112, 2015.
- F. Dondelinger, S. Mukherjee, and Alzheimer’s Disease Neuroimaging Initiative, “The joint lasso: high-dimensional regression for group structured data,” Biostatistics, Sep. 2018.
- E. Ollier and V. Viallon, “Joint estimation of K𝐾Kitalic_K related regression models with simple L1subscript𝐿1L_{1}italic_L start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT-norm penalties,” Nov. 2014.
- ——, “Regression modeling on stratified data with the lasso,” Aug. 2015.
- S. Kakade, O. Shamir, K. Sindharan, and A. Tewari, “Learning exponential families in High-Dimensions: Strong convexity and sparsity,” in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, ser. Proceedings of Machine Learning Research, Y. W. Teh and M. Titterington, Eds., vol. 9. Chia Laguna Resort, Sardinia, Italy: PMLR, 2010, pp. 381–388.
- S. Negahban and M. J. Wainwright, “Restricted strong convexity and weighted matrix completion: Optimal bounds with noise,” Journal of Machine Learning Research, vol. 13, no. May, pp. 1665–1697, 2012.
- Y. Plan and R. Vershynin, “Robust 1-bit compressed sensing and sparse logistic regression: A convex programming approach,” IEEE transactions on information theory / Professional Technical Group on Information Theory, vol. 59, no. 1, pp. 482–494, 2013.
- ——, “The generalized lasso with Non-Linear observations,” IEEE transactions on information theory / Professional Technical Group on Information Theory, vol. 62, no. 3, pp. 1528–1537, Mar. 2016.
- Z. Yang, Z. Wang, H. Liu, Y. Eldar, and T. Zhang, “Sparse nonlinear regression: Parameter estimation under nonconvexity,” in Proceedings of The 33rd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 2016, pp. 2472–2481.
- J. Chen, J. Liu, and J. Ye, “Learning incoherent sparse and Low-Rank patterns from multiple tasks,” ACM transactions on knowledge discovery from data, vol. 5, no. 4, p. 22, 2012.
- R. Vershynin, “Introduction to the non-asymptotic analysis of random matrices,” in Compressed Sensing. Cambridge University Press, Cambridge, 2012, pp. 210–268.
- A. Banerjee, S. Chen, F. Fazayeli, and V. Sivakumar, “Estimation with Norm Regularization,” in Advances in Neural Information Processing Systems, 2014, pp. 1556–1564.
- S. N. Negahban, P. Ravikumar, M. J. Wainwright, and B. Yu, “A Unified Framework for High-Dimensional Analysis of $M$-Estimators with Decomposable Regularizers,” Statistical Science, vol. 27, no. 4, pp. 538–557, 2012.
- G. Raskutti, M. J. Wainwright, and B. Yu, “Restricted eigenvalue properties for correlated gaussian designs,” Journal of Machine Learning Research, vol. 11, pp. 2241–2259, 2010.
- S. Mendelson, “Learning without concentration,” Journal of the ACM, vol. 62, no. 3, pp. 21:1–21:25, 2015.
- J. A. Tropp, “Convex recovery of a structured signal from independent random linear measurements,” in Sampling Theory, a Renaissance. Springer, 2015, pp. 67–101.
- M. Rudelson and S. Zhou, “Reconstruction from anisotropic random measurements,” IEEE Transactions on Information Theory, vol. 59, no. 6, pp. 3434–3447, 2013.
- P. J. Bickel, Y. Ritov, A. B. Tsybakov et al., “Simultaneous analysis of lasso and dantzig selector,” The Annals of Statistics, vol. 37, no. 4, pp. 1705–1732, 2009.
- E. Candes, T. Tao et al., “The dantzig selector: Statistical estimation when p is much larger than n,” The Annals of Statistics, vol. 35, no. 6, pp. 2313–2351, 2007.
- V. Chandrasekaran, B. Recht, P. A. Parrilo, and A. S. Willsky, “The convex geometry of linear inverse problems,” Foundations of Computational Mathematics, vol. 12, no. 6, pp. 805–849, 2012.
- S. Chatterjee, S. Chen, and A. Banerjee, “Generalized dantzig selector: Application to the k-support norm,” in Advances in Neural Information Processing Systems, 2014, pp. 1934–1942.
- S. Oymak, B. Recht, and M. Soltanolkotabi, “Sharp time–data tradeoffs for linear inverse problems,” IEEE Transactions on Information Theory, vol. 64, no. 6, pp. 4129–4158, 2017.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
