Multi-Target Regression via Input Space Expansion: Treating Targets as Inputs (1211.6581v5)

Abstract: In many practical applications of supervised learning the task involves the prediction of multiple target variables from a common set of input variables. When the prediction targets are binary the task is called multi-label classification, while when the targets are continuous the task is called multi-target regression. In both tasks, target variables often exhibit statistical dependencies and exploiting them in order to improve predictive accuracy is a core challenge. A family of multi-label classification methods address this challenge by building a separate model for each target on an expanded input space where other targets are treated as additional input variables. Despite the success of these methods in the multi-label classification domain, their applicability and effectiveness in multi-target regression has not been studied until now. In this paper, we introduce two new methods for multi-target regression, called Stacked Single-Target and Ensemble of Regressor Chains, by adapting two popular multi-label classification methods of this family. Furthermore, we highlight an inherent problem of these methods - a discrepancy of the values of the additional input variables between training and prediction - and develop extensions that use out-of-sample estimates of the target variables during training in order to tackle this problem. The results of an extensive experimental evaluation carried out on a large and diverse collection of datasets show that, when the discrepancy is appropriately mitigated, the proposed methods attain consistent improvements over the independent regressions baseline. Moreover, two versions of Ensemble of Regression Chains perform significantly better than four state-of-the-art methods including regularization-based multi-task learning methods and a multi-objective random forest approach.

• The paper demonstrates that treating target variables as additional inputs can significantly improve multi-target regression performance.
• The paper introduces Stacked Single-Target (SST) and Ensemble of Regressor Chains (ERC) to capture inter-target dependencies for enhanced predictive accuracy.
• The paper addresses training-prediction discrepancies by using cross-validated out-of-sample target estimates, ensuring robust performance improvements.

Multi-Target Regression via Input Space Expansion

The paper "Multi-Target Regression via Input Space Expansion: Treating Targets as Inputs" by Eleftherios Spyromitros-Xioufis, Grigorios Tsoumakas, William Groves, and Ioannis Vlahavas explores a novel approach to multi-target regression (MTR) by adapting methodologies from multi-label classification (MLC). The core objective is to exploit statistical dependencies between multiple target variables to enhance predictive accuracy, a tactic that has shown success in MLC but has not been extensively studied in the MTR domain until this research.

The authors introduce two methods, Stacked Single-Target (SST) and Ensemble of Regressor Chains (ERC). Both methods draw inspiration from MLC strategies where target variables are recursively treated as inputs, thereby converting the regression task into a series of interconnected sub-tasks. Specifically:

• Stacked Single-Target (SST) involves a two-layer learning process where initial predictions from single-target models are augmented with additional target variables' estimates for a subsequent learning phase. However, these augmentations can introduce discrepancies due to differences in the values of targets used during training and prediction.
• Ensemble of Regressor Chains (ERC) extends the concept of chaining models, allowing for target variables' predictive correlations to be implicitly modeled by ordering them into multiple chains and integrating their predictions.

The paper also uncovers a potential pitfall in these methods. Discrepancies arise because of differences between estimated target values used for training and the actual values during prediction. To address this, the authors propose extensions that use out-of-sample estimates during training, derived via cross-validation, to mitigate this issue.

Empirical evaluation was comprehensive, involving 18 datasets with diverse characteristics and comparing against various state-of-the-art methods, including regularization-based multi-task learning and random forest techniques. The results conclusively show that when discrepancies are effectively mitigated, the proposed methods — notably ERC variants — consistently outperform the baseline approach (independent regressions) and exhibit superior performance against several existing strategies. ERC, in particular, was highlighted as a strong contender, significantly improving accuracy with minimized risk of degrading baseline performance.

From a theoretical perspective, the paper situates SST and ERC as methods for leveraging target dependencies and explores the bias-variance trade-offs inherent in these approaches. By introducing additional input features relating to target variables, both methods reduce model bias at the expense of increased variance. The choice of base regressor greatly influences the effectiveness of these methods. The performance enhancement realized with non-linear base learners, such as bagged trees, underscores their ability to adapt and leverage the noisy meta-features introduced by targets-turned-inputs.

In conclusion, this research not only expands MTR's methodological landscape by adapting successful MLC principles but also critically addresses implementation challenges that arise from target dependencies. It points toward a future where abstracting and adapting solutions from related domains may yield benefits across various machine learning tasks. This work suggests future investigations might focus on optimizing base learner choice and further fine-tuning strategies to improve computational efficiency and adaptability to different dataset characteristics in MTR contexts.