Mixed-Integer Optimization with Constraint Learning (2111.04469v3)

Abstract: We establish a broad methodological foundation for mixed-integer optimization with learned constraints. We propose an end-to-end pipeline for data-driven decision making in which constraints and objectives are directly learned from data using machine learning, and the trained models are embedded in an optimization formulation. We exploit the mixed-integer optimization-representability of many machine learning methods, including linear models, decision trees, ensembles, and multi-layer perceptrons, which allows us to capture various underlying relationships between decisions, contextual variables, and outcomes. We also introduce two approaches for handling the inherent uncertainty of learning from data. First, we characterize a decision trust region using the convex hull of the observations, to ensure credible recommendations and avoid extrapolation. We efficiently incorporate this representation using column generation and propose a more flexible formulation to deal with low-density regions and high-dimensional datasets. Then, we propose an ensemble learning approach that enforces constraint satisfaction over multiple bootstrapped estimators or multiple algorithms. In combination with domain-driven components, the embedded models and trust region define a mixed-integer optimization problem for prescription generation. We implement this framework as a Python package (OptiCL) for practitioners. We demonstrate the method in both World Food Programme planning and chemotherapy optimization. The case studies illustrate the framework's ability to generate high-quality prescriptions as well as the value added by the trust region, the use of ensembles to control model robustness, the consideration of multiple machine learning methods, and the inclusion of multiple learned constraints.

• The paper presents the OptiCL framework that embeds learned constraints into mixed-integer optimization to enhance decision-making under uncertainty.
• The methodology leverages various ML models, employing ensemble and trust region approaches to maintain constraint satisfaction and mitigate risk.
• Empirical case studies in food planning and chemotherapy design demonstrate the framework’s practical impact on optimizing complex, real-world problems.

An Overview of Mixed-Integer Optimization with Constraint Learning

This paper presents an innovative methodological framework for incorporating learned constraints into mixed-integer optimization (MIO) formulations, specifically through a synergy of ML and optimization techniques. Notably, it introduces a comprehensive end-to-end pipeline referred to as OptiCL (Optimization with Constraint Learning), which is aimed at improving data-driven decision making.

Methodological Contributions

The paper contributes significantly to the field by demonstrating how a variety of ML models, such as linear models, decision trees, ensemble methods, and multi-layer perceptrons, can be embedded into MIO formulations. The embedding approach leverages the ability of many ML models to be represented within mixed-integer constraints, thereby facilitating the optimization of decisions under uncertainty.

For learning constraints, the authors introduce two methodologies to manage uncertainty: an innovative ensemble approach that maintains constraint satisfaction across multiple model predictions, and a trust region constraint defined by the convex hull of observed data. The ensemble approach mitigates risk by permitting constraint violations in a controlled number of model predictions, improving robustness to model specification errors. The trust region offers a further layer of validation by ensuring solutions remain within a credible interpolation space defined by training data.

Implementation and Empirical Analysis

The OptiCL framework is implemented as a Python package, allowing for practical use by researchers and practitioners. The framework's validity and efficacy are demonstrated in two real-world case studies: one involving World Food Programme planning and the other focusing on optimization of chemotherapy regimens. The first case evaluates the framework's capability to generate high-quality food basket prescriptions based on palatability data, while the second explores the design of chemotherapy regimens to balance survival against toxicity constraints.

The results from these case studies emphasize OptiCL’s ability to handle complex datasets and diverse decision-making contexts. Importantly, they highlight how the use of constraint learning and ensemble methods can lead to better-prescribed solutions, while trust regions help avoid model extrapolation errors.

Implications and Future Directions

Practically, this research advances the application of ML within optimization tasks, providing a structured approach to integrating learned models for prescription generation. Theoretically, it bridges a key gap in ML-based optimization by not only predicting outcomes but integrating such predictions into prescriptive analytics.

Moving forward, the framework opens several avenues for future exploration. Potential improvements include incorporating causal inference for more nuanced decision-making scenarios and extending robust optimization techniques to address prediction uncertainty directly within ML models. The versatility of OptiCL positions it as a promising tool for addressing emerging optimization challenges across various domains.

Ultimately, this work represents a marked shift towards data-driven optimization frameworks that embed machine learning insights, offering both researchers and practitioners new mechanisms to harness the predictive power of ML in crafting effective, real-world solutions.