Machine Learning Augmented Branch and Bound for Mixed Integer Linear Programming

Abstract: Mixed Integer Linear Programming (MILP) is a pillar of mathematical optimization that offers a powerful modeling language for a wide range of applications. During the past decades, enormous algorithmic progress has been made in solving MILPs, and many commercial and academic software packages exist. Nevertheless, the availability of data, both from problem instances and from solvers, and the desire to solve new problems and larger (real-life) instances, trigger the need for continuing algorithmic development. MILP solvers use branch and bound as their main component. In recent years, there has been an explosive development in the use of machine learning algorithms for enhancing all main tasks involved in the branch-and-bound algorithm, such as primal heuristics, branching, cutting planes, node selection and solver configuration decisions. This paper presents a survey of such approaches, addressing the vision of integration of machine learning and mathematical optimization as complementary technologies, and how this integration can benefit MILP solving. In particular, we give detailed attention to machine learning algorithms that automatically optimize some metric of branch-and-bound efficiency. We also address how to represent MILPs in the context of applying learning algorithms, MILP benchmarks and software.

• The paper introduces a novel integration of ML with branch-and-bound to enhance MILP solving efficiency.
• ML techniques improve primal heuristics, branching decisions, and cutting plane selections to reduce computational effort.
• The study leverages graph neural networks and established benchmarks to refine MILP problem representation and solver performance.

Machine Learning Augmented Branch and Bound for Mixed Integer Linear Programming

This paper explores the integration of ML techniques with the branch-and-bound (B&B) algorithm, which is a core strategy for solving Mixed Integer Linear Programs (MILPs). The study reviews the progress in ML applications within the B&B framework, emphasizing ML's role in enhancing solver efficiency by optimizing key tasks like branching, cutting planes, and node selection. The paper also addresses the mathematical representation of MILPs and the benchmarks and software used in this context. Through this synthesis, it presents a vision of ML and mathematical optimization as complementary, rather than competing, technologies.

Introduction to Mixed Integer Linear Programming

Mixed Integer Linear Programming is a fundamental area of mathematical optimization, extensively used across various domains such as transportation, energy systems, and production planning. MILPs are characterized by the presence of both continuous and discrete decision variables, modeled through linear relationships. The problem involves optimizing a linear objective function subject to a set of linear equality and inequality constraints. Solving MILPs often involves techniques like the B&B algorithm, which systematically explores the solution space by partitioning it into smaller subproblems through branching and bounding techniques. The efficiency of B&B heavily relies on heuristic rules for decision-making processes such as node selection and variable branching.

Figure 1: Markov Decision Process.

Machine Learning Integration in Branch-and-Bound

Primal Heuristics

Primal heuristics are crucial in finding feasible solutions quickly, which helps in improving the primal bound early in the solution process. ML techniques offer promising approaches for enhancing primal heuristics by:

• Solution Prediction: Using ML to predict optimal or near-optimal solutions to guide heuristic search.
• Neighborhood Selection: Applying learning methods to select subsets of variables for problem decomposition.
• Heuristic Scheduling: Leveraging ML to decide which heuristic to apply during different stages of the B&B process.

  Figure 2: Three learning problems related to primal heuristics: (a) predict a reference solution and search in its neighborhood, (b) neighborhood selection -- which and/or how many variables to unfix and re-optimize, (c) heuristic scheduling -- which heuristics to run and/or for how long.

Branching

Branching rules are pivotal in B&B algorithms to determine which variable to branch on, impacting the size of the search tree and overall computational effort. ML can enhance branching decisions by:

• Learning to approximate strong branching scores using historical data and expert-designed heuristics.
• Implementing online learning methods that adaptively refine branching strategies based on real-time data.
• Employing graph neural networks (GNNs) to leverage graphical representations of MILPs for improved variable selection.

Cutting Planes

Cutting planes are utilized to tighten the linear relaxation of MILP without excluding any feasible integer solutions. ML approaches in cutting plane selection include:

• Employing imitation learning to predict effective cuts based on historical solver data.
• Utilizing reinforcement learning to dynamically select cuts, aiming to minimize the LP relaxation gap.

Node Selection

Node selection, although often secondary in effect compared to branching or cuts, benefits from ML models that identify productive paths through the B&B tree, balancing between depth-first and best-first searches.

Representations and Benchmarks

Leveraging ML for MILP solving requires effective representation of MILP problems, where GNNs have shown substantial promise due to their ability to handle the structural properties of MILPs through graph-based representations (Figure 3).

Figure 3: Embedding computation with a Graph Neural Network. Here, we use the abbreviation $\xi^t_v$ for $\xi^t(G,v)$. To update the embedding $\xi^t_v$ of node $v$ at time $t$, the embeddings of neighboring nodes are added and then combined with the current embedding using the comb function. The result is $\xi^{t+1}_v$.

Publicly available MILP benchmarks, such as MIPLIB, provide extensive datasets for training and evaluating ML models in this context. These benchmarks, combined with standardized software libraries like SCIP and Gurobi, support the development and testing of ML-enhanced MILP solving algorithms.

Conclusion

The synthesis of machine learning and mathematical optimization outlined in this paper showcases significant advancements and potential in optimizing MILP solving. The integration of ML techniques offers notable strides toward more efficient and adaptive algorithms, aiding in solving increasingly complex and large-scale optimization problems. Future research directions include enhancing generalization capabilities of ML models to tackle heterogeneous instance distributions and further exploiting solver-generated data to inform more dynamic decision-making processes within the branch-and-bound framework.