Graph Q-Learning for Combinatorial Optimization (2401.05610v1)

Abstract: Graph-structured data is ubiquitous throughout natural and social sciences, and Graph Neural Networks (GNNs) have recently been shown to be effective at solving prediction and inference problems on graph data. In this paper, we propose and demonstrate that GNNs can be applied to solve Combinatorial Optimization (CO) problems. CO concerns optimizing a function over a discrete solution space that is often intractably large. To learn to solve CO problems, we formulate the optimization process as a sequential decision making problem, where the return is related to how close the candidate solution is to optimality. We use a GNN to learn a policy to iteratively build increasingly promising candidate solutions. We present preliminary evidence that GNNs trained through Q-Learning can solve CO problems with performance approaching state-of-the-art heuristic-based solvers, using only a fraction of the parameters and training time.

• The paper presents a novel GNN-based Q-learning model that applies reinforcement learning to graph-structured CO problems.
• It reformulates the Flexible Job Shop Scheduling problem as an MDP to minimize makespan with efficient scheduling decisions.
• Experimental results show improved scalability and runtime efficiency compared to traditional heuristic solvers.

Overview

Graph-structured data pervades numerous fields within natural and social sciences, and the effectiveness of Graph Neural Networks (GNNs) has been established for learning on such data. In a significant step forward, researchers have expanded the applications for GNNs to include Combinatorial Optimization (CO) problems. These problems, prevalent across multiple sectors, often do not lend themselves to exact solutions due to the vastness of the discrete solution space they inhabit.

Reinforcement Learning and GNNs for CO Problems

The paper introduces a novel approach whereby instances of CO problems are viewed as graphs, and the search for an optimized solution is framed as a sequential decision-making task. This structure allows GNNs to be paired with reinforcement learning (RL) techniques to learn policies for constructing increasingly viable candidate solutions. A striking aspect of their model is its ability to perform comparably to advanced heuristic-based solvers while utilizing fewer parameters and training resources.

Architectural Insights and Methodology

Pivotal to the implementation is a reinforcement learning framework termed Q-learning, which guides the optimization process. The authors focused on the Flexible Job Shop Scheduling problem (FJSP) to showcase their methodology. The FJSP was turned into a Markov Decision Process (MDP), where actions represented scheduling decisions and the goal was to minimize the "makespan," the total time to complete a set of tasks. Different states of the MDP were represented through a heterogeneous graph, capturing various relationships between operations and machines, significantly enhancing the model's ability to solve CO problems.

Experimental Outcomes

The research conducted extensive experiments, comparing the newly proposed method to established baselines such as simulated annealing and other meta-heuristic techniques. Their findings indicated that while the model could solve problems of various sizes effectively, it showed a particularly better scale in runtime as the problem size increased. Furthermore, by utilizing a graphical representation that is not limited by problem size, the approach embodies a form of meta-learning, adapting to different instances with relative ease.

The researchers' conclusion reinforces the versatility and efficiency of their GNN-based model to solve complex CO problems, potentially rivaling traditional heuristic solvers, thereby opening pathways to more advanced solutions in this domain.