Transfer Learning from Foundational Optimization Embeddings to Unsupervised SAT Representations

Abstract: Foundational optimization embeddings have recently emerged as powerful pre-trained representations for mixed-integer programming (MIP) problems. These embeddings were shown to enable cross-domain transfer and reduce reliance on solver-generated labels. In this work, we investigate whether such representations generalize beyond optimization to decision problems, focusing on Boolean satisfiability (SAT). We adapt the foundational optimization architecture to SAT by mapping CNF formulas into the same bipartite constraint-variable graph representation used for MIPs. This allows direct reuse of the pre-trained embedding model without architectural changes or supervised fine-tuning. Our results show that these embeddings capture structural regularities in SAT instances and support unsupervised tasks such as instance clustering and distribution identification. We demonstrate, for the first time, that foundational optimization embeddings can transfer to constraint satisfaction domains. Our findings is a step toward a unified representational framework for both optimization and decision problems.

• The paper demonstrates that pretrained MIP embeddings can be directly transferred to unsupervised SAT representations using three distinct transfer paradigms.
• It empirically evaluates the effectiveness of zero-change transfer, feature-adapted transfer, and SAT-native pretraining on challenging SAT benchmarks with high NMI and Purity metrics.
• The study implies that foundational, solver-agnostic models can capture robust structural information across both optimization and decision problem domains.

Transfer Learning from Foundational Optimization Embeddings to Unsupervised SAT Representations

Introduction

The paper "Transfer Learning from Foundational Optimization Embeddings to Unsupervised SAT Representations" (2604.15448) investigates the generalization ability of foundational representation learning architectures, specifically Forge, initially developed for Mixed Integer Programming (MIP) problems, in the context of Boolean satisfiability (SAT). The study analyzes whether pretrained optimization embeddings can be effectively transferred for unsupervised representation learning in the decision problem setting of SAT. The research presents three key transfer paradigms: direct zero-change transfer, feature-adapted transfer, and SAT-native pretraining, providing an empirical evaluation on challenging benchmark datasets.

Foundational Representation Learning via Forge

Forge leverages a bipartite constraint-variable graph representation of MIPs, incorporating domain-specific node features. The architecture consists of a two-layer GraphSAGE encoder, followed by a vector-quantized (VQ) codebook and a decoder that simultaneously reconstructs node features and graph edges. The training objective combines feature and edge reconstruction losses with the commitment term of VQ. The result is a highly general instance-, constraint-, and variable-level embedding scheme that is completely unsupervised and solver-independent.

Figure 1: The Forge architecture: GNN-based bipartite encoding of MIP instances, followed by VQ autoencoding, yields transferable embeddings.

The key property of Forge is its foundational nature—it is pretrained on large, diverse MIP corpora without any explicit solver supervision or task-specific labels, aiming for highly transferable optimization representations.

Transfer Methodology: Bridging MIP and SAT

To enable transfer to SAT, SAT formulas in CNF are translated to equivalent 0–1 MIP instances via the standard SAT-to-MIP encoding. This transformation preserves the bipartite constraint-variable graph structure, thereby allowing the Forge architecture and (potentially) its pretrained weights to be applied directly to SAT data. The study defines three variants:

• Forge-Mip (Zero-change Transfer): Direct application of the MIP-pretrained Forge model to SAT instances encoded as MIPs, using MIP-specific features.
• Forge-Mip-Sat (Feature-adapted Transfer): The pretrained weights are retained, but node features are replaced with SAT-specific statistical descriptors (e.g., clause width, positive/negative literal counts for constraints; degree features for variables).
• Forge-Sat (SAT-native Model): The architecture is retained, but all parameters are randomly initialized and unsupervised pretraining is performed exclusively on SAT instances with SAT features.

This setup allows for examining both weight-level and architecture-level transfer between optimization and decision domains.

Figure 2: Workflow for transfer learning—SAT instances are converted to MIP form, featurized (MIP or SAT features), and processed by Forge for unsupervised SAT embeddings.

Empirical Evaluation on SAT Benchmarks

Experiments are conducted using the G4SATBench suite, a challenging and diverse SAT benchmark spanning random, pseudo-industrial, and combinatorial distributions with varying difficulty and feasibility assignments. Embeddings from each Forge variant are clustered using unsupervised methods and assessed in terms of their ability to recover (i) underlying problem type and (ii) feasibility (SAT/UNSAT) categories. The evaluation uses Normalized Mutual Information (NMI) and Purity as cluster quality metrics.

• Baselines: A static aggregation of SAT-specific features (Static-Sat) is used as a non-learned control.

Clustering Results and Analysis

Visualization and quantitative results reveal that all Forge-based representation variants produce meaningful instance-level embeddings. Notably:

• Static-Sat achieves baseline clustering performance but suffers from poor separation of problem types and feasibility.
• Forge-Mip and Forge-Mip-Sat demonstrate substantial cross-domain transfer—the pretrained MIP weights induce semantically coherent structure in SAT instance embedding space, with feature adaptation further improving separation.
• Forge-Sat achieves the most distinct and compact cluster structure, coming close to perfectly separating the 14 ground-truth (problem type, feasibility) groups except for minor overlap among certain combinatorial categories.

  Figure 3: Instance-level embedding projections show cluster separation improves from Static-Sat to Forge-Mip, Forge-Mip-Sat, and is maximized in Forge-Sat.

Quantitatively, Forge-Sat achieves NMI of 0.79 and Purity of 0.66 on the hardest split, surpassing static and transferred baselines. Forge-Mip-Sat (NMI 0.77) outperforms simple feature transfer, confirming that MIP-pretrained weights, when combined with domain-adapted features, provide the strongest transfer performance short of full SAT pretraining.

Key empirical claim: Pretrained foundational optimization embeddings, even without supervised fine-tuning or architecture modification, are directly useful for unsupervised SAT representation tasks.

Implications, Limitations, and Future Directions

The results indicate that foundational representation learning paradigms can generalize beyond optimization to decision problems. Forge’s unsupervised framework yields embeddings capable of distinguishing SAT instance structure and feasibility categories across multiple distributions, with zero reliance on supervised solver labels or SAT-specific training.

However, the current study stops at unsupervised evaluation. Downstream tasks such as satisfiability/assignment prediction, core UNSAT prediction, or guided solver heuristics are natural next steps to thoroughly probe the utility of these transfer embeddings. Node feature selection is limited to simple statistics; more expressive or learned descriptors could yield further improvements. The reliance on a single SAT benchmark restricts conclusions regarding robustness and universality, motivating the development of broader, multi-domain foundational corpora.

A critical theoretical implication is the possibility of learning universal structural embeddings for discrete combinatorial domains, bridging the gap between optimization (MIP), decision (SAT), and by extension, more general constraint satisfaction paradigms (CSP, CP). Practically, the open-source release of pretrained Forge models and pipelines enables practitioners to obtain SAT embeddings “out-of-the-box” for arbitrary instances at the instance, clause, and variable granularity—enabling plug-and-play integration for hybrid neuro-symbolic pipelines.

Conclusion

This research empirically establishes that foundational unsupervised graph-based embeddings, trained for optimization, transfer robustly to the unsupervised representation of decision problems such as SAT. Both weight-level and architecture-level transfer are feasible with strong out-of-the-box performance, pointing toward the emergence of general-purpose, solver-agnostic, unified models for discrete reasoning tasks. Future work should target scaling foundational pretraining to hybrid and multimodal combinatorial corpora and explicitly tie the learned embeddings to downstream solver performance and integration in end-to-end neuro-symbolic reasoning systems.