GPTOpt: GPT-Enabled Optimization Systems

• GPTOpt is a family of optimization frameworks that embed GPT models to generate, repair, and orchestrate optimization formulations in various workflows.
• The approach leverages GPT-based repair loops and symbolic orchestration to improve model validity and efficiency in tasks like MiniZinc code generation and black-box optimization.
• Hybrid architectures are central to GPTOpt, where GPT proposals are refined by external solvers, balancing expressive power with rigorous validation.

Searching arXiv for papers directly related to "GPTOpt" and adjacent GPT-assisted optimization work. GPTOpt denotes a family of GPT-centered optimization systems in which a generative pre-trained transformer is embedded directly in an optimization workflow, and it also names a specific 2025 method for continuous black-box optimization. In the cited literature, the transformer may act as a modeling assistant that generates and repairs MiniZinc programs, a symbolic assistant that reformulates robust and adaptive robust models, a dual-agent controller for AutoML pipelines, a circuit synthesizer for QAOA, or a fine-tuned policy that proposes black-box query points from textual histories (Almonacid, 2023, Bertsimas et al., 2024, Tsai et al., 2023, Tyagin et al., 23 Apr 2025, Meindl et al., 29 Oct 2025). A central unifying feature is that the GPT model typically does not execute the downstream solver itself; rather, it amortizes expensive steps in formulation, search, parameterization, or structured synthesis.

1. Scope and terminological usage

In the cited literature, GPT-centered optimization systems fall into several recurring roles.

Role of the GPT-style model	Artifact produced	Representative paper
Modeling assistant with repair loop	MiniZinc code validated by MiniZinc + Gecode	(Almonacid, 2023)
Direct optimizer for continuous black-box functions	Next query point from evaluation history	(Meindl et al., 29 Oct 2025)
Symbolic optimization assistant	Robust/adaptive robust counterpart and JuMP+Gurobi code	(Bertsimas et al., 2024)
Workflow controller for AutoML	Preprocessing, model choice, and hyperparameter tuning	(Tsai et al., 2023)
Variational quantum circuit generator	QAOA or adaptive-QAOA circuit plus parameters	(Tyagin et al., 23 Apr 2025)

This range of usages matters because “GPTOpt” is not confined to a single algorithmic template. In one line of work, it denotes a solver-in-the-loop architecture for optimization modeling; in another, it denotes an LLM that directly performs sample-efficient black-box optimization after fine-tuning on synthetic Bayesian optimization trajectories (Almonacid, 2023, Meindl et al., 29 Oct 2025). Other papers explicitly describe related systems as “GPTOpt-like,” especially when a GPT-style autoregressive model learns a prior over structured outputs and is later conditioned on instance-specific context (Fu et al., 30 Jun 2026).

2. Optimization-model generation and solver-coupled repair

An early concrete instantiation of GPTOpt is the “Automatic Optimisation Model Generator” for MiniZinc, which uses GPT-3.5 to take a high-level specification, generate an initial optimization model, compile and run that model with MiniZinc 2.7.1 and Gecode 6.3.0, and automatically repair build errors by feeding compiler output to a second GPT-3.5 editing model (Almonacid, 2023). The workflow is a loop: user specification in natural language, generation of MiniZinc source code with text-davinci-003, compilation and execution through the MiniZinc Python API, and, if compilation fails, editing with text-davinci-edit-001 using the exact error message as feedback.

The target problems are small constraint programming models involving discrete decision variables, arrays of discrete variables, open or defined domains, and either no constraints, simple constraints, or the global constraint all_different. Prompt design is deliberately minimalistic: dialogue-style context establishes MiniZinc expertise, the request constrains output to source code only, and deterministic generation is enforced with temperature=0. The repair loop similarly uses a short instruction of the form “Fix the minizinc code. The Error code is …”, making error feedback the primary conditioning signal.

The experimental results make the feasibility claim precise. Across 10 test instances, 8 models were valid in the sense of compiling and running, and 5 were both valid and correct with respect to the requested specification. Instances 1, 2, 3, 4, and 9 were valid and correct; instances 6, 7, and 8 were valid but not correct; instances 5 and 10 remained invalid after 10 repair steps. The most persistent failure mode involved all_different: the generated code included the global constraint but omitted the required include "alldifferent.mzn";, and the repair loop did not consistently fix the omission even after repeated iterations (Almonacid, 2023).

The significance of this system is architectural rather than purely empirical. It establishes a canonical GPTOpt loop in which GPT is neither the optimizer nor the solver, but a modeling assistant tightly integrated with validation, execution, and machine-generated debugging. That pattern reappears in later work, often with richer target languages, more structured prompting, or stronger external evaluators.

3. GPTOpt as a learned black-box optimizer

The specific method titled “GPTOpt: Towards Efficient LLM-Based Black-Box Optimization” formulates the problem as

$$\mathbf{x}^* = \arg\min_{\mathbf{x}\in\mathcal{X}} f(\mathbf{x}),$$

where $f$ is an expensive derivative-free black-box objective on a bounded domain $\mathcal{X}\subseteq\mathbb{R}^d$ with $d\le 10$ (Meindl et al., 29 Oct 2025). Its motivation is the gap between classical Bayesian optimization, which is sample-efficient but sensitive to kernel and acquisition-function choices, and out-of-the-box LLMs, which exhibit weak performance on continuous numerical optimization tasks.

GPTOpt addresses this by fine-tuning Llama 3.2 3B via LoRA on a large synthetic corpus of optimization trajectories. The training distribution spans Gaussian processes, random neural networks, ODE-based functions, expression trees, and Fourier expressions, together with augmentations such as nonlinear input warps, discontinuity staircases, nonlinear kinks and softplus hinges, soft plateaus, and frequency modulated ripples. For each synthetic function, the pipeline runs 10 Bayesian optimization configurations—LogEI with $\xi\in\{0.0,0.01,0.1\}$, PI with $\xi\in\{0.0,0.01,0.1\}$, and UCB with $\kappa\in\{0.1,1.0,2.576,10.0\}$—starting from 10 random points and continuing for 40 optimization steps. This yields about 2.5M trajectories.

A crucial implementation choice is numeric tokenization. Inputs and objective values are discretized into single tokens in $[0,999]$, because Llama 3.2 has dedicated numeric tokens for integers 0–999 and multi-token numerals are less reliable for small models. The model is then trained as a next-token predictor over textual trajectory histories. At inference time, GPTOpt adapts objective normalization without access to the true global range by fixing the largest observed objective to 999 and decreasing the minimum encoding level $C_{\min}(t)$ linearly from 500 to 100 over the optimization horizon. It then samples $k=4$ candidate continuations and selects the action with the highest expected-improvement-style score in token space.

The reported evaluation spans holdout synthetic functions, BBOB, and VLSE, always with 10 initial random samples and 40 optimization steps, for a total budget of 50 evaluations. The main reported metric is

$f$0

where $f$1 is the true global minimum and $f$2 is the median objective value of the randomly sampled initial points (Meindl et al., 29 Oct 2025). GPTOpt significantly outperforms all individual BO methods and other optimizers on the synthetic holdout distribution, and it remains the top performer on BBOB and VLSE across dimensions $f$3 through $f$4. In this usage, GPTOpt is not a solver wrapper but a learned optimizer proper: a direct policy over continuous query points conditioned on the history of previous evaluations.

4. Symbolic orchestration in robust optimization and AutoML

A distinct branch of GPTOpt research treats the LLM as a symbolic controller for optimization workflows. In “Robust and Adaptive Optimization under a LLM Lens,” ChatGPT 3.5 is prompted step by step to formulate nominal linear problems, introduce uncertainty, derive tractable robust or adaptive robust counterparts, and emit executable JuMP+Gurobi code (Bertsimas et al., 2024). The paper works in the robust and adaptive robust optimization framework of Bertsimas & den Hertog and uses informational prompts that explicitly encode support-function identities, dual-norm rules, and linear decision rules. For simple problems, ChatGPT is reported to perform the sequence with little to no errors; however, the paper also documents characteristic failure modes, including basic algebraic sign mistakes, incorrect handling of intersections of uncertainty sets, and plausible-looking but mathematically wrong robust counterparts.

The design lesson is that GPTOpt-style symbolic optimization requires decomposition into explicit stages: formalize the nominal model, add uncertainty, derive the robust counterpart, then generate solver code. The paper argues, in effect, for a verified co-pilot architecture rather than autonomous symbolic optimization, because errors often appear at the level of theorem selection rather than syntax (Bertsimas et al., 2024).

AutoML-GPT extends the same orchestration logic to machine-learning pipeline search. It is a dual-agent system built with LangChain ReAct, in which a Reasoning agent interprets the user’s objective and plans the pipeline, while a Coding agent generates and executes Python code for dataset exploration, preprocessing, feature engineering, model selection, and hyperparameter tuning (Tsai et al., 2023). The high-level loop is explicit: “Explore the dataset,” “Process the dataset,” “Select the model,” and “Fine tune the parameters.” The framework compares against auto-sklearn, TPOT, Auto-WEKA, H2O AutoML, GCP-Tables, and AutoGluon under an 8-hour training budget, and it is reported to achieve competitive Kaggle leaderboard percentile ranks while relying on single models rather than heavy ensembling (Tsai et al., 2023).

Together, these systems show a broader GPTOpt pattern in which optimization is mediated by planning, tool invocation, and code synthesis rather than by direct proposal of candidate numeric solutions.

5. Quantum and structured generative variants

QAOA-GPT pushes GPTOpt into variational quantum optimization. The task is MaxCut on weighted graphs, expressed as

$f$5

with QAOA or ADAPT-QAOA circuits used as the variational ansatz (Tyagin et al., 23 Apr 2025). QAOA-GPT learns the mapping

$f$6

thereby replacing much of the classical loop used in standard QAOA and ADAPT-QAOA. Training data are generated offline by running ADAPT-QAOA on random weighted Erdős–Rényi graphs, filtering circuits by target approximation ratio, tokenizing graph edge lists and circuit layers, and conditioning a decoder-only transformer with FEATHER graph embeddings.

The reported quantitative results are specific. On 1000 random graphs for each of $f$7, the best-of-five QAOA-GPT generations achieve mean approximation ratios of $f$8, $f$9, and $\mathcal{X}\subseteq\mathbb{R}^d$0, while ADAPT-QAOA reaches $\mathcal{X}\subseteq\mathbb{R}^d$1, $\mathcal{X}\subseteq\mathbb{R}^d$2, and $\mathcal{X}\subseteq\mathbb{R}^d$3, and standard QAOA at depth $\mathcal{X}\subseteq\mathbb{R}^d$4 reaches $\mathcal{X}\subseteq\mathbb{R}^d$5, $\mathcal{X}\subseteq\mathbb{R}^d$6, and $\mathcal{X}\subseteq\mathbb{R}^d$7 (Tyagin et al., 23 Apr 2025). QAOA-GPT circuits are slightly deeper than ADAPT-QAOA on average, but still compact, and inference is reported to run in nearly constant time as problem size grows, unlike the gradient-heavy adaptive baseline.

TopoGPT is not an optimization paper in the narrow sense, but it explicitly presents a GPT-style recipe that is close in spirit to GPTOpt. It serializes lane graphs into token sequences, pre-trains an autoregressive transformer on 3.3M map scenes to learn a geometry prior, and then aligns perception features to that prior through a perception adapter (Fu et al., 30 Jun 2026). This suggests a transferable GPTOpt template: serialize the structured object, pre-train a generative prior on a large corpus of high-quality solutions or outputs, then condition generation on instance-specific context. The paper’s own discussion makes this connection explicit for planning, trajectory generation, combinatorial optimization, and graph generation (Fu et al., 30 Jun 2026).

6. Adjacent usages, misconceptions, and recurrent limitations

The GPTOpt label can be confused with work that optimizes GPT or OPT models themselves rather than using GPT as the optimizer. GPTQ is a one-shot, weight-only, post-training quantization method for GPT and OPT models that uses approximate second-order information to quantize models up to 175B parameters in approximately four GPU hours, supports 3- and 4-bit quantization with negligible accuracy degradation, and enables OPT-175B to fit on a single A100 80GB GPU (Frantar et al., 2022). KnGPT2 applies Kronecker decomposition and intermediate-layer knowledge distillation to compress GPT-2 Small from 124M to 83M parameters, approaching GPT-2 Small performance while outperforming DistilGPT2 at a matched parameter budget (Edalati et al., 2021). SimpleGPT studies GPT training optimization directly: by replacing each linear projection with a SimpleNorm operator, it reduces activation-Hessian spectral norm, tolerates learning rates $\mathcal{X}\subseteq\mathbb{R}^d$8–$\mathcal{X}\subseteq\mathbb{R}^d$9 larger than standard convention, and lowers training loss at 1B, 1.4B, 7B, and 8B scales (Chen et al., 1 Feb 2026). These papers are adjacent to GPTOpt because they optimize the substrate on which GPTOpt systems may run, but they are not themselves GPT-assisted optimization workflows.

Across the GPTOpt family proper, several limitations recur. Solver-coupled model generation can achieve syntactic validity while still violating the requested semantics, and global constraints such as all_different expose brittle gaps in library handling and specification fidelity (Almonacid, 2023). Symbolic robust and adaptive optimization remains vulnerable to algebraic mistakes and theorem misapplication even when detailed informational prompts are supplied (Bertsimas et al., 2024). Direct black-box GPTOpt is currently limited to continuous, single-objective, unconstrained optimization in up to 10 dimensions, and its generalization remains tied to the synthetic training distribution used to construct expert demonstrations (Meindl et al., 29 Oct 2025). QAOA-GPT depends on expensive offline generation of high-quality adaptive circuits and has not yet demonstrated the same level of success for larger problem sizes or richer operator pools under high-precision targets (Tyagin et al., 23 Apr 2025).

A plausible implication is that the most stable form of GPTOpt is hybrid. The literature repeatedly points toward architectures in which GPT handles proposal, formulation, or amortized initialization, while external solvers, verifiers, and evaluators enforce correctness. Future directions named in the cited works include richer validation loops, machine-centered error messages, broader optimization classes such as constrained, mixed-integer, and multi-objective problems, more diverse training corpora, and energy- or reward-aware training objectives for structured generation (Almonacid, 2023, Bertsimas et al., 2024, Meindl et al., 29 Oct 2025, Tyagin et al., 23 Apr 2025).