--- title: 'ZeroShotOpt: Pretrained Framework for Optimization' url: https://www.emergentmind.com/topics/zeroshotopt type: topic --- # ZeroShotOpt: Pretrained Framework for Optimization ZeroShotOpt encompasses a family of pretrained, generally applicable optimization frameworks and algorithms for efficient zero-shot black-box optimization, especially in settings where per-instance or per-task tuning is infeasible, and sample efficiency is critical. Various instantiations operate for continuous or discrete spaces, across domains such as hyperparameter optimization, reinforcement learning, and general function minimization. Distinct from conventional optimizers, ZeroShotOpt approaches optimize or learn end-to-end from rich offline data, using meta-learning or reinforcement learning objectives, to yield models capable of immediate deployment on new, diverse tasks with minimal or no adaptation. Recent work has focused on transformer-based sequence models trained with offline reinforcement learning, population-based neural optimizers, and feature-based RL policy representations, often leveraging large-scale synthetic or real expert data to learn transferable optimization policies [2510.03051] [2405.03728] [2502.10792]. ## 1. Problem Formulation and Zero-Shot Objective ZeroShotOpt methods address settings where the goal is to minimize a continuous, expensive, derivative-free black-box objective $f:X\to\mathbb{R}$ over $X\subset\mathbb{R}^d$, typically under strict evaluation budgets $L_{total}$ and with few initial samples. The zero-shot requirement mandates that the optimizer, once pretrained, must generalize to unseen tasks and achieve near-optimal performance using only its pretrained parameters, without any additional problem-specific search or tuning [2510.03051]. The zero-shot optimization objective is often formalized as \[ x^* = \arg\min_{x\in X} f(x) \] with evaluation efficiency quantified by regret metrics and normalized performance aggregated across multiple test functions or benchmarks. ## 2. Pretraining Data Generation and Expert Trajectories High-performing ZeroShotOpt frameworks rely on large-scale pretraining data harvested from synthetic or recorded optimization processes. For example, "ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization" generates 1.6 million synthetic functions via Gaussian processes (GPs), sampling from a diverse set of 78 kernel compositions and length scales to capture broad variability of continuous landscapes. Each function is optimized offline by 12 "expert" Bayesian optimization (BO) algorithms, spanning six acquisition functions (Expected Improvement, LogEI, Upper Confidence Bound, Joint Entropy Search, Max-value Entropy Search, Thompson Sampling) each with multiple kernel configurations. Each expert trajectory consists of a sequence of initial random points followed by sequential decisions optimized to minimize $f$ under a tight query budget (typically $50$ queries), resulting in a pretraining database of over 19 million expert trajectories [2510.03051]. This procedure enables the pretrained model to learn policies that exploit the structure discovered by state-of-the-art optimization heuristics across broad synthetic landscapes. ## 3. Model Architectures and Input Encoding State-of-the-art ZeroShotOpt models typically leverage transformer-based architectures adapted for the sequential nature of optimization. The Decision Transformer (DT) decoder-only transformer framework is commonly used. Inputs are constructed as concatenated token sequences consisting of normalized and discretized regret signals, remaining budget indicators, action tokens for proposed solution coordinates, and state tokens for observed objective values, all embedded and passed through a deep transformer (e.g., 16 layers, 16 attention heads, 200M+ parameters). This tokenization and embedding strategy allows integration of arbitrary-length optimization histories and conditioning on target performance metrics. The transformer outputs, at each time step, categorical distributions over possible discretized coordinate values for the next query as well as corresponding predictive distributions for expected objective outcomes, supporting sample-based candidate generation and value estimation [2510.03051]. Alternative architectures based on population-based evolutionary strategies have also been developed. For example, the Pretrained Optimization Model (POM, also referenced as GPOM) organizes the optimizer into differentiable modules for mutation, crossover, and selection, typically modeled after Differential Evolution, with deep neural networks parameterizing attention, mixture weights, and per-individual crossover rates, all amenable to end-to-end meta-learning [2405.03728]. ## 4. Offline RL Pretraining and Loss Design ZeroShotOpt is trained offline, leveraging behavioral cloning or offline RL objectives tailored to optimization performance across expert trajectories. The principal loss is conditioned on the normalized regret: \[ R(\tau) = \sqrt{\frac{\min_{i\leq L} f(x_i) - f^*}{\text{median}_{i\leq m} f(x_i) - f^*}} \] where $f^*$ is the known global minimum, $m$ is the initial random sample count, and $L$ is the total budget. The transformer’s parameters are optimized to maximize the likelihood of expert actions (next queries) and predicted objective values at each step, under fixed regret and length conditioning, regularized by AdamW weight decay and cosine learning rate scheduling. For population-based models, the pretraining loss is often the normalized improvement over successive generations, averaged across a meta-batch of tasks and generations, optimized by stochastic gradient descent or Adam. Pseudocode example for transformer-based ZeroShotOpt training [2510.03051]: ```python for batch in DataLoader: R, L, {x_i, f_i}_{i=1..T} = batch Embed(R), Embed(L) for t in range(1, T): predict a_t ← Transformer(s_{