Learning-to-Warm-Start

• Learning-to-Warm-Start is a method where a model learns an initialization from past problem instances to reduce iterations and computational time in iterative solvers.
• It is applied in diverse areas such as neural training, reinforcement learning, and power systems, often yielding substantial efficiency gains and improved feasibility.
• Successful warm-start strategies depend on aligning predictions with solver structure, addressing distribution shifts, and implementing feasibility repair to enhance overall performance.

Searching arXiv for recent and foundational papers on learning-to-warm-start. Learning-to-warm-start is a family of methods in which a model learns an initialization for a downstream optimizer, solver, search procedure, or learner from prior instances of related problems. Across the literature, the warm-start object ranges from neural network weights, Q-tables, policy-search priors, and hyperparameter initial designs to primal or primal-dual iterates for AC optimal power flow, quadratic programming, and sequential convex programming. The unifying aim is to preserve the structure of a problem family while reducing wall-clock time, iteration count, early-stage regret, feasibility failures, or generalization loss.

1. Conceptual scope and formal problem classes

A recurrent formalization appears in sequential learning, fixed-point computation, and parametric optimization. In sequential neural training with cumulative data $S_t=\bigcup_{i=1}^t D_i$, the objective is to solve

$\theta_t=\arg\min_\theta L(\theta;S_t),$

with either a warm start from $\theta_{t-1}$ or a random restart (Ash et al., 2019). In fixed-point optimization, the problem is to find $z\in\mathbb R^p$ such that $z=T_\theta(z)$, and the learned component is a map $h_w:\mathbb R^d\to\mathbb R^p$ that predicts an initial iterate before applying $K$ fixed-point steps (Sambharya et al., 2023). In real-time quadratic programming, the same pattern appears as a feed-forward network $h_{\mathcal W}$ that maps QP parameters to a warm-start for Douglas-Rachford splitting (Sambharya et al., 2022). In AC optimal power flow, a multi-target Random Forest maps load vectors directly to approximate generator and voltage setpoints used as an initial guess for a nonlinear solver (Baker, 2019).

These formulations differ in their downstream operators, but they share the same structural decomposition: a learned predictor produces an initialization, and a trusted solver or learner performs the final refinement. That decomposition is explicit in end-to-end warm-starting for fixed-point algorithms and DR splitting, in meta-feature-based initialization for Bayesian hyperparameter optimization, and in multi-stage pipelines that predict, repair, and then warm-start a constrained optimizer (Kim et al., 2017).

Problem family	Learned warm-start object	Downstream procedure
Sequential neural training	Parameters $\theta$	SGD-based retraining
Fixed-point / QP solving	Initial iterate $z^0$	Fixed-point iterations or DR splitting
ACOPF	$\theta_t=\arg\min_\theta L(\theta;S_t),$0 or subsets	Interior-point AC solver
Bayesian hyperparameter optimization	Initial design points	GP-based Bayesian optimization
Unit commitment	Commitment schedule	MILP solver with warm start
Space manipulation SCP	Stage-2 state-action trajectory	Sequential convex programming

A plausible implication is that learning-to-warm-start is best understood not as a single algorithmic family, but as an interface between statistical prediction and iterative refinement.

2. Warm-starting neural training under incremental data

The deep-learning literature shows that warm-starting is not automatically beneficial. In data-incremental training, warm starts and random restarts can exhibit nearly identical training trajectories while differing substantially in test accuracy. On CIFAR-10 with ResNet-18 and SGD, the reported test accuracy is $\theta_t=\arg\min_\theta L(\theta;S_t),$1 for random initialization and $\theta_t=\arg\min_\theta L(\theta;S_t),$2 for warm start, even though final training losses are similar; logistic regression does not exhibit this pathology, indicating that the effect is specific to deep learning rather than convex optimization (Ash et al., 2019).

The analysis in this line of work attributes the pathology to gradient imbalance between new data $\theta_t=\arg\min_\theta L(\theta;S_t),$3 and old data $\theta_t=\arg\min_\theta L(\theta;S_t),$4, proximity to the previous solution basin, and a bias toward sharper minima under naïve reuse of $\theta_t=\arg\min_\theta L(\theta;S_t),$5 (Ash et al., 2019). The proposed remedy is the “shrink-and-perturb” re-initialization

$\theta_t=\arg\min_\theta L(\theta;S_t),$6

with typical $\theta_t=\arg\min_\theta L(\theta;S_t),$7 and $\theta_t=\arg\min_\theta L(\theta;S_t),$8. On online CIFAR-10 with ResNet-18 and 1000-sample batches, $\theta_t=\arg\min_\theta L(\theta;S_t),$9 and $\theta_{t-1}$0 allow test accuracy to track random restart at approximately $\theta_{t-1}$1 while keeping total training time near $\theta_{t-1}$2 min versus $\theta_{t-1}$3 min for fresh starts (Ash et al., 2019).

A more recent formulation, CKCA, argues that the appropriate response is to warm-start while “stepping out” of the previous converged point and then “seeking around” with feature-space consolidation and adaptive distillation (Shen et al., 2024). CKCA combines Feature Regularization,

$\theta_{t-1}$4

with Adaptive Knowledge Distillation, whose coefficient decays from

$\theta_{t-1}$5

to $\theta_{t-1}$6 over training epochs (Shen et al., 2024). On ImageNet splits with ResNet-50, CKCA reports up to $\theta_{t-1}$7 higher top-1 accuracy than vanilla warm-starting, and in the no-access scenario reaches $\theta_{t-1}$8 top-1 at Stage 10 versus $\theta_{t-1}$9 for warm-start and $z\in\mathbb R^p$0 for iCaRL (Shen et al., 2024).

Taken together, these results establish a central distinction: reusing a checkpoint is not equivalent to learning an effective warm start. The latter must account for basin geometry, gradient balance, and the mismatch between old and new data.

3. Reinforcement learning and interactive systems

In reinforcement learning, warm-starting addresses the cold-start problem: poor early policies, expensive exploration, and, in human-facing systems, unacceptable initial behavior. One approach is to encode prior structure directly into the policy architecture. A neural decision tree framework permits encoding domain knowledge into an initial policy and then improving it with policy-gradient updates; on two OpenAI Gym tasks and two modified StarCraft 2 tasks, it outperforms multilayer-perceptron and recurrent architectures, and untrained humans can provide a reported $z\in\mathbb R^p$1 increase in expected reward relative to baselines prior to training ($z\in\mathbb R^p$2), yielding a $z\in\mathbb R^p$3 increase after policy optimization ($z\in\mathbb R^p$4) (Silva et al., 2019).

A closely related HCI perspective replaces early real-user interaction with synthetic experience from cognitive models. In a standard finite-horizon MDP

$z\in\mathbb R^p$5

a cognitive simulator $z\in\mathbb R^p$6 supplies trajectories for offline pre-training before deployment to real users (Zhang et al., 2021). The paper frames this as a warm-start for applications such as physical-activity notifications and intelligent driving assist, and emphasizes that the gain depends on the fidelity of the underlying cognitive model (Zhang et al., 2021).

In control-oriented RL for hybrid electric vehicles, warm-starting takes the form of informed Q-table initialization rather than policy pretraining. Q-learning is initialized from either Equivalent Consumption Minimization Strategy control or heuristic rules, instead of zero or random Q values. The proposed warm-start Q-learning requires $z\in\mathbb R^p$7 fewer iterations than cold-start Q-learning, improves first-iteration fuel economy from $z\in\mathbb R^p$8 mpg to $z\in\mathbb R^p$9 mpg with heuristic warm start and $z=T_\theta(z)$0 mpg with ECMS warm start, and yields $z=T_\theta(z)$1–$z=T_\theta(z)$2 MPG improvement relative to ECMS on WLTP and HWFET validation cycles (Xu et al., 2020).

Search-based deep RL exhibits a different warm-start mode. In AlphaZero-like training, an adaptive switch replaces the fixed warm-start length $z=T_\theta(z)$3 by

$z=T_\theta(z)$4

where $z=T_\theta(z)$5 is the arena win rate of network-guided MCTS against the enhancement (Wang et al., 2021). The reported average switch points are approximately $z=T_\theta(z)$6–$z=T_\theta(z)$7 iterations for Connect Four, $z=T_\theta(z)$8–$z=T_\theta(z)$9 for Othello, and $h_w:\mathbb R^d\to\mathbb R^p$0–$h_w:\mathbb R^d\to\mathbb R^p$1 for Gobang, and adaptive warm-start achieves higher Elo than both no warm-start and fixed $h_w:\mathbb R^d\to\mathbb R^p$2 schemes (Wang et al., 2021).

LLMs have also been used to warm-start RL through data collection rather than direct policy deployment. LORO first uses an LLM to generate an off-policy dataset for $h_w:\mathbb R^d\to\mathbb R^p$3 episodes, then performs offline pre-training, and finally fine-tunes online. The paper reports up to $h_w:\mathbb R^d\to\mathbb R^p$4 the cumulative rewards of the pure RL baseline, and across six Gym environments reduces the number of episodes needed to reach near-optimal performance by a factor of $h_w:\mathbb R^d\to\mathbb R^p$5 to $h_w:\mathbb R^d\to\mathbb R^p$6 (2505.10861).

These examples show that, in RL, warm-starts may target policy representations, value tables, self-play search policies, or replay data. This suggests that the warm-start object is domain-dependent, but the purpose is stable: improve early behavior without abandoning subsequent policy improvement.

4. End-to-end warm-starting for optimization and fixed-point methods

A distinct research thread treats warm-starting as a supervised or end-to-end prediction problem for iterative solvers. In “End-to-End Learning to Warm-Start for Real-Time Quadratic Optimization,” a feed-forward network predicts an initial iterate $h_w:\mathbb R^d\to\mathbb R^p$7, after which a fixed number of Douglas-Rachford iterations is unrolled: $h_w:\mathbb R^d\to\mathbb R^p$8 The empirical loss is the fixed-point residual $h_w:\mathbb R^d\to\mathbb R^p$9, and the stated generalization bound contains both a Rademacher term and an exponentially decaying DR term $K$0 (Sambharya et al., 2022). On oscillating masses, vehicle dynamics tracking, and Markowitz portfolio optimization, learned warm-starts reduce iteration counts by $K$1–$K$2 with $K$3, and up to $K$4 with $K$5; for example, at $K$6 the portfolio problem drops from about $K$7 iterations with no warm-start to about $K$8 with learned $K$9 and about $h_{\mathcal W}$0 with learned $h_{\mathcal W}$1 (Sambharya et al., 2022).

The broader fixed-point framework generalizes this idea beyond DR splitting. A neural network predicts $h_{\mathcal W}$2, followed by $h_{\mathcal W}$3 iterations of $h_{\mathcal W}$4, and training can minimize either the fixed-point residual $h_{\mathcal W}$5 or the regression loss $h_{\mathcal W}$6 (Sambharya et al., 2023). The paper proves PAC-Bayes bounds for contractive, linearly convergent, and averaged operators, and reports substantial gains across gradient descent, proximal methods, OSQP, and SCS, including up to $h_{\mathcal W}$7 fewer iterations on an unconstrained diagonal QP, about $h_{\mathcal W}$8 savings for Lasso, about $h_{\mathcal W}$9 speed-up for image deblurring, and about $\theta$0 speed-up for phase retrieval and sparse PCA under suitable losses and $\theta$1 (Sambharya et al., 2023).

A theoretical refinement appears in discrete convex optimization. Prior prediction-dependent bounds for $\theta$2-/$\theta$3-convex minimization scaled with the distance between a prediction and one optimal solution, but this can be arbitrarily large when multiple optimal solutions exist. The revised framework replaces that dependence by the distance from the prediction to the set of all optima, proving time complexity proportional to $\theta$4 and giving the first polynomial-time learnability of predictions provably close to sets of optimal solutions via online gradient descent (Sakaue et al., 2023). This directly addresses a common hidden assumption in warm-start analysis: uniqueness of the target solution.

Warm-starting has also entered variational quantum algorithms. In WS-VarQPI, the variational parameters from one policy-evaluation stage are reused at the next stage because successive greedy policies are close in amplitude space. On FrozenLake $\theta$5 with a $\theta$6 linear system, vanilla VarQPI requires $\theta$7 total steps, while WS-VarQPI requires $\theta$8 with similar fidelity ($\theta$9 versus $z^0$0), yielding about $z^0$1 fewer training steps without changing the final greedy policy (Meyer et al., 2024).

5. Power systems, unit commitment, and trajectory optimization

Power-system optimization has produced some of the clearest evidence that warm-start quality depends on what is predicted, how completely it is predicted, and how solver internals use the prediction. In AC optimal power flow, a multi-target Random Forest is trained on historical solutions to map loads $z^0$2 directly to approximate decision variables $z^0$3, using only active and reactive loads as features and no topology or line-parameter information (Baker, 2019). On IEEE 14-, 57-, 118-, and 300-bus networks, average relative errors range from $z^0$4 to $z^0$5 for generation and from $z^0$6 to $z^0$7 for voltage, and the resulting warm start reduces total MIPS solve time on 400 test instances from $z^0$8 s to $z^0$9 s on the 14-bus case and from $\theta_t=\arg\min_\theta L(\theta;S_t),$00 s to $\theta_t=\arg\min_\theta L(\theta;S_t),$01 s on the 300-bus case, with benefits described as solver and network dependent (Baker, 2019).

A later benchmark sharply qualifies the notion that “more information” always helps. For IPOPT on 19 PGLib-OPF instances, only the complete oracle primal-plus-dual restart reliably converges on every baseline-convergent case, with a $\theta_t=\arg\min_\theta L(\theta;S_t),$02 median solve-time speedup. Twelve of the fourteen partial oracle primal-plus-dual combinations have negative median speedups, and block-matched bound-multiplier coverage drops to $\theta_t=\arg\min_\theta L(\theta;S_t),$03 convergence and $\theta_t=\arg\min_\theta L(\theta;S_t),$04 median speedup, whereas full bound coverage reaches $\theta_t=\arg\min_\theta L(\theta;S_t),$05 convergence and $\theta_t=\arg\min_\theta L(\theta;S_t),$06 median speedup (Taheri et al., 8 Jun 2026). The recommended ordering is therefore explicit: predict the full primal vector first; if only partial coverage is possible, prioritize voltage variables; and avoid partial or inconsistent dual predictions unless the primal estimate is nearly complete (Taheri et al., 8 Jun 2026).

In unit commitment, the warm-start pipeline is explicitly multi-stage. A transformer predicts 72-hour generator commitment schedules, deterministic post-processing enforces minimum up/down times and capacity requirements, and a confidence-based variable-fixation scheme reduces the MILP search space before handing the result to a solver (Za'ter et al., 23 Apr 2026). On a single-bus test system with 500 held-out instances, the best pipeline configuration achieves $\theta_t=\arg\min_\theta L(\theta;S_t),$07 feasibility, a median Time Ratio of approximately $\theta_t=\arg\min_\theta L(\theta;S_t),$08 (about $\theta_t=\arg\min_\theta L(\theta;S_t),$09 speedup), Optimality Ratio quartiles of $\theta_t=\arg\min_\theta L(\theta;S_t),$10, $\theta_t=\arg\min_\theta L(\theta;S_t),$11, and $\theta_t=\arg\min_\theta L(\theta;S_t),$12, and about $\theta_t=\arg\min_\theta L(\theta;S_t),$13 of cases with $\theta_t=\arg\min_\theta L(\theta;S_t),$14 relative to the baseline MILP with $\theta_t=\arg\min_\theta L(\theta;S_t),$15 gap (Za'ter et al., 23 Apr 2026).

Sequential convex programming for space manipulation offers a related two-stage architecture. The terminal-approach problem is decomposed into a translational planning stage and a coupled attitude-manipulator torque-allocation stage; a causal transformer warm-start is used for the second stage, which dominates computational cost (Takubo et al., 15 Jun 2026). Over 300 held-out scenarios, the learned warm-start reduces second-stage SCP iteration count by up to $\theta_t=\arg\min_\theta L(\theta;S_t),$16 and runtime by $\theta_t=\arg\min_\theta L(\theta;S_t),$17 while preserving the final control-cost distribution. Under feasibility projection, the learned warm-start nearly halves runtime relative to cost-optimal SCP and avoids the catastrophic high-cost tail observed under heuristic initialization (Takubo et al., 15 Jun 2026).

These cases illustrate a broader operational pattern: raw learned predictions are valuable only when they are aligned with solver structure, feasibility requirements, and variable coverage.

6. Meta-learning, architecture search, active learning, and federated learning

Learning-to-warm-start is also prominent in meta-optimization, where the goal is not to solve the current task directly but to choose a better initial search state. In Bayesian hyperparameter optimization, a Siamese network learns dataset meta-features so that Euclidean distance in meta-feature space reflects historical validation-error similarity. For a new dataset, the $\theta_t=\arg\min_\theta L(\theta;S_t),$18 nearest prior datasets are retrieved, and their best configurations are used as the initial design for Bayesian optimization (Kim et al., 2017). With $\theta_t=\arg\min_\theta L(\theta;S_t),$19 and budget $\theta_t=\arg\min_\theta L(\theta;S_t),$20, the method reaches a given validation error with $\theta_t=\arg\min_\theta L(\theta;S_t),$21–$\theta_t=\arg\min_\theta L(\theta;S_t),$22 fewer GP evaluations than uniform, Latin-hypercube, or Halton initialization, and Bi-LSTM aggregation typically outperforms simple arithmetic aggregation by $\theta_t=\arg\min_\theta L(\theta;S_t),$23–$\theta_t=\arg\min_\theta L(\theta;S_t),$24 (Kim et al., 2017).

The same meta-learning logic appears in differentiable neural architecture search. Warm-started DARTS uses Task2Vec to choose a transfer architecture or a meta-transfer architecture learned over multiple tasks, and then initializes DARTS in a reduced search space (Grobelnik et al., 2022). Averaged over five Meta-Dataset image-classification tasks, search cost is reduced by approximately $\theta_t=\arg\min_\theta L(\theta;S_t),$25; the Aircraft example is reported as $\theta_t=\arg\min_\theta L(\theta;S_t),$26 h to $\theta_t=\arg\min_\theta L(\theta;S_t),$27 h, and accuracy improves on three of five tasks, including Flower from $\theta_t=\arg\min_\theta L(\theta;S_t),$28 to $\theta_t=\arg\min_\theta L(\theta;S_t),$29 and Aircraft from $\theta_t=\arg\min_\theta L(\theta;S_t),$30 to $\theta_t=\arg\min_\theta L(\theta;S_t),$31 (Grobelnik et al., 2022).

Active learning in software engineering provides another variant. Here the warm-start is an initial batch of candidate examples. Using Gemini 1.5 Pro to generate warm-starts, the reported best method is LLM/Exploit for low-dimensional tasks, achieving rank 0 on $\theta_t=\arg\min_\theta L(\theta;S_t),$32 of the 12 low-dimensional datasets and $\theta_t=\arg\min_\theta L(\theta;S_t),$33 of the medium-dimensional datasets, whereas in high-dimensional tasks Gaussian Process Models with UCB or EI are preferred and LLM/Exploit falls below $\theta_t=\arg\min_\theta L(\theta;S_t),$34 rank 0 (Senthilkumar et al., 2024). This directly shows that warm-start quality can be regime-dependent even when the same acquisition loop is used afterward.

Federated learning extends the idea from optimization state to model initialization and synthetic-data priors. WarmFed begins from a pre-trained diffusion model, fine-tunes LoRA adapters on each client, generates client-conditioned synthetic data at the server, trains a global model on the synthetic corpus, and then performs server-side fine-tuning plus dynamic self-distillation for personalized models (Feng et al., 5 Mar 2025). Reported one-shot results include Office-Caltech10 global-model accuracy $\theta_t=\arg\min_\theta L(\theta;S_t),$35 and personalized-model accuracy $\theta_t=\arg\min_\theta L(\theta;S_t),$36, while five-round results report global average $\theta_t=\arg\min_\theta L(\theta;S_t),$37 and personalized average $\theta_t=\arg\min_\theta L(\theta;S_t),$38; the extra LoRA upload is $\theta_t=\arg\min_\theta L(\theta;S_t),$39 MB once, and the method is described as converging to at least $\theta_t=\arg\min_\theta L(\theta;S_t),$40 global accuracy in $\theta_t=\arg\min_\theta L(\theta;S_t),$41–$\theta_t=\arg\min_\theta L(\theta;S_t),$42 rounds versus $\theta_t=\arg\min_\theta L(\theta;S_t),$43 rounds for FedAvg (Feng et al., 5 Mar 2025).

A plausible implication is that warm-starting is especially powerful when the downstream procedure is expensive but structurally repetitive: the warm start effectively transfers task-family information into the initial search geometry.

7. Limitations, misconceptions, and recurrent open problems

A persistent misconception is that any informative initialization should help. Several results contradict that view. Naïve checkpoint reuse can harm generalization in deep learning even when training loss is unchanged (Ash et al., 2019). In ACOPF, most partial primal-plus-dual restarts increase solve time or reduce convergence reliability, and practical DC seeding loses statistical significance once DCOPF presolve cost is included end-to-end (Taheri et al., 8 Jun 2026). In discrete convex optimization, prediction-dependent bounds based on one arbitrarily chosen optimum can become meaningless when the solution set is non-unique, which is why the distance-to-set formulation was introduced (Sakaue et al., 2023).

Another recurrent issue is feasibility. In ACOPF, the Random Forest prediction is almost always infeasible, though its violation is often small enough for the solver to recover quickly (Baker, 2019). In unit commitment, transformer-only predictions are infeasible about $\theta_t=\arg\min_\theta L(\theta;S_t),$44 of the time, and warm-start only reaches about $\theta_t=\arg\min_\theta L(\theta;S_t),$45 feasibility, which is why deterministic post-processing and confidence-based fixation are integral to the full pipeline (Za'ter et al., 23 Apr 2026). These results suggest that, in constrained problems, warm-start quality cannot be evaluated solely by prediction error; solver compatibility and feasibility repair are often decisive.

A third limitation is distributional mismatch. Cognitive-model-based RL warm-starts depend on simulator fidelity (Zhang et al., 2021). LLM warm-starts for SE active learning are effective in low- and medium-dimensional tasks but lose their edge in high-dimensional problems (Senthilkumar et al., 2024). LORO’s analysis assumes offline coverage of the optimal policy support, and the paper explicitly notes that this assumption may fail on highly complex tasks (2505.10861). WarmFed, for its part, states that it does not provide formal convergence proofs, instead offering empirical evidence and design intuition (Feng et al., 5 Mar 2025).

Open directions follow directly from these limitations. The literature repeatedly points toward better coverage metrics, learned feasibility restoration, principled target selection for partial warm-starts, stronger theory under non-unique optima, and more careful treatment of train-test or sim-to-real shift. The most stable conclusion is therefore not that warm starts are universally beneficial, but that learned warm starts are effective when the predicted object matches the downstream algorithm’s geometry, constraints, and error tolerance.