Keep Rehearsing and Refining: Lifelong Learning Vehicle Routing under Continually Drifting Tasks

Abstract: Existing neural solvers for vehicle routing problems (VRPs) are typically trained either in a one-off manner on a fixed set of pre-defined tasks or in a lifelong manner on several tasks arriving sequentially, assuming sufficient training on each task. Both settings overlook a common real-world property: problem patterns may drift continually over time, yielding massive tasks sequentially arising while offering only limited training resources per task. In this paper, we study a novel lifelong learning paradigm for neural VRP solvers under continually drifting tasks over learning time steps, where sufficient training for any given task at any time is not available. We propose Dual Replay with Experience Enhancement (DREE), a general framework to improve learning efficiency and mitigate catastrophic forgetting under such drift. Extensive experiments show that, under such continual drift, DREE effectively learns new tasks, preserves prior knowledge, improves generalization to unseen tasks, and can be applied to diverse existing neural solvers.

• The paper presents the DREE method, which combines problem instance replay, behavior replay, and experience enhancement to effectively manage lifelong learning under continual drift in vehicle routing problems
• DREE outperforms existing methods in continually drifting tasks, maintaining both low forgetting and high plasticit
• In the authors' experiments, DREE shows strong generalization to unseen benchmark instances and is robust across different base solvers, highlighting its versatility and effectiveness in various problem scenarios.

Motivation and problem setting

Neural solvers for vehicle routing problems (VRPs) are almost universally trained under a stationary-task assumption: a fixed, pre-defined instance distribution supplies millions of training instances over hundreds of epochs, after which the parameters are frozen. Existing lifelong learning extensions relax this only partially — they assume periodically stationary tasks, where each task remains fixed for long enough to be learned thoroughly and to yield high-quality task-specific knowledge (stored models or behaviors) for later consolidation (Pei et al., 26 Sep 2025). This paper identifies and formalizes a more demanding regime: the continually drifting scenario, in which the task distribution changes at every learning time step, so that no task ever receives sufficient training. The motivating examples are practical: in urban delivery, customer-location distributions and traffic conditions shift gradually with population migration and construction, while historical patterns re-emerge stochastically (holidays, road restoration).

The authors formulate the setting as lifelong reinforcement learning over a drifting MDP distribution. Each instance $p$ induces an instance-conditioned MDP $\mathcal{M}(p)$; a task $P_t$ defines a distribution $\mathbf{D}_t$ over such MDPs, with $\mathbf{D}_t \neq \mathbf{D}_{t'}$ for all $t \neq t'$. The drift studied is restricted to problem scale and node-coordinate distribution, leaving action space, reward, and transition dynamics unchanged — an assumption that bounds the scope of the formulation.

The DREE framework

The proposed method, Dual Replay with Experience Enhancement (DREE), rests on the observation that when per-task training is scarce, neither instances nor behaviors alone suffice as replay experience; they carry complementary information, and both are of low quality unless actively refined. DREE maintains three mechanisms:

• Problem instance replay (PIR): buffered instances are re-solved by the current solver and trained on with the base solver's standard DRL loss. Unlike prior work that regenerates instances from stored task distributions, DREE replays raw buffered instances, so it applies even when instance generation is uncontrollable.
• Behavior replay (BR): a KL-based behavior-consolidation loss between the current policy's action distributions at buffered states and the stored behaviors, weighted by confidence as in LLR-BC (Pei et al., 26 Sep 2025).
• Experience enhancement (EE): whenever PIR produces a better objective value $f(\theta,p) < f_b$ on a buffered instance, the stored trajectory and objective are overwritten. BR therefore always imitates best-so-far behaviors, directly countering the misleading-gradient problem caused by low-quality experiences from insufficient per-task training.

Experiences are buffered at batch level via reservoir sampling, giving each seen experience equal retention probability. PIR is scheduled adaptively: the interval $N$ shrinks toward a lower bound when many replayed instances improve ($M^+/M$ large), and grows otherwise, balancing plasticity against stability. The total loss is $$\mathcal{L}_{\mathrm{DRL}} + \alpha\,\mathcal{L}_{\mathrm{BR}} + \beta\,\mathcal{L}_{\mathrm{PIR}}$$, with $\mathcal{M}(p)$0, $\mathcal{M}(p)$1.

Experimental setup

No dataset existed for continual drift, so the authors construct one from the six principal tasks of LLR-BC (varying node distributions and scales 20/50/100), assigning them to epochs 0 through $\mathcal{M}(p)$2 and interpolating linearly between consecutive principal tasks to form intermediate tasks. Training runs $\mathcal{M}(p)$3 epochs of 128 batches each, so each principal task receives only 4,096 training instances — orders of magnitude fewer than the millions used in one-off or periodically stationary studies. The buffer retains roughly 0.2% of all generated instances. For fairness, baselines run extra batches so that all methods perform the same number of solving episodes. Evaluation uses average performance (AP), forgetting metrics (AFB, AMFB), and best plasticity (ABPl), plus generalization tests on TSPLIB/CVRPLIB.

Results

The headline comparison shows DREE achieving the best AP among lifelong methods on both CVRP (3.11 vs. 4.19 for LLR-BC and 6.16 for Li) and TSP (1.21 vs. 1.36 and 4.43), with the smallest variance across five task orders — and performing close to MT-Ref, an idealized multi-task oracle trained jointly on all tasks:

Method	CVRP AP	CVRP ABPl	TSP AP	TSP ABPl
MT-Ref (oracle)	3.04	–	0.91	–
Fine-tuning	8.75	3.28	1.97	1.11
Li	6.16	6.07	4.43	4.42
LLR-BC	4.19	3.09	1.36	1.19
DREE	3.11	2.88	1.21	1.16

A notable contrast emerges in the stability–plasticity trade-off: Li attains near-zero forgetting but sacrifices plasticity severely (ABPl of 6.07 on CVRP, worse than fine-tuning's 3.28), whereas DREE keeps forgetting low (AFB 0.23 on CVRP) while retaining strong plasticity. On unseen benchmark instances, DREE generalizes well — 13.78% mean gap on CVRPLIB versus 35.57% for LLR-BC, and it slightly beats the MT-Ref oracle on CVRPLIB (13.78 vs. 12.92 is actually marginally worse; on individual instances DREE often matches or exceeds it). On TSPLIB, DREE achieves the lowest mean gap (18.77%).

Two further results strengthen the claims. First, DREE transfers back to the periodically stationary scenario, matching or slightly exceeding LLR-BC there (CVRP AP 2.30 vs. 2.55). Second, replacing the base solver with Omni or INViT preserves the ranking — e.g., with INViT on CVRP, DREE reaches AP 21.93 versus 23.83 for LLR-BC — indicating the gains stem from the framework rather than idiosyncrasies of POMO. Ablations confirm each component matters: removing PIR, BR, or EE degrades AP on both problems (e.g., nPIR raises CVRP AP from 3.16 to 4.07), with EE contributing the smallest but still measurable share. Hyperparameter sweeps show only minor fluctuations, suggesting robustness to buffer size, loss coefficients, and the PIR interval range.

Limitations and open questions

Several caveats qualify these results. The drift model is deliberately narrow: only scale and node-distribution drift are considered, while reward functions and transition dynamics remain fixed, so the formulation does not yet cover drifting constraints or objectives. The dataset itself is synthetic, built by linear interpolation between principal tasks; whether real-world drift follows such smooth mixtures is unverified. The adaptive PIR interval range (LB–UB = 1–4) is set heuristically, and although sensitivity analysis suggests insensitivity, no principled selection rule is given. DREE also treats all time steps equally, ignoring which steps accumulate the most significant drift; the authors note that selectively emphasizing distinctive steps could improve efficiency, and suggest connections to dynamic optimization and online learning that remain unexplored. Finally, the continual-drift evaluation necessarily restricts testing to the shared principal tasks, since intermediate tasks differ across orders — a reasonable but partial view of overall performance.

Conclusion

This paper extends lifelong learning for neural VRP solvers into a continually drifting regime where sufficient per-task training is unavailable, formalizes the setting as lifelong RL over drifting MDP distributions, and introduces DREE, which combines instance-level and behavior-level replay with an experience-enhancement loop that continuously upgrades buffered knowledge. Empirically, DREE outperforms existing lifelong VRP solvers under continual drift, remains competitive in the periodically stationary scenario, generalizes to unseen benchmarks, and is solver-agnostic. The central empirical lesson is that in low-resource lifelong learning, buffering both instances and behaviors — and actively refining their quality — is essential for effective replay.