TransGP: Task-Conditioned Transformer-Guided Genetic Programming for Multitask Dynamic Flexible Job Shop Scheduling

Abstract: Hyper-heuristics have become a popular approach for solving dynamic flexible job shop scheduling (DFJSS) problems. They use gradient-free optimization techniques like Genetic Programming (GP) to evolve non-differentiable heuristics. However, conventional GP methods tend to converge slowly because they rely solely on evolutionary search to find good heuristics. Existing multitask GP methods can solve multiple tasks simultaneously and speed up the search by transferring knowledge across similar tasks. But they mostly exchange heuristic building blocks without truly generating heuristics conditioned on task information. In this paper, we aim to accelerate convergence and enable task-specific heuristic generation by incorporating a task-conditioned Transformer model. The Transformer works in two ways. First, it learns the distribution of elite heuristics, biasing the search toward promising regions of the heuristic space. Second, through conditional generation, it produces heuristics tailored to specific tasks, allowing the model to handle multiple scheduling tasks at once and improving overall optimization efficiency. Based on these ideas, we propose TransGP, a Task-Conditioned Transformer-Guided GP framework. This evolutionary paradigm integrates generative modeling with GP, enabling efficient multitask heuristic learning and knowledge transfer. We evaluate TransGP on a range of DFJSS scenarios. Experimental results show that TransGP consistently outperforms multitask GP baselines, widely used handcrafted heuristics, and the pure Transformer model, achieving faster convergence, superior solution quality, and enhanced robustness.

• The paper introduces TransGP, which uses task-conditioned Transformer decoders as grammar-constrained mutation operators so genetic programming can generate specialized heuristics while transferring useful structures across scheduling tasks.
• TransGP achieves the best test performance on all nine evaluated tasks, significantly outperforming standard GP, multitask GP, and handcrafted rules while converging faster and often producing smaller heuristic trees.
• The paper shows that evolutionary refinement stabilizes learned generation: PureTrans has occasional strong solutions but far greater variance, whereas TransGP delivers consistently reliable performance across dynamic flexible job shop environments.

Motivation and problem setting

Dynamic flexible job shop scheduling (DFJSS) requires assigning operations to alternative machines and sequencing them under stochastic job arrivals and shop-floor disturbances. Genetic programming (GP) hyper-heuristics are attractive for this problem because they evolve explicit, interpretable expression trees for sequencing and routing decisions, but they converge slowly: crossover and mutation are essentially random perturbations that frequently destroy the deeply nested conditional subtrees and nonlinear feature interactions characteristic of high-performing heuristics, producing many low-fitness offspring and wasting evaluations. Existing multitask GP methods accelerate search by exchanging building blocks across related tasks, but they do not generate heuristics conditioned on task information.

TransGP addresses both deficiencies by training a task-conditioned Transformer decoder on a corpus of elite heuristics and using it as the mutation operator inside GP. The Transformer serves two roles: it learns the distribution of elite heuristic structures, biasing mutation toward promising regions, and its task embeddings enable conditional generation, so a single model produces heuristics specialized to each DFJSS environment (differing in machine count and utilization) while transferring structural knowledge across tasks.

Method

Heuristics are linearized into prefix-order token sequences with START/END delimiters, giving a discrete sequence amenable to autoregressive modeling. Two Transformer decoders are trained—one for sequencing rules, one for routing rules—on cross-entropy over next-token prediction. Task-specific conditional parameters are linearly projected and added to token and positional embeddings at every position, so generation is conditioned on the target task.

During evolution, each individual carries a task tag and is evaluated only on that task. Mutation proceeds by optionally switching the task tag (with probability $p_t$), selecting a mutation point, and autoregressively generating a replacement subtree under the conditioning embedding. A stack-based grammar mask enforces syntactic validity: invalid tokens receive logits of $-\infty$, and sampling uses a temperature-controlled softmax. This guarantees well-formed prefix expressions while allowing temperature $\gamma$ to modulate exploration.

The authors argue the computational overhead is modest: offline Transformer training is performed once per scenario on 3,600 elite heuristics; online inference cost per mutation is $O(LdQ^2)$ with $L$ bounded by tree depth, and is negligible relative to simulation-based fitness evaluation, which dominates runtime.

Experimental design

Evaluation uses a discrete-event simulator with Poisson job arrivals, heterogeneous machines, and transportation times. Three scenarios are defined, each containing three tasks spanning objectives Fmax, Fmean, or Tmean crossed with utilization levels 0.75–0.95 and 6–10 machines. All methods use population size 600, 50 generations, tournament selection, elitism, and mutation-only variation (crossover disabled to isolate the effect of guided mutation). Comparisons include standard GP, TGP (a multitask GP extension with probabilistic task-tag mutation), eight handcrafted rule combinations (SPT/FIFO/LPT/EDD × NIQ/WIQ), and PureTrans (Transformer generation without evolutionary refinement). Results are averaged over 30 independent runs with Wilcoxon rank-sum tests at $p<0.05$.

Main results

TransGP achieves the best test performance on all nine tasks. Representative figures: in scenario 3, TransGP reaches 234.87 versus GP's 248.32 and TGP's 241.02 on task 1, and 278.71 versus 299.04 (GP) and 292.57 (TGP) on task 3. In scenarios 2 and 3, improvements over both baselines are statistically significant on every task; in scenario 1, TransGP significantly beats GP everywhere and matches or beats TGP. Convergence curves show faster and lower-variance improvement, which matters under limited evaluation budgets.

Against handcrafted rules, TransGP is significantly better on every task—for example, 566.21 versus roughly 655–681 for the best handcrafted combinations in scenario 2 task 1. The PureTrans comparison yields an important negative result for pure generative approaches: although PureTrans occasionally produces excellent rules (its best-of-30 runs sometimes beat TransGP), its mean performance is catastrophically unstable, with means such as 2,424.80 (std 2,618.89) against TransGP's 1,276.79 (std 33.11) on scenario 1 task 1. This indicates that generative models alone cannot reliably produce high-quality heuristics, and that evolutionary filtering and refinement are necessary to stabilize learned generation—a claim the paper supports directly through the variance statistics.

A sensitivity analysis over decoding temperatures ($\gamma \in \{0.5, 0.8, 1.0, 1.2, 1.5\}$) finds no consistent effect on performance; only isolated significant differences appear, mostly at $\gamma = 1.5$, which the authors recommend using cautiously. This robustness to temperature simplifies deployment but also suggests the method's performance is not highly sensitive to the exploration–exploitation balance in sampling.

Heuristic structure and interpretability

TransGP evolves substantially smaller trees than GP and TGP in scenarios 1 and 2 (e.g., 52.87 nodes versus 80.60 for GP on scenario 1 task 1), indicating that guided mutation suppresses bloat early in the search. In scenario 3 the size advantage disappears on two of three tasks, a task dependency the paper acknowledges without full explanation.

Structural analysis shows routing rules are larger and more complex than sequencing rules, with function-to-terminal ratios around 0.90–0.94 across all rule types. Function and terminal usage heatmaps reveal shared cores (+, , min; PT, MWT, NIQ, WKR) alongside task-specific preferences (e.g., * used more in task 2). Cross-task similarity analysis using Jaccard and size similarity shows that structurally aligned tasks share logic (routing Jaccard 0.727 between tasks 2 and 3) while divergent tasks produce distinct rules (sequencing Jaccard 0.364 between tasks 2 and 3)—evidence that conditional generation balances transfer with specialization. Five recurring subtrees were identified across rules, the most frequent being /((MWT, WIQ), NIQ), supporting the claim that the generative operator reuses effective modular primitives rather than searching blindly.

Analysis of the trained routing Transformer's learned distribution shows the top-10 structural patterns cover 16.07% of observed subtrees, concentrated on terminals like WIQ plus composites like *(MWT, TIS) and *(WKR, WIQ). A traced mutation exhibits high confidence (>0.85 probability) at every generation step, e.g., 0.9955 for WKR after selecting /, demonstrating learned semantic compatibility rather than random sampling. Across five offspring from one parent, average pattern similarity to training data is 74.29% (±21.00%), ranging from pure recombination (100%) to partially novel compositions (57.14%), consistent with a controlled exploitation–exploration balance.

Limitations and open questions

Several constraints qualify the results. The elite-heuristic dataset is constructed from GP runs on the same three-task scenarios used for evaluation, so the degree to which the pretrained Transformer generalizes to genuinely unseen environments or to heuristics from different terminal/function sets remains untested—the framework permits external data sources in principle, but this is not demonstrated. Crossover is disabled throughout, so the interaction of Transformer-guided mutation with crossover-based variation is unexplored. The heuristic-size advantage is inconsistent across scenarios, and the mechanism behind the scenario 3 exception is not analyzed. Evaluation relies on a single simulation replication per generation with randomized seeds rather than multiple replications, a common but acknowledged trade-off in GP-based DFJSS work. Finally, the experiments cover single-objective tasks only; multi-objective formulations and richer task parameterizations are explicitly left open, as is validation outside the DFJSS problem class.

Conclusion

TransGP integrates task-conditioned Transformer generation into GP as a learned, semantically aware mutation operator for multitask DFJSS. Across nine tasks spanning three objectives and varying utilization and machine counts, it consistently outperforms standard GP, multitask GP with task-tagged mutation, handcrafted dispatching-rule combinations, and unrefined Transformer generation, while evolving smaller and more interpretable heuristics. The comparison with PureTrans is particularly instructive: generative modeling alone is unreliable, whereas coupling it with evolutionary selection yields both quality and stability. The main open questions concern generalization beyond the training task family, integration with crossover, and extension to multi-objective settings.