Conservative Discrete Quantile Actor-Critic

• CDQAC is an offline RL algorithm that employs distributional quantile critics combined with conservative Q-learning to tackle discrete combinatorial scheduling in job-shop and flexible job-shop environments.
• It integrates a dueling architecture with a delayed discrete actor and entropy regularization to separate state value from advantage, ensuring stable policy updates.
• Empirical experiments show CDQAC significantly improves scheduling performance over traditional heuristics, especially when trained on diverse, low-quality trajectories.

Searching arXiv for the primary paper and closely related scheduling/RL methods to ground the article with citations. Conservative Discrete Quantile Actor–Critic (CDQAC) is an offline, value-based reinforcement learning algorithm for discrete combinatorial scheduling, introduced for the Job-Shop Scheduling Problem (JSP) and the Flexible Job-Shop Scheduling Problem (FJSP). It is designed to learn directly from historical or simulated trajectories generated by suboptimal heuristics, without online interaction during training, while still improving upon the behavior policies that produced the dataset. The method combines a distributional quantile critic over discrete machine–operation actions, a Conservative Q-Learning (CQL) regularizer for offline stability, and a delayed discrete actor with entropy regularization. In the reported experiments, CDQAC learns effective scheduling policies from Priority Dispatching Rules (PDR), genetic algorithm (GA), and random datasets, and it exhibits the notable empirical property that random, low-quality but diverse data can outperform higher-quality yet less diverse training data (Remmerden et al., 12 Sep 2025).

1. Problem formulation and offline RL motivation

CDQAC is formulated for scheduling environments in which there are $n$ jobs $\mathcal{J}$ and $m$ machines $\mathcal{M}$. Each job $J_i \in \mathcal{J}$ has an ordered sequence of operations

$$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$

with precedence constraints requiring $O_{i,j}$ to begin only after $O_{i,j-1}$ completes. In JSP, each operation is assigned to one fixed machine; in FJSP, each operation can be processed by any machine in a compatible set $\mathcal{M}_{i,j} \subseteq \mathcal{M}$, with processing times $p^k_{i,j} > 0$ for compatible pairs. The optimization objective is to minimize makespan,

$\mathcal{J}$0

The scheduling process is cast as an MDP

$\mathcal{J}$1

A state $\mathcal{J}$2 encodes the current partial schedule, including available operations $\mathcal{J}$3 and currently free machines $\mathcal{J}$4. The feasible action space at decision step $\mathcal{J}$5 is

$\mathcal{J}$6

Choosing an action schedules an operation on a machine, updates machine availability and precedence, advances time to the next event, and yields a new state. Reward is shaped as the negative increment in partial makespan; the paper notes that the typesetting is garbled, but explicitly states that the cumulative return is proportional to $\mathcal{J}$7. The discount factor is set to $\mathcal{J}$8, which the paper describes as standard in finite-horizon scheduling (Remmerden et al., 12 Sep 2025).

The offline RL setting uses a static dataset

$\mathcal{J}$9

collected from behavior policies $m$0, including non-Markovian heuristics such as PDRs, GA, or random choice. The motivation is operational: online RL approaches such as L2D, DANIEL, and FJSP-DRL require millions of simulator interactions, use simulators that may abstract away real-world complexities such as machine breakdowns and delays, and begin from random initialization, which the paper identifies as a source of poor sample efficiency. CDQAC is positioned as an offline alternative that learns value functions from static logs and can “stitch” good trajectory segments, thereby generalizing beyond suboptimality rather than merely cloning the behavior policy.

2. Core algorithmic structure

CDQAC combines three components: a distributional critic, a conservative value regularizer, and a delayed discrete-action actor. Its action space is discrete and consists of machine–operation pairs. The critic models the full return distribution $m$1 rather than only its expectation, and the actor defines a categorical policy over feasible actions.

The critic follows a QR-DQN-style quantile parameterization. For $m$2 quantiles,

$m$3

and the return distribution is represented as

$m$4

There are two critic heads, $m$5 and $m$6, and the algorithm defines

$m$7

using the element-wise minimum to reduce overestimation in a manner analogous to double Q-learning or discrete SAC. The scalar value used by the actor and the conservative regularizer is the mean of the quantile distribution,

$m$8

The conservative component is the discrete CQL penalty,

$m$9

which is added to the critic loss. The stated purpose is to suppress overestimation for out-of-distribution actions that are absent or weakly represented in the offline dataset. In the scheduling setting, this is particularly relevant because many feasible machine–operation pairs are never selected by a given heuristic, so naive offline bootstrapping can mis-rank them.

The actor is a discrete categorical policy $\mathcal{M}$0 over feasible pairs $\mathcal{M}$1. It is updated less frequently than the critic. This delayed update schedule is an explicit design choice: the critic is trained every step, whereas the actor is updated every $\mathcal{M}$2 steps, with $\mathcal{M}$3 in the main experiments. The paper directly compares this design to TD3’s delayed policy update and attributes the stabilization of offline distributional learning to allowing the critic to track the Bellman target and CQL regularizer before the actor reacts to its estimates (Remmerden et al., 12 Sep 2025).

3. Critic, actor, and network parameterization

The critic uses a distributional Bellman target,

$\mathcal{M}$4

where $\mathcal{M}$5 is a target network updated by Polyak averaging,

$\mathcal{M}$6

with $\mathcal{M}$7. The temporal-difference component of the critic objective is the asymmetric Huber quantile loss from QR-DQN,

$\mathcal{M}$8

The full critic objective is

$\mathcal{M}$9

with $J_i \in \mathcal{J}$0 or $J_i \in \mathcal{J}$1 in experiments. The actor objective is

$J_i \in \mathcal{J}$2

with an entropy regularizer intended to prevent premature collapse to a single action.

Both actor and critic use the Dual Attention Network (DAN) encoder from DANIEL. The encoder produces operation embeddings $J_i \in \mathcal{J}$3, machine embeddings $J_i \in \mathcal{J}$4, and a global embedding

$J_i \in \mathcal{J}$5

For each feasible machine–operation pair, the actor uses

$J_i \in \mathcal{J}$6

as input to an MLP that produces the action logit. Pair features include $J_i \in \mathcal{J}$7, multiple processing-time ratios, and joint waiting time; operation and machine features include workload- and availability-related descriptors.

The critic adopts a dueling architecture specialized to scheduling. A value stream $J_i \in \mathcal{J}$8 depends only on the global embedding, and an advantage stream $J_i \in \mathcal{J}$9 depends on operation, machine, pair, and global features: $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$0 This subtraction of the mean advantage over available actions enforces identifiability of value and advantage. The implementation details reported in the appendix include $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$1 quantiles, DAN output dimensions $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$2, two hidden layers of 64 units for both critic and policy MLPs, Adam optimization, normalized input features, and GPU training (Remmerden et al., 12 Sep 2025).

4. Offline training procedure and datasets

Training is performed entirely on a static dataset $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$3. The loop samples mini-batches of transitions, computes next actions under the current policy, forms target quantiles, updates the critic on the combined quantile-TD and CQL loss, updates the actor every $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$4 steps, and then updates the target critic by Polyak averaging. The appendix reports indicative hyperparameters: $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$5, $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$6, $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$7, $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$8, $$\mathcal{O}_i = \{O_{i,1}, O_{i,2}, \dots, O_{i,n_i}\},$$9, $O_{i,j}$0, $O_{i,j}$1, batch size $O_{i,j}$2, and training steps $O_{i,j}$3.

The reported experimental setting spans generated and benchmark JSP and FJSP instances. For FJSP, training uses 500 generated instances per size for $O_{i,j}$4, $O_{i,j}$5, and $O_{i,j}$6; each job has between $O_{i,j}$7 and $O_{i,j}$8 operations, with processing times $O_{i,j}$9. Evaluation includes generated instances at $O_{i,j-1}$0, $O_{i,j-1}$1, $O_{i,j-1}$2, $O_{i,j-1}$3, and $O_{i,j-1}$4, as well as Brandimarte and Hurink benchmarks. JSP training uses 500 generated instances for $O_{i,j-1}$5 and $O_{i,j-1}$6, and evaluation uses the Taillard and Demirkol benchmarks.

Four offline datasets are constructed from heuristic behavior policies. The PDR dataset uses four job selection rules for JSP—MOR, LOR, MWR, and LWR—and 16 FJSP combinations obtained by pairing those job rules with SPT, LPT, EST, and LST machine rules. The GA dataset is built from a two-stage GA with population 200 and 100 generations, retaining the entire final population of 200 solutions per instance. The Random dataset uses 100 trajectories per instance, each generated by selecting a uniformly random feasible machine–operation pair at each decision step. A combined PDR-GA dataset is the union of PDR and GA solutions, with duplicates removed. The paper emphasizes the contrast that GA data are higher quality but less diverse, whereas random data are lower quality but highly diverse.

Performance is evaluated by the optimality gap,

$O_{i,j-1}$7

where $O_{i,j-1}$8 is the makespan obtained by method $O_{i,j-1}$9, and $\mathcal{M}_{i,j} \subseteq \mathcal{M}$0 is the optimal or best-known upper bound. Results are reported as mean $\mathcal{M}_{i,j} \subseteq \mathcal{M}$1 standard deviation over instances and seeds. Evaluation is conducted both in greedy mode, selecting $\mathcal{M}_{i,j} \subseteq \mathcal{M}$2, and in sampling mode, generating multiple schedules and reporting the best makespan (Remmerden et al., 12 Sep 2025).

5. Empirical behavior and comparative results

Across FJSP evaluation sets, CDQAC is reported to consistently outperform both Offline-LD variants—maskable QR-DQN and discrete maskable SAC—across all dataset types and in both greedy and sampling evaluation. The paper summarizes average FJSP greedy gaps for CDQAC in the range $\mathcal{M}_{i,j} \subseteq \mathcal{M}$3 to $\mathcal{M}_{i,j} \subseteq \mathcal{M}$4, with the best result on the Random dataset at $\mathcal{M}_{i,j} \subseteq \mathcal{M}$5; corresponding sampling gaps are approximately $\mathcal{M}_{i,j} \subseteq \mathcal{M}$6 to $\mathcal{M}_{i,j} \subseteq \mathcal{M}$7, again with the best results on Random and PDR-GA. By comparison, the summarized Offline-LD ranges are $\mathcal{M}_{i,j} \subseteq \mathcal{M}$8–$\mathcal{M}_{i,j} \subseteq \mathcal{M}$9 for mQRDQN greedy, about $p^k_{i,j} > 0$0–$p^k_{i,j} > 0$1 for d-mSAC greedy, about $p^k_{i,j} > 0$2–$p^k_{i,j} > 0$3 for mQRDQN sampling, and about $p^k_{i,j} > 0$4–$p^k_{i,j} > 0$5 for d-mSAC sampling.

Against online RL baselines on FJSP benchmarks, CDQAC trained only on random data from $p^k_{i,j} > 0$6 or $p^k_{i,j} > 0$7 is reported to outperform FJSP-DRL and DANIEL in most settings. On the Brandimarte mk benchmark in greedy mode with $p^k_{i,j} > 0$8 training, the reported gaps are $p^k_{i,j} > 0$9 for FJSP-DRL, $\mathcal{J}$00 for DANIEL, and $\mathcal{J}$01 for CDQAC. On Hurink rdata sampling, DANIEL slightly outperforms CDQAC, $\mathcal{J}$02 versus $\mathcal{J}$03, which the paper identifies as an exception. A Wilcoxon test is reported as showing that the improvement over DANIEL is significant, with $\mathcal{J}$04.

On generated FJSP instances, CDQAC is described as competitive or superior at $\mathcal{J}$05 and $\mathcal{J}$06, and it generalizes effectively to larger unseen instance sizes. When trained only on $\mathcal{J}$07, the reported results on $\mathcal{J}$08 and $\mathcal{J}$09 are stronger than DANIEL in both greedy and sampling modes. For example, on $\mathcal{J}$10, the greedy gap is $\mathcal{J}$11 for DANIEL versus $\mathcal{J}$12 for CDQAC, and the sampling gap is $\mathcal{J}$13 for DANIEL versus $\mathcal{J}$14 for CDQAC. The paper summarizes this as achieving at least comparable generalization to the best online baselines while using half the training instances and no environment interaction.

On JSP benchmarks, CDQAC trained on random $\mathcal{J}$15 instances is reported to outperform classic dispatching rules, L2D, DANIEL, and Offline-LD. On the Taillard benchmark in greedy mode, the mean gaps reported are $\mathcal{J}$16 for L2D, $\mathcal{J}$17 for DANIEL, $\mathcal{J}$18 for Offline-LD, and $\mathcal{J}$19 for CDQAC; in sampling mode, DANIEL reports $\mathcal{J}$20 and CDQAC $\mathcal{J}$21. For Demirkol, the paper states that CDQAC also improves over these baselines, while noting that CP/MIP can still outperform it at high computation time. The Wilcoxon test over Taillard is reported as significant, with $\mathcal{J}$22.

A central empirical claim is that CDQAC improves upon the heuristics that generated its training data. On FJSP in sampling mode over all sizes, the behavior policy for the Random dataset has a gap of $\mathcal{J}$23, whereas CDQAC trained on Random yields $\mathcal{J}$24 in sampling and $\mathcal{J}$25 in greedy evaluation. For GA, the best behavior policy reports $\mathcal{J}$26, while CDQAC trained on GA achieves about $\mathcal{J}$27 in sampling and $\mathcal{J}$28 in average greedy evaluation. For PDR datasets, where the heuristic performance is much worse, the paper describes the improvement as dramatic, with the best PDR around $\mathcal{J}$29–$\mathcal{J}$30 versus CDQAC around $\mathcal{J}$31 (Remmerden et al., 12 Sep 2025).

6. Ablations, interpretation, and limitations

The ablation studies isolate the effect of the quantile critic, the dueling architecture, the delayed actor update, and dataset type. Replacing the quantile critic with a standard DQN-style critic while retaining the dueling architecture reduces performance slightly; the best configuration is reported as Quantile + Dueling, with generated greedy gap $\mathcal{J}$32 and benchmark greedy gap $\mathcal{J}$33, compared with approximately $\mathcal{J}$34–$\mathcal{J}$35 and $\mathcal{J}$36–$\mathcal{J}$37 for non-quantile variants. Dueling improves both generated and benchmark performance for quantile and non-quantile critics alike, supporting the separation of state value and action-dependent advantage.

The delay hyperparameter $\mathcal{J}$38 is also influential. The appendix reports that $\mathcal{J}$39, corresponding to no delay, yields the worst performance and higher variance, while increasing $\mathcal{J}$40 to $\mathcal{J}$41 steadily improves both greedy and sampling gaps and reduces variance. This is consistent with the paper’s hypothesis that the critic should stabilize before the actor is updated.

The most distinctive empirical result concerns dataset quality versus diversity. The paper explicitly states that the Random dataset yields the best performance for both CDQAC and Offline-LD, despite contradicting common offline RL wisdom in which high-quality expert data usually dominate. Its analysis attributes this to higher state–action coverage (SACo) and more diverse counterfactual examples. The paper further states that CDQAC’s performance correlates strongly with coverage and the presence of poor solutions, which provide “what not to do” signals, rather than with average solution quality alone. This suggests that, in discrete combinatorial scheduling, offline value learning may benefit more from broad support over feasible action configurations than from narrowly concentrated high-quality trajectories.

The method is also described as highly sample efficient. Training on only 10–20 instances, or 2–4% of the full Random dataset, still yields near-full performance on FJSP benchmarks. For $\mathcal{J}$42, the greedy gap changes only from about $\mathcal{J}$43 with 10 instances to about $\mathcal{J}$44–$\mathcal{J}$45 with 500 instances, and standard deviations remain small except at extremely small dataset sizes. The paper interprets this as substantial data efficiency relative to online RL methods that train on 1000–10,000 instances.

The stated limitations are threefold. First, scalability degrades on larger action spaces: when trained directly on $\mathcal{J}$46 FJSP instances, performance is weaker, and the paper attributes this to the difficulty of Q-learning with action spaces of up to 200 machine–operation pairs. Second, CDQAC depends on coverage: if historical data are narrow and biased, such as trajectories from a single deterministic high-quality heuristic, generalization may be weaker. Third, although training requires no simulator, distributional GNN-based offline RL remains compute-intensive. The paper does not present new formal theorems; its theoretical positioning relies on existing CQL results about conservative lower-bounding, established benefits of distributional RL for value approximation, and known challenges of large discrete action spaces. Future directions proposed in the paper include improving scalability in large discrete action spaces, extending the method to other combinatorial optimization problems such as TSP and VRP, and combining offline CDQAC with limited online fine-tuning when a simulator or digital twin is available (Remmerden et al., 12 Sep 2025).