Algorithmic Control Policy Training

• The paper demonstrates that embedding symbolic planners within neural architectures significantly boosts sample efficiency and generalization in control tasks.
• It details a hybrid methodology combining convolutional cost prediction with a time-dependent shortest-path solver through differentiable relaxation.
• Empirical results show superior performance in structured environments, reducing the generalization gap and achieving optimal path quality versus traditional methods.

An algorithmic approach for training control policies encompasses the formalization of the control learning problem, architectural designs that integrate neural networks and symbolic or algorithmic computation, specialized training methodologies that enable gradient-based optimization through non-differentiable combinatorial modules, and rigorous evaluation to quantify generalization and efficiency. The recent literature establishes that classical deep learning methods often underperform on tasks requiring fast combinatorial generalization, particularly for control problems with rich planning structure and combinatorial complexity in environment variations. Embedding algorithmic solvers—such as shortest-path planners—within trainable policy architectures leads to significantly improved sample efficiency and generalization capabilities in offline imitation learning and related settings (Vlastelica et al., 2021).

1. Formal Problem Setting: Structured Markov Decision Process

Algorithmic policy training is best illustrated in structured MDP families where the agent interacts with environments that possess explicit combinatorial structure:

• State and Action Spaces: Consider a finite, discrete, deterministic, goal-conditioned MDP (termed ddgcMDP), where the state space $S = S^a \times S^e$ factorizes into agent-controlled and environment-autonomous components. Actions operate on a finite set (e.g., grid moves), and transitions are one-hot deterministic.
• Reward Function: Rewards are sparse and goal-conditioned, with highest return for reaching a specified terminal goal state. Collisions or illegal actions receive an infinite penalty.
• Expert Supervision: Offline datasets consist of demonstration trajectories $\tau_0,\dotsc,\tau_T$ sampled from an expert policy $\pi^*$. The central objective is to learn a policy $\pi_\theta$ that imitates the expert while generalizing to environment configurations not encountered during training.

This setting sharply distills the challenge of combinatorial generalization—solving previously unseen combinations of environment elements given limited exposure during training (Vlastelica et al., 2021).

2. Neuro-Algorithmic Policy Architecture

A neuro-algorithmic policy is characterized by its fusion of neural networks for perception/cost prediction and a symbolic planner for structured decision-making:

• Perceptual Backbone: A convolutional neural network $\phi_\theta$ ingests recent observations (e.g., image frames) and outputs both:
  1. A time-indexed, non-negative cost tensor $C \in \mathbb{R}_+^{T \times |V|}$ ("vertex costs"),
  2. Distributions over start- and goal-vertices within a fixed planning graph $G=(V,E)$.

• Symbolic Planning Module: A time-dependent shortest-path (TDSP) solver receives $(C, v_s, v_g)$ and returns an optimal indicator tensor $Y \in \{0,1\}^{T \times |V|}$ encoding the path from $v_s$ to $v_g$ over $T$ steps in a time-expanded graph $G^*$.

• Action Mapping: The first edge in $Y$ is mapped via a fixed function $\psi$ back into a discrete action.

The resulting architecture supports end-to-end differentiable training entirely via backpropagation through both neural and symbolic components (see Section 4). The overall dataflow is summarized as follows:

function POLICY(I_{t–1}, I_t): C, logits_s, logits_g ← CNN(I_{t–1}, I_t) v_s ← argmax(logits_s), v_g ← argmax(logits_g) Y ← TDSP(C, v_s, v_g) # Dijkstra on time-expanded graph return ψ(first_edge(Y))

(Vlastelica et al., 2021)

3. Embedded Shortest-Path Solver: Combinatorial Layer

The time-dependent shortest-path module forms the core of combinatorial generalization:

• Graph Construction: The task environment is mapped to a fixed underlying graph $G=(V,E)$ (e.g., gridworld) and a corresponding time-expanded graph $G^*=(V^*,E^*)$.
• Optimization Problem: At every decision step, the TDSP computes

$$\operatorname{TDSP}(C, v_s, v_g) = \arg\min_{Y \in \mathrm{Adm}(G, v_s, v_g)} \langle Y, C \rangle$$

subject to $Y$ encoding a legal path from $(v_s,1)$ to $(v_g,T)$ and each $C_i^t$ representing vertex-wise time-dependent costs.

• Boundary and Invalidity Representation: Any illegal action or boundary-violating node is assigned $C^t = \infty$ to disallow plan selection.
• Continuous Relaxation: Training employs a convex relaxation of the combinatorial polytope, enabling implicit differentiation.

This embedded planning layer shifts the explicit combinatorial reasoning from the neural network to a solver with algorithmic guarantees and computational efficiency.

4. End-to-End Training via Blackbox Differentiation

Gradient-based training is realized by differentiating through combinatorial solvers:

• Loss Terms: The composite training loss includes:
  1. Path Loss $J^C(\theta)$: The Hamming distance between planned paths and expert-provided ground truth.
  2. Start/Goal Loss $J^P(\theta)$: Cross-entropy between predicted and expert start/goal vertices.
  3. Additional Margin Regularization: Margins are added to cost predictions for ground-truth vs. alternate path entries.

• Blackbox Differentiation: The mapping $(C \mapsto Y)$ is inherently piecewise constant. To enable meaningful gradients, training employs a smoothed, piecewise-linear interpolation $f_\lambda(C)$, conducting an additional forward pass:

  1. Perturb $C$ by the gradient signal scaled by $\lambda$.
  2. Re-solve TDSP with perturbed costs.
  3. Use finite-difference to estimate gradient w.r.t $C$:

  $$\partial L / \partial C \approx (Y_\lambda - Y) / \lambda$$

1. Backpropagate through neural predictions $\phi_\theta$. The hyperparameter $\lambda$ trades off gradient informativeness against solver calls (Algorithm 1 in (Vlastelica et al., 2021)).

• Parameter Localization: Only the cost-predicting neural module $\phi_\theta$ is trainable; the TDSP and $\psi$ mapping are fixed.

This methodology ensures that complex combinatorial decision structures remain differentiable and efficiently trainable using standard optimization routines.

5. Empirical Evaluation and Generalization Behavior

Rigorous empirical studies demonstrate key properties of algorithic policy training:

• Benchmark Environments: NAPs have been tested in environments such as Crash Jewel Hunt (dynamic obstacle avoidance), ProcGen Maze, Leaper (frogger analog), and Chaser (collect-and-avoid tasks), all of which require robust long-horizon planning and adaptivity to combinatorial task randomizations (Vlastelica et al., 2021).
• Sample Complexity and Generalization Gap:
  • NAPs achieve near-zero generalization gap after exposure to only 100–500 unique levels, compared to requirements of ≫10,000 levels for PPO/DrAC (model-free RL baselines).
  • In gridworld mazes, NAPs trained with 100 levels reach ≈80% success rate on 1,000 unseen test mazes; PPO trained with 200,000 step samples fails to match this success rate.
• Path Quality: NAP plans show lower mean and variance in episode length, indicating more optimal, direct solutions than purely model-free or imitation-learned policies.
• Sensitivity to Horizon: Tasks with highly dynamic or adversarial environments benefit from longer planning horizons. In static mazes, performance saturates for $T=1$.

Thus, the explicit transfer of combinatorial burden from neural nets to a fast symbolic solver results in orders-of-magnitude improved sample efficiency and generalization to novel combinations of environment elements.

Algorithm	Sample Complexity to <10% Gen. Gap	Test Success (Maze, 100 levels)	Path Optimality
NAP	~100–500 episodes	≈80%	Lower median,
PPO	≫10,000 episodes	<60%	Higher variance
DrAC	≫10,000 episodes	<60%	Suboptimal

6. Implications and Connections

The neuro-algorithmic policy paradigm illustrates that combinatorial generalization bottlenecks observed in deep RL and imitation learning for control can be alleviated via hybrid algorithmic architectures (Vlastelica et al., 2021). Key properties revealed include:

• Decoupling of Perception and Planning Complexity: Neural modules are strictly responsible for cost estimation and target localization, while the solver ensures correctness of combinatorics.
• Differentiability Guarantees: Blackbox differentiation techniques allow for seamless integration of non-differentiable planners in end-to-end learning.
• Planning-First Policy Architectures: This framework substantiates a trend toward modular policy designs, where symbolic solvers and learned perception modules are composed, inheriting both combinatorial soundness and data-adaptive cost modeling.

This architecture is not limited to shortest-path planning; it generalizes to any combinatorial routine (e.g., matching, constraint satisfaction) that admits either exact or relaxed differentiable implementations.

7. Limitations and Future Research

While neuro-algorithmic policy architectures demonstrate markedly improved generalization in structured combinatorial control tasks, certain limitations remain:

• Solver Bottleneck: The approach’s scalability is bounded by the complexity of the embedded combinatorial problem and the tractability of its relaxation for efficient differentiation.
• Domain Restriction: The method is directly applicable to tasks with explicit planning structure (e.g., gridworlds, routing), and may require adaptation for applications with less interpretable combinatorial structure.
• Transferability of Solvers: The construction of the underlying graph and assumptions about planning structure must match the environment; significant task shifts may require new solver instantiations.

Further research could examine integration with stochastic combinatorial planners, address scaling to larger state/action graphs, or extend to continuous-state planning via appropriate discretization or graph representations.

---

References:

Neuro-algorithmic policy architecture and results: (Vlastelica et al., 2021)