Agenda-Based Scheduling in DyNet

• Agenda-based scheduling is a dynamic batching mechanism that automatically groups compatible operations in a computation DAG using signature functions and priority queues.
• It leverages a greedy, priority-driven traversal with a 'cheap-op-first' heuristic to efficiently batch operations, achieving performance nearly comparable to manual batching.
• Empirical evaluations show significant speedups—in some cases up to 10×—and reduced overhead, highlighting its practical benefits for complex neural architectures.

Agenda-based scheduling in DyNet is an on-the-fly operation batching algorithm designed to maximize computational efficiency in dynamic computation graphs, typically found in flexible neural network toolkits. Rather than requiring the user to construct static batched graphs or explicitly organize manual batching strategies, the scheduler enables automatic grouping of batch-compatible operations at runtime. This mechanism achieves throughput comparable to manual batching while relieving the developer of batching logistics, especially in architectures where manual batching is challenging (Neubig et al., 2017).

1. Core Data Structures and Formalism

At the heart of agenda-based scheduling is a directed acyclic graph (DAG) representation of the computation, denoted as $G = (N, E)$, where $N$ is the set of nodes (each corresponding to an elemental operation, such as $\tanh$, matrix-vector multiply, or parameter lookups) and $E$ is the set of edges (data dependencies). Key formal definitions include:

• Input Set $\mathrm{In}(n) \subseteq N$: the nodes supplying inputs to node $n$.
• Successor Set $\mathrm{Succ}(n) \subseteq N$: nodes for which $n$ serves as an input.
• Operation Label $\mathrm{op}(n)$: identifies the operation at node $n$.
• Indegree $N$0: the number of pending input dependencies for $N$1; $N$2 initially and is decremented as dependencies are resolved.
• Ready Set $N$3: nodes with $N$4, formally $N$5.

The batching mechanism relies on signatures, where a function $N$6 assigns a signature to each operation (encoding the operation type and any necessary context such as parameter sharing and tensor shape), ensuring nodes $N$7 with $N$8 are batch-compatible.

A priority queue ("agenda" $N$9) is constructed from $\tanh$0 and ordered using a user-defined priority $\tanh$1. DyNet’s default $\tanh$2 is the average depth of all nodes sharing the same signature, allowing signatures occurring early in the graph to be scheduled first. Ties are broken using a "cheap-op-first" heuristic (e.g., scheduling elementwise ops before matrix multiplications).

2. Algorithmic Loop of Agenda-Based Batching

The batching is realized through a greedy, priority-driven traversal over the computation DAG. The main loop, which governs both forward and reverse (backward) passes, proceeds as follows:

1. Pop Highest Priority Node: Remove the node $\tanh$3 with highest priority from the agenda $\tanh$4.
2. Batching by Signature: Collect all additional nodes in $\tanh$5 with the same signature as $\tanh$6 (given by $\tanh$7), forming a batch.
3. Execute Batched Operation: Pack the inputs into contiguous arrays where necessary and invoke the backend to process the corresponding batched kernel.
4. Update Successor Indegree: For every $\tanh$8 in the batch, iterate through $\tanh$9, decrementing $E$0. When $E$1, push $E$2 to $E$3.

The process repeats until $E$4 is empty, ensuring all nodes are visited exactly once. Packing can be optimized if inputs are already laid out contiguously in memory.

3. Illustrative Graph Walkthrough

To elucidate the scheduler’s behavior, consider a small computation graph with nodes:

• $E$5, $E$6, $E$7, $E$8
• $E$9, $\mathrm{In}(n) \subseteq N$0

Initially, $\mathrm{In}(n) \subseteq N$1, with $\mathrm{In}(n) \subseteq N$2 and $\mathrm{In}(n) \subseteq N$3 pending due to unsatisfied dependencies. The scheduler proceeds as follows:

Iteration	Agenda Contents	Popped Node(s)	Batch Signature	Successor Updates
1	n1, n2, n3, n4	n1, n2, n4	"tanh"	n5 (part), n6 (part/ready)
2	n3, n6	n3	"log"	n5 (ready)
3	n6, n5	n5, n6	"add"	--

After three iterations, all nodes are computed in batched form where possible, and the agenda is exhausted.

4. Computational Complexity and Memory Overhead

Queue operations dominate the overhead, with each of the $\mathrm{In}(n) \subseteq N$4 real op-nodes pushed and popped once, yielding $\mathrm{In}(n) \subseteq N$5 complexity in the worst case. DyNet optimizes practical runtime by bucketing signatures and employing integer keys, achieving very low per-node overhead in practice.

Grouping by signature at each pop is $\mathrm{In}(n) \subseteq N$6 on average, where $\mathrm{In}(n) \subseteq N$7 is the small size of per-signature buckets. The requisite indegree counters and buffers are also $\mathrm{In}(n) \subseteq N$8. Empirically, total overhead for graph scheduling and batching is on the order of 10–20% of runtime on CPU and below 10% on GPU in benchmarked BiLSTM experiments, with observed raw speedups of up to 10× over the unbatched baseline (Neubig et al., 2017).

5. Practical Usability and Performance Trade-Offs

The agenda-based scheduler affords several practical advantages:

• Ease of implementation: The user authors serial code for single-instance computation, sums up loss values, and invokes a single forward pass. No explicit padding, masking, or manual data reshaping is required.
• Competitive throughput: For fixed-length sequence tagging (amenable to manual batching), the scheduler yields throughput within 1.1–1.3× of hand-tuned manual batching on GPU and 1.2–1.4× on CPU, presenting an effective trade-off for developer productivity.
• Largest gains on complex architectures: In models such as tree LSTMs or transition-based parsers, where manual batching necessitates intricate workarounds (e.g., padding or parallel state-handling), agenda-based batching delivers speedups ranging from 3× to 9× over naïve single-instance execution, sometimes outperforming frameworks that demand static batch graphs.
• Micro-batching opportunities: Even in code with manual minibatching, the scheduler can further gang-schedule compatible operations across RNN timesteps or loss computations, producing an additional 5–10% performance gain on GPU.

6. Architectural and Implementation Notes

Agenda-based scheduling is implemented as a light-weight, $\mathrm{In}(n) \subseteq N$9 traversal and greedy grouping algorithm that identifies and fuses batch-compatible operations at runtime. Correctness is preserved, hardware parallelism is exposed through batched kernels, and the necessity for users to maintain batch-oriented boilerplate is eliminated. Internal heuristics (signature-based bucketing, depth-averaged priority, cheap-op-first tie breaking) further optimize both throughput and developer usability (Neubig et al., 2017).

7. Context within Dynamic Computation Frameworks

Agenda-based scheduling addresses a primary challenge in dynamic neural network toolkits: achieving efficient batching without sacrificing model expressivity or developer productivity. By shifting batching responsibility from the user to the execution engine, DyNet and similar frameworks enable rapid prototyping and deployment of models with variable-dimensional or nontrivial architectures, providing practical performance without static graph declarations or extensive low-level batching management. This approach exemplifies a key advance in dynamic computation graph toolkits, balancing flexibility and efficiency (Neubig et al., 2017).