---
title: Continuous Depth-wise Batching
url: https://www.emergentmind.com/topics/continuous-depth-wise-batching
type: topic
---

# Continuous Depth-wise Batching

Continuous Depth-wise Batching (CDB) is a dynamic inference paradigm that leverages the parameter-tied (recursive) structure of Transformers to pipeline sequence processing not only in the conventional time (token) dimension, but also across model depth (layer iterations). CDB exploits repeated blocks within a recursive Transformer—enabled by weight sharing across layers—to maximize hardware utilization and reduce idle compute slots, achieving substantial improvements in throughput for autoregressive generation, especially when combined with early-exit mechanisms. When deployed in practice, CDB demonstrates a 2–3× increase in token-generation throughput relative to static, layer-distinct Transformers, subject to model configuration and early-exit dynamics [2410.20672].

## 1. Formal Definition and Mechanism

Continuous Depth-wise Batching operates in recursive Transformers constructed from a total depth $L$, reorganized as $B$ looping blocks of size $K=L/B$, with $f_k(h;\,\Phi'_k)$ denoting the $k$th layer of a shared block. For token $t$ and layer index $\ell$, the forward pass is:
$$
h_{t}^{(\ell)} = f_{((\ell-1)\bmod K)+1}\left(h_{t}^{(\ell-1)};\,\Phi'_{((\ell-1)\bmod K)+1}\right)
$$
CDB maintains, during runtime, up to $N_{\max}$ active samples at each block-iteration stage $j=1,\ldots,B$. For each compute step, up to $N_{\max}$ samples in $\mathcal{S}_j = \{i : \text{sample } i \text{ has reached block iteration } j \text{ and has not exited}\}$ are processed in parallel by the corresponding shared block $f_j$. Freed slots are immediately filled with either survivors from deeper iterations or new requests entering at the initial block stage. This batched scheduling occurs in both time (token) and depth (block iteration) dimensions and maximizes accelerator utilization for each block function [2410.20672].

## 2. Scheduling: Pseudocode and Operational Overview

A high-level scheduling procedure for CDB paired with early-exiting utilizes $B$ separate FIFO queues, one per block iteration. At each scheduler tick, a batch is formed (up to $N_{\max}$) for each queue, the shared block $f_j$ is applied, and samples either exit (if the early-exit criterion is satisfied) or proceed to the next block. New requests enter at the first block whenever there is available capacity. The general scheduler pseudocode is:

```python
Inputs: B, K, N_max, {f_j()}, exit_criterion, request_stream

Initialize: Q[1..B] ← empty FIFO queues, sample_counter ← 0

Procedure SCHEDULE_NEXT():
    For j in 1..B:
        batch ← dequeue up to N_max from Q[j]
        if batch is empty: continue
        For each sample in batch:
            sample.h ← f_j(sample.h, Φ'_j)
            sample.pos += 1
        survivors ← []
        For each sample in batch:
            if exit_criterion(sample.h):
                record_final_output(sample)
            else:
                survivors.append(sample)
        For s in survivors:
            next_block ← (j % B) + 1
            enqueue s into Q[next_block]
    While |Q[1]| < N_max and request_stream not empty:
        req ← request_stream.pop()
        sample_counter += 1
        initial_h ← embed_start_token(req)
        enqueue (sample_id=sample_counter, h=initial_h, pos=0) into Q[1]

Main Loop:
    While there are unfinished samples or new requests:
        SCHEDULE_NEXT()
```

Key features include up to $B$ simultaneous batches (one per block), flexible early-exiting, and backfilling to maintain throughput. Batching is conducted across samples invoking the same block parameters, which is feasible only in weight-sharing architectures [2410.20672].

## 3. Theoretical and Empirical Throughput Gains

CDB’s effectiveness is quantified against two baselines: static synchronous batching and continuous sequence-wise batching (CSB):

- **Static batching**: All $N_{\max}$ slots process in lock-step with throughput $R_\mathrm{static} = N_{\max}/(B\,T_{\mathrm{blk}})$.
- **CSB**: Batching across token sequences at the same block depth, empirically yielding a speedup $S_{\mathrm{seq}} \approx 1.4$ (e.g., Gemma 2B: 1080 tok/s → 1528 tok/s).
- **CDB**: Enables $B$-fold depth-wise pipeline; for $B=2$ and $S_{\mathrm{seq}}=1.4$, this yields a theoretical $\approx 2.8\times$ speedup.

With early-exit, if the mean exit depth is $\bar\ell < B$, the depth-wise pipeline is shortened, and the effective speedup approaches $S_{\mathrm{seq}} \times \bar\ell$. For example, with Gemma 2B and $\bar\ell\approx 1.9$, the observed throughput was $2877$ tok/s, corresponding to a $\times2.66$ speedup [2410.20672].

## 4. Experimental Configurations and Results

Key configuration details include:

- **Models evaluated**: Gemma 2B (18 layers, 2 blocks of 9), TinyLlama 1.1B (22 layers, 2 blocks of 11), Pythia 1B (16 layers, 2 blocks of 8).
- **Batch size**: $N_{\max}=32$.
- **Early-exit criterion**: Confidence score (e.g., max-token log-probability) checked after each block iteration; oracle simulations provided idealized throughput.
- **Hardware profiling**: V100/A100 GPU, per-block timing denoted by $T_{\mathrm{blk}}$.
- **Token-generation throughput** (Gemma 2B, SlimPajama/RedPajama/PG19):
  - Stat

Source: https://www.emergentmind.com/topics/continuous-depth-wise-batching