Multi-Stage ChunkFormer: Efficient Temporal Modeling

• The paper presents a Transformer-based architecture that segments sequences into non-overlapping chunks, capturing local then global features efficiently.
• It processes time series through successive stages with increasing chunk sizes, substantially lowering computational and memory costs compared to global attention.
• Empirical results show superior performance and robustness on imbalanced classification tasks, demonstrating improved Macro F₁ scores across varying sequence lengths.

Multi-Stage ChunkFormer is a Transformer-based architecture designed to provide efficient and effective modeling of long time series. Unlike the original Transformer, which applies global attention across the entire sequence, ChunkFormer splits the input sequence into multiple non-overlapping chunks at each stage, progressively increasing chunk sizes in a multi-stage cascade. This approach enables the model to sequentially acquire local then global temporal dependencies while substantially reducing both computational and memory requirements compared to standard attention mechanisms (Ju et al., 2021).

1. Architecture Overview

ChunkFormer processes a time series $$X = [x_1, x_2, \dots, x_N] \in \mathbb{R}^{N \times d}$$ using a sequence of $S$ Transformer blocks, each operating on progressively larger non-overlapping chunks. At stage $s$, the input sequence $X^{(s-1)}$ is partitioned into $B_s = N / c_s$ chunks of size $c_s$, then standard Transformer self-attention is computed independently within each chunk:

• In early stages (small $c_s$), local features such as seasonality or abrupt fluctuations are captured.
• In later stages (large $c_s$), the receptive field expands, integrating dependencies over larger temporal ranges.

Block-wise composition:

$$X^{(0)} \xrightarrow{\,c_1\,} \text{Chunked-Attention} \xrightarrow{\,c_2\,} \cdots \xrightarrow{\,c_S\,} \text{Chunked-Attention} \rightarrow X^{(S)}$$

Each chunked-attention unit is a standard Transformer encoder with layer normalization, residuals, and feed-forward sub-layers, except that attention is only permitted within a chunk.

2. Stage-wise Chunked Self-Attention

For stage $s$, the projection to queries, keys, and values is applied:

$S$0

where $S$1. These are reshaped into $S$2 chunks of shape $S$3.

Within each chunk $S$4:

$S$5

Here, $S$6 is a positional bias or relative position matrix ($S$7), which may be sinusoidal, learned, or relative inside each chunk. The outputs $S$8 are concatenated to re-form a length-$S$9 sequence at the end of the stage.

3. Progressive Multi-Stage Composition

The schedule of chunk sizes $s$0 is monotonic, with possible choices including exponential growth ($s$1) or domain-driven intervals (e.g., hourly, daily, weekly). This design systematically increases each stage's receptive field:

• Stage-1 focuses on fine-grained localized context.
• Stage-2 and later stages aggregate patterns that span multiple finer chunks.
• The final stage enables near-global temporal integration while maintaining fixed sequence length.

The staged composition in pseudocode: $B_s = N / c_s$5

4. Computational and Memory Complexity

The vanilla Transformer's self-attention forms an $s$2 matrix, resulting in both time and memory complexity of $s$3. In contrast, each ChunkFormer stage costs $s$4 due to $s$5 blocks, each of size $s$6:

$s$7

with $s$8 the largest chunk size. When $s$9, ChunkFormer effectively achieves linear time and space scaling, better than even many sparse attention methods ($X^{(s-1)}$0) for long sequences. For $X^{(s-1)}$1 and $X^{(s-1)}$2, total cost $X^{(s-1)}$3.

5. Empirical Results

ChunkFormer was evaluated on three real-world, highly imbalanced classification benchmarks:

• Content-Delivery-Network (CDN) failure detection (5.4M logs)
• TalkingData fraud-click detection (10M samples)
• Online Education answer-correctness (10M records)

All models utilized the same embedding layer, Adam optimizer, and binary cross-entropy loss. Comparative models included LSTM, vanilla Transformer, and LogSparseFormer, with Macro F₁ (sensitive to class imbalance) as the key metric. Across four sequence length settings (180, 240, 480, 720), ChunkFormer with $X^{(s-1)}$4 or $X^{(s-1)}$5 stages offered superior F₁ performance.

Dataset	Model	Macro F₁ @180	@240	@480	@720
CDN	LSTM	0.38	0.40	0.41	0.39
	Transformer	0.43	0.44	0.45	0.45
	ChunkFormer (S=2)	0.44	0.46	0.47	0.47
TD	Transformer	0.27	0.29	0.31	0.30
	ChunkFormer	0.29	0.30	0.32	0.32
OE	Transformer	0.51	0.52	0.53	0.52
	ChunkFormer	0.53	0.54	0.55	0.55

Reported improvements were +1–3 points in absolute Macro F₁, with increased robustness to sequence length. Ablation studies indicate that $X^{(s-1)}$6 or $X^{(s-1)}$7 already saturate gains, with chunk sizes $X^{(s-1)}$8 or $X^{(s-1)}$9 preferred in practice (Ju et al., 2021).

6. Practical Configuration and Observed Limitations

Guidelines for application:

• Number of stages $B_s = N / c_s$0: 2–3 suffices, as further stages provide marginal returns.
• Chunk sizes $B_s = N / c_s$1: Exponential scaling is simple and robust; domain-inspired intervals are also viable.
• Sequence length $B_s = N / c_s$2: Performance benefits increase with larger $B_s = N / c_s$3 (hundreds to thousands).
• GPU memory constraints require $B_s = N / c_s$4 to be manageable. Limitations and unresolved issues include:
• The selection of chunk sizes is a tunable hyperparameter without a canonical strategy.
• Positional encoding is strictly local within each chunk; distant global position information is propagated only indirectly through stages.
• Hard, non-overlapping chunk boundaries may limit modeling of features at chunk edges; overlapping or shifted chunking might ameliorate boundary artifacts.

A plausible implication is that augmenting the architecture with cross-chunk attention or global tokens could further improve modeling of long-range dependencies.

7. Context and Significance

Multi-Stage ChunkFormer introduces an efficient paradigm for handling long sequence data by decomposing resource-intensive global attention into a sequence of localized attention operations with increasing receptive field. By preserving sequence length at all stages and systemically capturing both local and global dependencies, it offers empirical and practical advantages over both traditional Transformer and sparse attention variants—demonstrated in highly imbalanced real-world classification tasks with significant sequence length variability (Ju et al., 2021). The methodology informs subsequent studies on memory-efficient attention and multi-scale sequence modeling.