Pixel-Level Tokenization in Vision Models

• Pixel-level tokenization is a method that decomposes images into patch tokens to enable sequence-based processing analogous to language models.
• It achieves linear compute complexity by processing discrete tokens sequentially, significantly reducing memory and computational costs compared to full self-attention.
• Empirical results from the Adventurer model show competitive classification and segmentation performance with enhanced throughput on high-resolution images.

Pixel-level tokenization refers to the process of decomposing an input image into discrete tokens at a granularity corresponding to individual pixels or small, fixed-size regions (“patches”), enabling sequence-based processing using deep learning architectures. This approach facilitates modeling images as ordered sequences, making it compatible with models developed for sequential data—such as LLMs—while supporting efficient scaling and fine-grained visual understanding. Contemporary architectures leverage patch or pixel-level tokenization to address the computational bottlenecks encountered in high-resolution and fine-grained image analysis, imposing structured constraints on data flow to achieve linear complexity in sequence length, as exemplified in the Adventurer architecture (Wang et al., 2024).

1. Formal Definition and Motivation

Pixel-level tokenization transforms a two-dimensional image input $x \in \mathbb{R}^{H \times W \times C}$ into an ordered sequence of low-dimensional embeddings by partitioning the image into non-overlapping patches of shape $p \times p$. Each patch is flattened and linearly projected to a vector in $\mathbb{R}^{d}$, resulting in a token sequence $[t_1, t_2, \dots, t_N]$, where $N = (H \cdot W) / (p^2)$. A learnable class token or pooled representation is frequently appended. This sequence-oriented representation allows architectures to employ causal, uni-directional, or bidirectional mixers, situating image analysis in a framework similar to text modeling.

The primary motivator is the prohibitive computational and memory cost of full self-attention, which is $O(N^2)$ in the number of patches $N$ per layer. For high spatial resolutions, $N$ becomes large, quickly overwhelming available resources. Sequential, pixel-level tokenization enables linear complexity $O(N)$ by replacing self-attention with recurrent or state-space mixers that process one token at a time, crucial for high-resolution or fine-grained vision tasks (Wang et al., 2024).

2. Sequential Processing and Architectural Realization

The Adventurer series demonstrates a practical realization of pixel-level tokenization within a causal visual modeling framework. The model processes images as strictly ordered patch token sequences. The input sequence is augmented with two special tokens at each layer: a global average “heading” token representation ($x_i^{\text{AVG}}$) prepended to the patch sequence, and a class (CLS) token appended at the end. With $p \times p$0 denoting the patch and CLS tokens at the $p \times p$1-th layer, the augmented input is

$p \times p$2

where

$p \times p$3

After causal token and channel mixing, the heading token is dropped and recalculated for the next layer. To address positional biases inherent in strictly causal scan paths, Adventurer applies an inter-layer flipping: between successive layers, the patch token order is reversed along the sequence dimension while the CLS token remains at the end (Wang et al., 2024).

3. Computational and Memory Complexity

Pixel-level tokenization, when combined with recurrent mixers such as Mamba-2, achieves linear complexity in both computation and memory with respect to the input sequence length $p \times p$4. Specifically, in the Adventurer model:

• Token mixer compute per layer: $p \times p$5
• Token mixer memory per layer: $p \times p$6
• Channel mixer (SwiGLU MLP) per layer: $p \times p$7

This is in stark contrast to standard Vision Transformers (ViT) where self-attention has $p \times p$8 compute and $p \times p$9 memory per layer. Empirically, this enables processing of image sequences with 3,000+ tokens (e.g., $\mathbb{R}^{d}$0 images) with tractable memory and computation—a regime inaccessible to quadratic-complexity models (Wang et al., 2024).

4. Empirical Efficacy and Ablation Analysis

Extensive empirical studies demonstrate that causal, pixel-level tokenization with heading-average and inter-layer flipping can closely match or exceed the accuracy of full self-attention models across classification (ImageNet), semantic segmentation (ADE20K), and instance/object detection (COCO). Notable results include:

• At standard $\mathbb{R}^{d}$1, throughput equals or surpasses DeiT-Base at equivalent accuracy (82.6% vs 81.8%).
• At $\mathbb{R}^{d}$2, Adventurer achieves higher accuracy (84.0% vs 83.5%) at 2.5x–4.4x faster training throughput (216 vs 86 images/s).
• Time and memory scale linearly with sequence length, as confirmed by empirical curves $\mathbb{R}^{d}$3.

Ablations show that omitting either the heading-average or inter-layer flipping induces ≈1% accuracy drop, while both together restore or improve performance compared to full self-attention baselines. No gain is observed from using multiple average region tokens or from fancier global token designs; a single mean-pooled (heading) token is optimal. SwiGLU-augmented mixers further optimize the efficiency-accuracy trade-off with no additional latency (Wang et al., 2024).

5. Theoretical and Biological Connections

The sequential, causal modeling of pixel-level tokens draws formal analogy to human visual processing, especially saccadic foveal vision. In this analogy, the model’s patch-by-patch sequential pass corresponds to visual attention scanning, integrating evidence over time. The architectures further formalize early visual feature integration by promoting global context (via the heading-average token) and distributing positional context through inter-layer flipping. The approach thereby unifies recurrent computations, linear complexity, and context mixing in a manner reminiscent of biological sequence processing (Wang et al., 2024).

6. Limitations and Future Directions

Despite the tractability and performance advantages conferred by pixel-level tokenization in causal image models, several limitations persist. Positional encoding in Adventurer is restricted to simple absolute embeddings, possibly suboptimal for strictly causal processing regimes. Dynamic patching or adaptive scan paths—where tokenization varies adaptively over spatial regions—remain unaddressed, representing promising directions for further efficiency improvements. The uniform patch size is a simplifying assumption; more nuanced forms of discretization, as well as more causal-compatible positional encoding mechanisms, are viable areas for future investigation. The design of heading tokens (beyond simple averages) and hybridization with advanced attention mechanisms may offer additional gains (Wang et al., 2024).

---

References:

• "Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency" (Wang et al., 2024)