--- title: 'QuadTok: Hierarchical Visual Tokenization' url: https://www.emergentmind.com/topics/quadtok type: topic --- # QuadTok: Hierarchical Visual Tokenization QuadTok is a visual tokenizer and autoregressive image-generation framework based on a content-adaptive quadtree representation. Rather than encoding a $256\times256$ image with a fixed $16\times16$ grid of 256 tokens or with an unstructured variable-length sequence, QuadTok organizes visual tokens hierarchically: homogeneous regions remain represented by coarse parent nodes, while visually complex regions are recursively expanded into finer child tokens. The resulting representation combines explicit two-dimensional spatial correspondence with one-dimensional sequence flexibility. In the reported ImageNet experiments, QuadTok reduces the mean token count by approximately 10% relative to a fixed 256-token grid on ImageNet and 9% under zero-shot transfer to COCO while maintaining comparable reconstruction fidelity. Its 947M-parameter autoregressive generator reaches a gFID of 2.08 on ImageNet at $256\times256$ resolution [2610.10497]. ## 1. Conceptual foundations and terminology QuadTok is centered on the premise that image information density is spatially heterogeneous. Smooth sky, walls, or uniform backgrounds can often be represented at coarse resolution, whereas object boundaries, foliage, animals, and highly textured regions benefit from finer spatial subdivision. A fixed grid allocates equal token capacity to all regions, while an unstructured one-dimensional tokenizer permits variable sequence lengths but weakens the correspondence between tokens and image regions. The method therefore combines two properties: - **Hierarchical spatial correspondence**: every token is associated with a quadtree level and spatial position. - **Variable-length serialization**: the selected tree is converted into a one-dimensional sequence suitable for autoregressive modeling. The full discrete representation is the pair $(Y,T)$ rather than the code sequence alone. Here, $Y=[y_1,\ldots,y_M]$ contains the discrete visual codes and $T=(n_1,n_2,\ldots,n_M)$ contains the topology, with each node represented as $$ n_i=(l_i,p_i), $$ where $l_i$ is the hierarchy level and $p_i$ indexes the spatial cell at that level. The same code index may have different spatial meanings at different levels or positions. The name QuadTok should be distinguished from the tokenizer conceptually called QuadTok in “Vision Transformers with Mixed-Resolution Tokenization” [2304.00287]. That earlier method uses a quadtree to allocate mixed-resolution image patches before a standard Vision Transformer and calls the complete model a Quadformer. The present QuadTok instead uses a quadtree for discrete visual tokenization and autoregressive image generation [2610.10497]. ## 2. Quadtree topology and sequence construction The default $256\times256$ configuration uses a two-level hierarchy: - level 1: an $8\times8$ coarse grid; - level 2: a $16\times16$ fine grid. All 64 coarse cells are retained. Each coarse cell either remains one token or is expanded into four child tokens. If $K$ coarse cells are expanded, the sequence length is $$ M=64+4K. $$ The minimum sequence length is therefore 64 tokens, while the maximum for the two-level hierarchy is 320 tokens. A three-level extension adds a $32\times32$ grid, forming the hierarchy $$ 8\times8\rightarrow16\times16\rightarrow32\times32. $$ QuadTok serializes tree nodes using breadth-first search (BFS). Coarse nodes occur before their descendants, yielding a coarse-to-fine sequence in which global structure precedes regional refinement. The sequence remains compatible with Transformer processing, while the node attributes retain information about hierarchy and spatial location. BFS supplies the autoregressive ordering used by the generator: 1. coarse level-1 codes are generated first; 2. child codes are generated for expanded regions; 3. increasingly fine tokens refine selected spatial areas. The topology is supplied before generation. The generator does not infer the topology from an image during unconditional generation because no image is available; instead, the topology is selected or constructed before visual-code sampling. ## 3. Token formation and topology-aware attention Given an image $I$, QuadTok first computes dense ViT features, $$ F=\operatorname{Encoder}(I),\qquad F\in\mathbb{R}^{N\times D}. $$ For each quadtree node $n_i=(l_i,p_i)$, a learnable structural instruction embedding is constructed: $$ q_i=\operatorname{Emb}_{l_i}(p_i). $$ The dense features and node instructions are jointly processed by an Aggregator Transformer: $$ [F'\parallel Z]=\operatorname{Aggregator}([F\parallel Q]), $$ where $Z=[z_1,\ldots,z_M]$ contains one continuous representation for each selected quadtree node. The structural instruction acts as a region-specific query: the node representation can access globally contextualized ViT features while retaining the level and spatial position of the region it represents. QuadTok then applies vector quantization using a shared codebook $$ \mathcal{C}=\{e_k\}_{k=1}^{K}. $$ The reported $256\times256$ configuration uses vocabulary size $K=16{,}384$ and code dimensionality $d=8$. Each node embedding is assigned to its nearest codebook vector: $$ y_i=\operatorname{Quant}(z_i) =\arg\min_{k\in\{1,\ldots,K\}}\|z_i-e_k\|_2^2. $$ The resulting representation is the variable-length discrete sequence $Y$ together with the topology $T$. ### Kinship Causal Attention QuadTok uses a topology-aware attention mask called the **Kinship Causal Mask** in the tokenizer’s Aggregator and Decoder. For the two-level tree: - coarse nodes attend bidirectionally to one another; - child nodes attend to their parents, but parents do not attend to children; - siblings attend bidirectionally to one another; - unrelated fine-level nodes do not attend directly. For deeper trees, a node can attend to all coarser nodes and same-level siblings, but not to finer-level descendants. The mask is therefore partly bidirectional and structured by tree relationships. It is distinct from the generator’s ordinary left-to-right causal mask. The mask preserves locality and region-code binding. In perturbation tests, changing a fine-level code produces a localized reconstruction change when the mask is used. Without it, perturbations spread across the image, and the representation behaves more like an unstructured one-dimensional code stream. The reported ablation gives: | Tokenizer | rFID | PSNR | gFID | |---|---:|---:|---:| | Without Kinship Mask | 1.41 | 19.16 | 3.17 | | With Kinship Mask | 1.46 | 20.37 | 2.23 | The unconstrained variant obtains slightly lower rFID but substantially lower PSNR and much worse generation gFID. For spatially grounded autoregression and topology-based control, the topology-aware mask preserves the more important structural correspondence. ## 4. Hierarchical decoding and adaptive topology selection After quantization, codebook embeddings are combined with structural instructions: $$ \tilde z_i=e_{y_i}+q_i. $$ A Transformer Decoder processes the sequence $\tilde Z=[\tilde z_1,\ldots,\tilde z_M]$. Unlike a conventional decoder that reshapes all tokens into a single dense grid, QuadTok projects each node into a region-specific feature patch according to its tree level and spatial position: $$ C_{l_i}(p_i)=U_{l_i}(\tilde z_i), $$ where $U_l$ is a level-specific projection or unpatching operator. Patches at each level are placed at their spatial positions to form a residual feature map $C_l$. Hierarchical feature maps are then combined through learned upsampling: $$ H_{l+1}=F^{\mathrm{up}_{l+1}}(H_l)+C_{l+1}, \qquad H_0=0. $$ The final feature representation is projected to RGB space: $$ \hat I=\operatorname{RGB}(H_L). $$ Coarse nodes establish broad image structure, while children contribute local residual detail. A homogeneous region can therefore be reconstructed from a parent token alone, whereas a complex region can be refined using four child tokens. ### Training-time tree sampling During training, all level-1 nodes are retained. Each parent independently expands into its four children with probability $\gamma$, using the default value $$ \gamma=0.5. $$ This exposes the tokenizer to multiple valid tree shapes and sequence lengths rather than a single deterministic topology. The tokenizer is trained with pixel, perceptual, vector-quantization, and adversarial losses: $$ \mathcal{L}_{\mathrm{total}} = \lambda_{\mathrm{MSE}}\mathcal{L}_{\mathrm{MSE}} +\lambda_{\mathrm{LPIPS}}\mathcal{L}_{\mathrm{LPIPS}} +\lambda_{\mathrm{VQ}}\mathcal{L}_{\mathrm{VQ}} +\lambda_{\mathrm{GAN}}\mathcal{L}_{\mathrm{GAN}}. $$ Adversarial training is initially disabled, with $\lambda_{\mathrm{GAN}}=0$ for the first 200,000 steps, and activated afterward. The reported tokenizer has approximately 120M parameters; its encoder, Aggregator, and Decoder each use eight Transformer blocks with hidden size 512. The ViT encoder uses $16\times16$ patches. ### Region-Wise Complexity Guidance QuadTok selects image-dependent topologies using **Region-Wise Complexity Guidance**. Two complementary probing trees are constructed: one expands approximately half of the coarse regions, and the other expands the complementary half. Each tree is decoded to produce a reconstruction and a spatial LPIPS error map. For a region $r$, the benefit of expansion is defined as $$ b_r=\operatorname{AvgPool}_r \left( E_r^{\mathrm{collapsed}} - E_r^{\mathrm{expanded}} \right). $$ The region is expanded when $$ b_r>\tau, $$ with default threshold $\tau=0.05$. The complete adaptive encoding procedure is: 1. compute ViT features; 2. construct probing trees; 3. decode the alternative tree configurations; 4. calculate regional LPIPS improvement; 5. expand regions satisfying $b_r>\tau$; 6. encode the image using the selected tree; 7. quantize the resulting node representations. This is a probing and selection algorithm rather than a conventional learned topology-prediction network. Its computational cost is substantial because topology search requires multiple reconstructions and LPIPS calculations. ## 5. Autoregressive generation and spatial control Given a class condition $c$, a topology $T$, and its BFS-ordered visual-code sequence $Y=[y_1,\ldots,y_M]$, QuadTok models $$ p(Y\mid c,T) = \prod_{i=1}^{M} p(y_i\mid y_{