(1D) Ordered Tokens Enable Efficient Test-Time Search

Abstract: Tokenization is a key component of autoregressive (AR) generative models, converting raw data into more manageable units for modeling. Commonly, tokens describe local information, such as regions of pixels in images or word pieces in text, and AR generation predicts these tokens in a fixed order. A worthwhile question is whether token structures affect the ability to steer the generation through test-time search, where multiple candidate generations are explored and evaluated by a verifier. Using image generation as our testbed, we hypothesize that recent 1D ordered tokenizers with coarse-to-fine structure can be more amenable to search than classical 2D grid structures. This is rooted in the fact that the intermediate states in coarse-to-fine sequences carry semantic meaning that verifiers can reliably evaluate, enabling effective steering during generation. Through controlled experiments, we find that AR models trained on coarse-to-fine ordered tokens exhibit improved test-time scaling behavior compared to grid-based counterparts. Moreover, we demonstrate that, thanks to the ordered structure, pure test-time search over token sequences (i.e., without training an AR model) can perform training-free text-to-image generation when guided by an image-text verifier. Beyond this, we systematically study how classical search algorithms (best-of-N, beam search, lookahead search) interact with different token structures, as well as the role of different verifiers and AR priors. Our results highlight the impact of token structure on inference-time scalability and provide practical guidance for test-time scaling in AR models.

• The paper shows that 1D coarse-to-fine tokenization makes image prefixes semantically meaningful, enabling beam search to improve COCO CLIPScore by 13.05 points versus 2.53 for a matched 2D grid tokenizer.
• The authors introduce Search-over-Tokens, which evaluates search algorithms, verifiers, and autoregressive priors, finding that ordered tokens generalize across models and support text-to-image, image-conditioned, and weak-prior generation.
• The paper theoretically links search quality to intermediate reconstruction error and finds that verifier hacking, decoding cost, and prior limitations remain important barriers to reliable test-time scaling.

Overview

This paper investigates how the structure of image tokens—rather than the search algorithm or verifier alone—determines whether test-time scaling (TTS) via verifier-guided search is effective in autoregressive (AR) image generation. The authors hypothesize that 1D ordered tokenizers such as FlexTok, which produce coarse-to-fine token sequences trained with nested dropout, are inherently more amenable to test-time search than classical 2D grid tokenizers, because intermediate token prefixes decode into semantically meaningful images that verifiers can reliably score. They validate this hypothesis through controlled experiments, a systematic evaluation framework ("Search-over-Tokens", SoTo), and a theoretical analysis linking search error to intermediate reconstruction error (2604.15453).

The central claim is strong: the advantage of 1D ordered tokens is a representation-level property that cannot be closed by choosing a better search algorithm for 2D grid tokens. This claim is supported by controlled comparisons in which data, architecture, and training compute are matched between tokenizer variants.

Why ordered tokens are searchable

Three lines of evidence establish that FlexTok's token space is structured for search:

• First-token semantics: decoding individual first-token entries from a 64K vocabulary with multiple random seeds yields semantically coherent clusters (plants, bags, food, furniture). In a 2D VQGAN-style tokenizer, by contrast, the first token corresponds only to a top-left pixel patch and carries no global information.
• Training-free generation: beam search directly over FlexTok tokens—with no AR model at all, using CLIP or ImageReward as verifier—produces coherent, prompt-aligned images whose detail refines progressively as more tokens are added. The authors note this behavior does not arise with 2D grid or unordered 1D tokenizations, where early tokens provide no signal about later ones.
• Theory: two propositions bound the search gap $\Delta$ by $2L\epsilon_{t_0} + \eta_{t_0}$, where $\epsilon_t$ is the reconstruction error of the prefix at the critical pruning step $t_0$, $L$ is the Lipschitz constant of the verifier, and $\eta_{t_0}$ is continuation suboptimality. Because nested dropout explicitly minimizes intermediate reconstruction error, 1D ordered tokens keep $\epsilon_t$ small throughout generation; 2D grid tokenizations enforce reconstruction only at $t=T$, leaving large $\epsilon_t$ at early steps. The bound relies on a Lipschitz assumption on the verifier and on the quality of partial decodes, both of which are assumptions rather than guarantees.

An implication of the training-free result is that image generation can be cast purely as inference-time optimization over a discrete latent space when the latent structure exposes semantics early—a capability unavailable under spatially-ordered tokenization.

The SoTo framework

The paper systematizes test-time search along three axes: search algorithms (best-of-$N$, beam search, lookahead search), verifiers (eight scorers spanning image–text alignment—CLIPScore, ImageReward, PickScore, HPSv2, CycleReward, likelihood, rule-based Grounded-SAM—image–image alignment via DreamSim, aesthetics, plus rank-based ensembles), and AR priors (text-conditional, unconditional, uniform). Compute is measured in NFE, counting each token sampling step and each verifier call.

Main experimental findings

Token structure determines which search algorithm works. Under matched base performance (comparable CLIP scores without search), best-of-$2L\epsilon_{t_0} + \eta_{t_0}$0 and lookahead scale similarly across 1D and 2D tokenizations, but beam search diverges sharply: it produces large gains for FlexTok and only marginal gains for the controlled 2D grid baseline. On COCO, FlexTok beam search improves CLIPScore from 80.39 to 93.44 (+13.05) while the 2D baseline improves from 79.06 to only 81.59 (+2.53); variance analysis over five random subsets confirms stability (mean +13.30 ± 0.29 vs. +2.43 ± 0.24). The same pattern holds on GenEval and against Janus-1.3B, where Janus starts slightly ahead but scales worse under beam search. The practical implication is that beam search—the most compute-efficient strategy—is viable only when prefixes are verifiable, i.e., under ordered tokenization; for 2D grids, best-of-$2L\epsilon_{t_0} + \eta_{t_0}$1 remains preferable within comparable budgets.

Generality beyond FlexTok. Semanticist versus LlamaGen-L on ImageNet-1K reproduces the trend: beam search adds +10.42 CLIPScore for Semanticist (simple prompts) versus +3.51 for LlamaGen-L, and +12.45 versus +4.04 on complex prompts. Infinity-2B, a scale-wise (multi-resolution ordering) model, falls between the extremes (+6.2 vs. Janus's +5.3 and FlexTok's +9.6), supporting the interpretation that any ordering helps search but semantic coarse-to-fine ordering helps most.

Zero-shot multimodal control. Using DreamSim as an image–image verifier, FlexTok performs concept-preserving text-to-image generation despite being trained only on text–image pairs. On DreamBench++, search improves DINO-I identity preservation by +18.4 (32.5 → 50.9) while roughly maintaining CLIP-T prompt following; Janus with full lookahead rollouts gains only +5.9 DINO-I at higher compute. This demonstrates that swapping verifiers enables new conditioning modalities without retraining—an orthogonal mechanism to ControlNet-style architectural conditioning or RL fine-tuning.

Generation by search with weak priors. On a 180-prompt GenEval subset, beam search with a uniform prior achieves 79% accuracy on single-object prompts and 32% on two-object prompts; an unconditional prior reaches 85%/33%; the conditional prior reaches 100%/81% (versus 97%/48% without search). Notably, search nearly doubles two-object accuracy over direct conditional decoding (48% → 81%), indicating that verifier guidance corrects compositional failures that the prior alone misses. Best-of-$2L\epsilon_{t_0} + \eta_{t_0}$2 fails entirely under a uniform prior, since unguided samples cannot form meaningful trajectories—only path-dependent search can exploit an uninformative prior.

Verifier analysis. All eight verifiers improve over the AR baseline on GenEval; each ranks first on its own objective, ImageReward and HPSv2 are the strongest individual general-purpose verifiers (overall GenEval accuracy 67 each), and the ensemble achieves the best average column-wise rank while typically ranking second per metric. The oracle GenEval evaluator reaches 76, quantifying remaining headroom. Score-trajectory analysis shows cross-verifier correlation during optimization, though trade-offs exist (e.g., optimizing CLIP can reduce aesthetic scores).

Model size versus test-time compute. A 530M-parameter AR model with sufficient search outperforms the 3.4B model with limited inference compute, and the optimal model size follows a power law in inference FLOPs ($2L\epsilon_{t_0} + \eta_{t_0}$3). Larger models exhibit stronger scaling once compute is abundant. This parallels compute-optimal allocation results in LLM reasoning and diffusion TTS, here established for AR image generation.

Limitations and open questions

The paper concedes several constraints that qualify its conclusions:

• Verifier hacking: with large search budgets, optimization exploits verifier blind spots, producing high-scoring but implausible images; most verifiers provide only global scalar feedback unsuitable for fine-grained correction.
• Detokenization bottleneck: FlexTok's flow-based decoder requires multiple denoising steps, making repeated partial decoding the dominant wall-clock cost during beam/lookahead search; intermediate fidelity may also degrade for very short prefixes.
• Search algorithms are generic: beam, best-of-$2L\epsilon_{t_0} + \eta_{t_0}$4, and lookahead were not designed for coarse-to-fine hierarchies; adaptive verification schedules and learned search policies remain open, and the paper notes exponential versus uniform verification spacing materially affects results without a principled schedule.
• Prior bottleneck: search cannot recover concepts absent from the prior (e.g., missing objects under uniform priors, imperfect reference preservation).
• Limited generality: findings cover a small set of models; whether they extend to text, video, or other modalities, and whether larger models retain searchability (rather than collapsing diversity), are unresolved questions the paper leaves open.

Conclusion

This paper establishes token structure as a first-class determinant of test-time scalability in autoregressive image generation. Through matched-control comparisons, cross-model replication (FlexTok, Semanticist, Infinity), a formal bound tying search error to nested-dropout-trained intermediate reconstruction error, and extreme demonstrations including prior-free generation-by-search and zero-shot image-conditioned control, it shows that 1D coarse-to-fine ordered tokens make intermediate states verifiable—and hence searchable—in ways 2D grid tokens are not. The practical guidance is concrete: pair ordered tokenizations with beam search, use human-preference verifiers or ensembles, and treat model size and inference compute as jointly optimizable resources.