Abstracted Shapes as Tokens

• Abstracted shapes as tokens are a method for converting continuous shape data into discrete tokens that capture semantic and geometric information across diverse modalities.
• They leverage adaptive techniques such as vector quantization, clustering, and Bayesian methods to efficiently compress and represent information from images, 3D models, biomolecules, time-series, and CAD.
• These tokens enhance deep learning systems by enabling interpretable models, precise generative capabilities, and cross-domain integration of spatial, temporal, and semantic cues.

Abstracted shapes as tokens represent a paradigm shift in how structural, geometric, and temporal information is discretized for high-level machine learning applications. Unlike conventional approaches that rely on uniform gridding or arbitrary partitioning, these methods extract and encode semantically meaningful shape primitives—spanning images, 3D geometry, time-series, biomolecules, and programmatic descriptions—directly as tokens amenable to deep learning, foundation models, and symbolic reasoning. This article surveys the landscape of abstracted shape tokenization, elucidating methodological frameworks, mathematical foundations, architectural innovations, and empirical advances across modalities, drawing on key developments from vision, 3D modeling, biomolecular informatics, time-series analysis, and CAD automation.

1. Foundational Concepts and Formal Definition

The core principle of abstracted shape tokenization is the transformation of continuous or structured shape data into a discrete vocabulary of tokens, each representing a localized or semantically atomic shape unit. This abstraction is operationalized differently across domains:

• Vision: Tokens correspond to arbitrarily-shaped image regions aligned to semantic boundaries (e.g., foreground objects, body parts) rather than fixed-grid patches (Zeng et al., 2022).
• 3D Geometry: Tokens are derived by adaptive clustering (octrees, semantic hierarchies, VQ) over volumetric or surface representations (voxel grids, triplanes, point clouds), or as parameters of geometric primitives (Dutt et al., 18 Mar 2026, Deng et al., 3 Apr 2025, Wu et al., 2022).
• Biomolecular Structures: Substructures (e.g., short protein backbone segments) are mapped to discrete tokens via vector quantization of local 3D geometry (Liu et al., 13 Nov 2025).
• Time-Series: Prototypical waveform fragments are vector-quantized into shape tokens, each representing recurring dynamical motifs (Wen et al., 2024).
• Programs and CAD: Sequences of visual or schematic primitives (e.g., sketch curves, extrusions, symbolic subroutines) are encapsulated as tokens, enabling interpretable and editable program synthesis (Wang et al., 25 Sep 2025, Jones et al., 2023).

Formally, given a geometric or structural signal $X$, an encoder $E$ maps $X$ into a sequence (or set) of latent vectors $z_i$; vector quantization or clustering assigns each $z_i$ to a codebook entry (token) $e_k$, yielding a discrete sequence $\{k_i\}$ or, in the continuous-token case, a compact real-vector representation preserving essential semantics.

2. Methodological Frameworks Across Modalities

Multiple methodological frameworks instantiate "shapes as tokens" to optimally abstract, compress, and model structural information:

• Progressive Clustering and Adaptive Merging: In vision transformers, tokens are adaptively merged via density-based clustering, yielding tokens with flexible, non-grid regions, facilitating focus on semantically salient image areas (prominently human figures vs. background) (Zeng et al., 2022).
• Vector Quantized Autoencoders: Ubiquitous in 3D geometry, protein structure, time-series, and CAD, VQ-VAEs learn discrete vocabularies mapping continuous substructures to prototypical codebook entries, e.g., local geometric patches, waveform snippets, CAD primitives. The commitment and codebook losses ensure stable and diverse codebooks (Liu et al., 13 Nov 2025, Wen et al., 2024, Wang et al., 25 Sep 2025, Medi et al., 2023).
• Nonparametric Bayesian Clustering: 3D objects are abstracted as sequences of generative geometric primitives, where clustering assignments and optimal model complexity are managed via Dirichlet processes and Gibbs sampling (Wu et al., 2022).
• Hierarchical Semantic Tokenization: Token orderings are explicitly optimized to front-load semantic content—enabling any prefix to decode into a plausible coarse shape, as in Level-of-Semantics Tokenization (LoST) (Dutt et al., 18 Mar 2026).
• Programmatic Abstractions as Tokens: DSL-based shape encodings are compressed by auto-discovering and integrating high-level abstractions as compositional tokens, thereby reducing program length, exposing meaningful degrees of freedom, and improving generativity (Jones et al., 2023).

The table below compares representative shape tokenization frameworks:

Framework/Domain	Token Unit	Abstraction Mechanism
TCFormer (Vision) (Zeng et al., 2022)	Irregular image region	Progressive clustering
LoST (3D Gen.) (Dutt et al., 18 Mar 2026)	Semantic token (triplane)	Register tokens, nested dropout, RIDA loss
Protein VQ-VAE (Liu et al., 13 Nov 2025)	Backbone fragment	VQ over geometric latent
CAD-Tokenizer (Wang et al., 25 Sep 2025)	CAD primitive	VQ-VAE with primitive pooling
VQShape (Time-Series) (Wen et al., 2024)	Waveform fragment	Patch-level VQ, diversity constraint
Bayesian primitives (Wu et al., 2022)	Geometric primitive param	DP + optimization
ShapeCoder DSL (Jones et al., 2023)	Program abstraction	Recognition + e-graph macros

3. Mathematical Foundations and Tokenization Algorithms

Shape abstraction relies on a range of mathematical formalisms:

• Vector quantization: Given an encoder output $z$, assign to the nearest codebook entry $e_k$, $k^* = \arg\min_k \|z - e_k\|_2$, and learn codebooks via codebook/commitment losses to balance reconstruction fidelity and code diversity.
• Density-peaks clustering: For token merging in vision, local density $E$0 and distance indicator $E$1, jointly scored as $E$2, are used to select cluster centers; merging and assignment are executed post hoc based on semantic/importance similarity (Zeng et al., 2022).
• Nonparametric Bayesian assignment: CRP prior and infinite mixtures yield a variable number of geometric primitive tokens (Wu et al., 2022).
• Hierarchical prefix dropout: Enforces semantic compactness by training with randomly varying prefix lengths, directly shaping token ordering to align with the "any-prefix" property (Dutt et al., 18 Mar 2026).
• Semantic alignment and distillation: Relational losses (e.g., RIDA) align learned token embeddings to reference semantic spaces, imposing global, inter-instance, and intra-instance structure (Dutt et al., 18 Mar 2026).

Algorithms are typically structured into stages: (1) encoding and patch extraction, (2) abstraction via VQ/clustering/clustering+parametric optimization, (3) optional codebook/dictionary learning, and (4) transformation into token sequences/sets for autoregressive modeling or classification.

4. Architectural Innovations and Token Integration

Abstracted shape tokenization leads to nontrivial architectural modifications:

• Flexible Token Backbones: Multi-stage processing (TCFormer, LoST) alternates between feature refinement and region-adaptive merging, with hierarchical channel scaling and spatial-reduction layers to manage computational complexity (Zeng et al., 2022, Dutt et al., 18 Mar 2026).
• Token-Efficient Transformers: By allocating tokens adaptively—proportional to semantic or geometric importance—models avoid redundancy and concentrate capacity on detail-critical regions. For instance, LoST achieves state-of-the-art 3D shape generation with only 128–512 semantic tokens, outperforming spatially regular tokenizers with orders-of-magnitude more tokens (Dutt et al., 18 Mar 2026).
• Structural Token Decoders: Decoders are designed to exploit token semantic alignment (e.g., Structure Token for segmentation employs repeated attention between global tokens and dense features, progressively improving mask fidelity) (Lin et al., 2022).
• Grammar- and Abstraction-Aware Decoders: In CAD and visual programming, finite-state automata or DSL interpreters enforce syntactic and semantic validity over tokenized abstractions during sequence decoding (Wang et al., 25 Sep 2025, Jones et al., 2023).
• Attention Integration: Shape tokens serve as cross-attention keys/values, often mixed with image/textual elements (ShapeWords for 3D-guided text-to-image blends Point-BERT shape tokens with OpenCLIP text tokens, using prompt-delta networks for residual fusion) (Petrov et al., 2024).

5. Empirical Evaluation and Performance Gains

Across domains, the shift to shape-based tokenization yields marked empirical gains:

• Vision (Human-centric tasks): TCFormer achieves superior localization and delineation of small, high-importance regions (hands, feet, facial landmarks), with up to 13.6% improvement over fixed-grid tokenizers in small body keypoint AP, attributable to semantic token alignment (Zeng et al., 2022).
• 3D Generation: LoST matches or surpasses prior methods at 0.1–10% of the token budget; e.g., a single semantic token can produce a plausible category-level shape, and 1–4 tokens outperform thousands of LoD-style tokens on Chamfer Distance and FID (Dutt et al., 18 Mar 2026). Octree-based schemes cut token count by 15–50% while increasing geometric fidelity (Deng et al., 3 Apr 2025).
• Protein Structure: Structural synonym redundancy enables fast local fluctuation sampling, achieving per-target RMSF correlation of 0.84 vs. 0.85 for expensive MD-based methods, at orders-of-magnitude lower computational cost (Liu et al., 13 Nov 2025).
• Time-Series: VQShape produces interpretable, domain-general tokens enabling near-SOTA zero-shot classification performance across diverse time-series datasets, validating the generality of shape abstraction (Wen et al., 2024).
• CAD and Programs: Primitive/abstraction-aware tokenization leads to 80–90% sequence compression and lower error/invalidity rates; integrating programmatic abstractions as tokens in DSLs halves program length and reduces degrees of freedom, enhancing generative flexibility (Wang et al., 25 Sep 2025, Jones et al., 2023).

6. Interpretability, Generalization, and Compositionality

A prominent advantage of shape tokenization is the emergence of interpretability and compositional flexibility:

• Interpretable Tokens: Codebooks can be visualized as canonical shape units (e.g., VQShape histograms map directly to waveform prototypes; PCT for poses maps tokens to consistent human limb substructures) (Wen et al., 2024, Geng et al., 2023).
• Compositional Abstraction: Hierarchical architectures (PCT, ShapeCoder, LoST) benefit from modular token composition, enabling coarse-to-fine refinement, parametric editing, and generalization across classes with minimal architectural change (Jones et al., 2023, Dutt et al., 18 Mar 2026).
• Cross-domain Transfer: Shape tokens pretrained over heterogeneous corpora (VQShape, protein VQ-VAEs) exhibit strong zero-shot or MSA-free generalization, supporting the claim that a small vocabulary of abstracted tokens can cover broad semantic ground (Wen et al., 2024, Liu et al., 13 Nov 2025).

Limitations sometimes appear: absence of explicit structural regularization may restrict token generality in open-world segmentation; fixed token budgets may limit applicability to expanding class sets; purely local synonym swapping in protein tokenization cannot capture collective structural transitions (Lin et al., 2022, Liu et al., 13 Nov 2025).

7. Implications and Future Directions

The abstraction of shapes as tokens is catalyzing a transition toward foundation models that natively operate over semantically and structurally grounded representations. Key implications and directions include:

• Token Scalability and Compression: Continued progress in adaptive, semantically-informed tokenization promises more compact, expressive models across modalities.
• Symbolic Integration: Discrete shape vocabularies enable symbolic world modeling, planning, interpretable downstream reasoning, and program synthesis (Baek et al., 17 Jun 2025, Jones et al., 2023).
• Unified Multi-modal Representations: Coupling shape tokens with language, vision, and programmatic interfaces (e.g., text-guided CAD, shape-conditioned diffusion) will accelerate advances in controllable generation and editing (Wang et al., 25 Sep 2025, Petrov et al., 2024).
• Hybrid Continuous–Discrete Tokenization: Several recent models pursue continuous-valued tokens for downstream diffusion or attention, avoiding quantization artifacts but preserving abstraction via intelligent parameterization (e.g., flow-matching shape tokens (Chang et al., 2024), LoST semantic tokens (Dutt et al., 18 Mar 2026)).
• Towards Universal Structural Vocabularies: A plausible implication is the emergence of universal, cross-domain token vocabularies, with structural primitives reused within and across modalities—including extension to macromolecules, physical simulations, and abstract graphs (Liu et al., 13 Nov 2025, Jones et al., 2023).

In summary, abstracted shapes as tokens instantiate a generalizable, interpretable, and compositionally rich formalism for converting geometric, structural, and temporal data into discrete units of modeling, underpinning generative, predictive, and symbolic systems across computational science and engineering.