Hierarchical Object-Centric Tokenization

• HOCT is a hierarchical, object-/region-centric tokenization paradigm that replaces fixed patches with semantically meaningful object tokens.
• It organizes hierarchical representations to bridge high-level semantic planning with low-level control, scene generation, and neural rendering.
• Its instantiations span robotics, generative modeling, video-language systems, and adaptive vision tokenization, enhancing efficiency and structure.

Searching arXiv for papers on hierarchical object-centric tokenization and closely related object-centric tokenization methods. Hierarchical Object-Centric Tokenization (HOCT) is a hierarchical, object-/region-centric tokenization paradigm in which the basic representational units are objects, object-relative attributes, or object-grounded aggregates rather than fixed patches or monolithic scene embeddings. In the recent literature, HOCT appears explicitly in some systems and is realized implicitly in others: as object-canonicalized skill primitives that bridge high-level semantic planning and low-level control in robot manipulation, as global-to-object latent hierarchies in scene generation, as triangle-to-object aggregation in neural rendering, as detect–segment–track object tokens in video-LLMs, and as differentiable region tokenizers for Vision Transformers (Zhao et al., 24 Oct 2025, Wang et al., 2023, Du et al., 29 Jun 2026, Feng et al., 2024). This suggests that HOCT is best understood as a representational principle: hierarchy organizes scale and composition, while object-centricity provides semantically meaningful and geometrically grounded token content.

1. Conceptual definition and scope

In robot manipulation, the framework "Generalizable Hierarchical Skill Learning" (GSL) realizes HOCT by making objects and object-relative skills the central tokens that bridge the high-level vision-LLM and the low-level visual-motor policy. Demonstrations are parsed into object-grounded skill primitives, canonicalized in the object frame, and composed at test time as a sequence of skill–object pairs executed by a low-level controller (Zhao et al., 24 Oct 2025).

In generative modeling, "Slot-VAE" implements hierarchical object-centric tokenization by coupling a global scene token $z_g$ with a set of object-centric slot latents $z_{1:K}$. The hierarchy is explicit: $z_g \rightarrow z_{1:K} \rightarrow$ per-slot renders $\rightarrow$ composited scene. The global token governs coherence, while the slots act as object tokens (Wang et al., 2023).

In feed-forward neural rendering, "RenderFormer++" uses HOCT to aggregate triangle-level features into compact object-level tokens via cross-attention with learnable queries. The hierarchy is triangles $\rightarrow$ object tokens $\rightarrow$ global transport tokens, with the compact global sequence feeding Physics-Informed Transport Guidance (PITG) (Du et al., 29 Jun 2026).

In video-language modeling, "VideoOrion" is not presented as a formal hierarchy, but it implicitly organizes information as frame-level dense features $\rightarrow$ per-frame instance masks $\rightarrow$ track-level mask lists $\rightarrow$ temporally fused object tokens $\rightarrow$ textual reasoning in the LLM. Its object tokens are built from a detect–segment–track pipeline and aggregated spatial-temporal object features (Feng et al., 2024).

In adaptive vision tokenization, "Differentiable Hierarchical Visual Tokenization" states directly that HOCT and the paper’s Differentiable Hierarchical Tokenization ($z_{1:K}$0HT or dHT) are the same concept: a hierarchical, object-/region-centric tokenizer trained end-to-end and retrofittable to pretrained Vision Transformers (Aasan et al., 4 Nov 2025).

A common misconception is to treat HOCT as a single architecture. The literature instead uses the same organizing idea in several incompatible but related forms: latent-variable scene decomposition, token aggregation over meshes, object-centric control interfaces, track-level video tokens, bottom-up clustering, and differentiable region partitioning. The common denominator is that token boundaries are tied to objects, parts, or coherent regions rather than to a fixed grid.

2. Token formation and hierarchical composition

The token-construction mechanism depends on the domain, but the object-centric commitment is consistent. In GSL, an object instance is represented by a multi-view binary mask $z_{1:K}$1 predicted per view and lifted to a 3D object point cloud $z_{1:K}$2 using calibrated RGB-D cameras. A skill primitive is the triplet $z_{1:K}$3, where $z_{1:K}$4 is a discrete skill label, $z_{1:K}$5 locates the object, and $z_{1:K}$6 is the interaction-relevant trajectory segment. Canonicalization via a mask-as-frame operator $z_{1:K}$7 yields $z_{1:K}$8, and the low-level controller consumes the skill token $z_{1:K}$9, with $z_g \rightarrow z_{1:K} \rightarrow$0 encoded as a learned language embedding $z_g \rightarrow z_{1:K} \rightarrow$1 (Zhao et al., 24 Oct 2025).

In RenderFormer++, the primitive inputs are per-triangle features built from geometry, radiometric/material features, Relative Spatial Positional Encoding (RSPE), and object membership. Triangles are grouped by object, encoded into per-object triangle sequences $z_g \rightarrow z_{1:K} \rightarrow$2, and aggregated with shared learnable queries $z_g \rightarrow z_{1:K} \rightarrow$3 through cross-attention:

$z_g \rightarrow z_{1:K} \rightarrow$4

This produces a fixed $z_g \rightarrow z_{1:K} \rightarrow$5 object tokens per object, regardless of object triangle count, and the global sequence is formed by concatenating $z_g \rightarrow z_{1:K} \rightarrow$6 (Du et al., 29 Jun 2026).

In VideoOrion, token construction proceeds through expert-model preprocessing. GroundingDINO proposes boxes on key frames, SAM converts them to masks, and XMem tracks masks across time, producing mask lists $z_g \rightarrow z_{1:K} \rightarrow$7. For each tracked object $z_g \rightarrow z_{1:K} \rightarrow$8, patch-level CLIP or SigLIP features are spatially pooled within the object mask and then temporally pooled across the track, yielding a single object token $z_g \rightarrow z_{1:K} \rightarrow$9 (Feng et al., 2024).

In Slot-VAE, the hierarchy is latent rather than geometric. A CNN backbone and slot attention infer slot embeddings $\rightarrow$0, a shared MLP maps them to Gaussian parameters of $\rightarrow$1, and a separate global encoder infers $\rightarrow$2. In the prior path, $\rightarrow$3 generates a feature map that is processed by a second slot attention module—sharing weights and random initialization with the posterior path—to parameterize $\rightarrow$4 (Wang et al., 2023).

In dHT, tokenization begins at the pixel lattice. A similarity kernel $\rightarrow$5 selects each vertex’s most similar neighbor,

$\rightarrow$6

and connected components of the induced edge set define regions at the next level. Repeated contraction builds a hierarchy of connected partitions, after which information criteria select an optimal partition $\rightarrow$7. Token features are extracted by mean injection and adaptive mask blending, making the region tokens compatible with standard ViT backbones (Aasan et al., 4 Nov 2025).

Agglomerative Token Clustering (ATC) provides a parameter-free bottom-up variant of hierarchical object-centric tokenization. Starting from singleton spatial tokens, it iteratively merges the closest clusters under single, complete, or average linkage, stopping either at a target keep rate or a distance threshold. The paper’s qualitative evidence shows that this local-first merging tends to preserve coherent semantic units such as the bird, pole, and background as separate clusters longer than partition-based alternatives (Haurum et al., 2024).

3. Mathematical structure and learning objectives

The mathematical role of HOCT is not merely compression; it defines the conditioning structure under which learning is performed. In GSL, object-frame normalization removes global placement variance by recentering at an anchor point $\rightarrow$8 sampled on $\rightarrow$9, so that

$\rightarrow$0

The world-frame execution mapping is translation-only:

$\rightarrow$1

The low-level controller is trained in the object frame with a diffusion noise-prediction objective,

$\rightarrow$2

which the paper describes as behavior cloning in the object frame (Zhao et al., 24 Oct 2025).

Slot-VAE formalizes HOCT as a hierarchical VAE:

$\rightarrow$3

with approximate posterior

$\rightarrow$4

Its ELBO is

$\rightarrow$5

augmented with

$\rightarrow$6

The auxiliary term regularizes slots toward object-level and attribute-level disentanglement (Wang et al., 2023).

In RenderFormer++, HOCT changes the computational regime. Triangle-level self-attention scales as $\rightarrow$7 per layer, while object-centric aggregation costs approximately $\rightarrow$8 and global attention over object tokens costs $\rightarrow$9. For the paper’s typical values, $\rightarrow$0, $\rightarrow$1, and $\rightarrow$2, so triangle self-attention at $\rightarrow$3 triangles is approximately $\rightarrow$4, whereas HOCT global attention at $\rightarrow$5 tokens is approximately $\rightarrow$6, with aggregation cross-attention approximately $\rightarrow$7 (Du et al., 29 Jun 2026).

In dHT, hierarchical model selection is explicit. The information criterion has the form

$\rightarrow$8

with degrees of freedom estimated from graph volume:

$\rightarrow$9

The selected region features are injected back into the image by

$\rightarrow$0

and the tokenizer can be pretrained with

$\rightarrow$1

The pruning decision is discrete, but gradients pass through the selected features via mean injection and weighted aggregation (Aasan et al., 4 Nov 2025).

The active-inference formulation of object-centric scene representation is also hierarchical. The generative model factorizes over objects, with identity $\rightarrow$2, allocentric translation $\rightarrow$3, pose $\rightarrow$4, object-centric crop $\rightarrow$5, image-space center $\rightarrow$6, and scale $\rightarrow$7. The variational free energy decomposes into terms for pose transition, object category classification, object-centric cropping, object positions, object-centric observation likelihood, and full observation likelihood; action selection minimizes expected free energy, balancing preferences over goal tokens and information gain about object position (Maele et al., 2023).

4. Instantiations across research areas

The same HOCT principle serves different computational roles in different subfields.

Work	Hierarchy	Downstream role
GSL (Zhao et al., 24 Oct 2025)	skill labels, masks, canonicalized point clouds, skill embeddings	interface between high-level planning and low-level execution
Slot-VAE (Wang et al., 2023)	global scene token $\rightarrow$8 object slots $\rightarrow$9 masked components	coherent structured scene generation
RenderFormer++ (Du et al., 29 Jun 2026)	triangle tokens $\rightarrow$0 object tokens $\rightarrow$1 global transport tokens	scalable global illumination rendering
VideoOrion (Feng et al., 2024)	patch features $\rightarrow$2 masks/tracks $\rightarrow$3 object tokens	video QA and video-based referring
dHT (Aasan et al., 4 Nov 2025)	pixels $\rightarrow$4 connected regions across hierarchy $\rightarrow$5 IC-selected partition	adaptive ViT tokenization for classification and segmentation
ATC (Haurum et al., 2024)	spatial tokens $\rightarrow$6 agglomerative clusters	token reduction with object/part coherence
Active inference (Maele et al., 2023)	scene state $\rightarrow$7 object identity/pose/translation tokens	active perception and viewpoint selection

In manipulation, the interface is action-oriented. The high-level module predicts a sequence of pairs $\rightarrow$8 from task text and current scene, the mask is lifted to $\rightarrow$9, canonicalized to $\rightarrow$0, and EquiDiff predicts the canonical trajectory $\rightarrow$1 before mapping back to the world frame. Free-space motion is delegated to a motion planner, so local manipulation remains object-relative (Zhao et al., 24 Oct 2025).

In scene generation, HOCT is coherence-oriented. Slot-VAE uses a global bottleneck to generate slots conditioned on scene structure; slot attention is run in both posterior and prior paths, with shared weights and shared random initialization values to stabilize slot order matching between $\rightarrow$2 and $\rightarrow$3 (Wang et al., 2023).

In neural rendering, HOCT is scalability-oriented. Object tokens summarize geometry, normals, materials, and emission at the object level so that PITG can model transport interactions between objects rather than between all triangles. Local geometric detail is then reintroduced by the geometry-guided decoder through G-buffer features (Du et al., 29 Jun 2026).

In video-language modeling, HOCT is semantics-oriented. VideoOrion concatenates video-centric context tokens and object-centric tokens in the LLM embedding space, using the prompt template User: <v> Here is a list of objects and instances in the video: <o_1>,<o_2>,...,<o_N> <Instruction> Assistant: <Answer>. The paper reports that attention on object tokens varies strongly with the question, and that the tokens naturally support video-based referring (Feng et al., 2024).

In adaptive vision tokenization, HOCT is a tokenizer replacement rather than a downstream latent interface. dHT replaces the fixed patch tokenizer with image-adaptive connected regions while remaining backward-compatible with existing architectures. ATC occupies a different point in the design space: it is inserted between multi-head self-attention and the MLP in a ViT block and merges tokens bottom-up without extra learnable parameters (Aasan et al., 4 Nov 2025, Haurum et al., 2024).

5. Empirical performance and ablations

The empirical literature consistently ties HOCT-style representations to generalization, structure accuracy, or efficiency, but the measured effect depends on the task.

In GSL, the effect is most directly attributed to object-canonicalized skills. On GemBench, GSL trained with only $\rightarrow$4 demonstrations per task achieves a mean $\rightarrow$5 success rate on zero-shot testing tasks and outperforms strong baselines trained with $\rightarrow$6 demonstrations by $\rightarrow$7 on unseen tasks; on training tasks with spatial variation, it achieves $\rightarrow$8 mean, slightly above the best baseline at $\rightarrow$9. In real-world experiments with $\rightarrow$0 demonstrations per source task, GSL surpasses a baseline trained with $\rightarrow$1 more data across six tasks, including $\rightarrow$2 versus $\rightarrow$3 on Teabag in cup, $\rightarrow$4 versus $\rightarrow$5 on Teabag in red cup, and $\rightarrow$6 versus $\rightarrow$7 on Prepare Tea. The ablations isolate canonicalization and interface design: removing canonicalization drops mean success to $\rightarrow$8; action keyposes or heatmaps as the interface reduce performance to $\rightarrow$9–$z_{1:K}$00; including the approach phase in skills reduces mean performance to $z_{1:K}$01; the complete model reaches $z_{1:K}$02 mean on the ablation set (Zhao et al., 24 Oct 2025).

In Slot-VAE, the key evidence is that hierarchical linkage from global token to object tokens improves sample quality and scene structure accuracy. The model achieves ARI-FG of approximately $z_{1:K}$03 on ObjectsRoom and approximately $z_{1:K}$04 on ShapeStacks, and FID of $z_{1:K}$05 on ObjectsRoom, $z_{1:K}$06 on ShapeStacks, and $z_{1:K}$07 on Arrow Room. On Arrow Room, structure accuracy (S-Acc) is $z_{1:K}$08, compared with $z_{1:K}$09 for GENESIS, $z_{1:K}$10 for GENESIS-V2, and $z_{1:K}$11 for SRI. The hierarchical ablations are severe: Slot-VAE-MLP gives FID $z_{1:K}$12 and S-Acc $z_{1:K}$13; Slot-VAE-Transformer gives FID $z_{1:K}$14 and S-Acc $z_{1:K}$15; removing weight sharing gives FID $z_{1:K}$16 and S-Acc $z_{1:K}$17; removing initialization value sharing gives FID $z_{1:K}$18 and S-Acc $z_{1:K}$19 (Wang et al., 2023).

In RenderFormer++, the central empirical point is scalability. On the small-scale test split, the model reports $z_{1:K}$20, $z_{1:K}$21, $z_{1:K}$22, and $z_{1:K}$23; on the large-scale test split, $z_{1:K}$24, $z_{1:K}$25, $z_{1:K}$26, and $z_{1:K}$27. The paper states that RenderFormer fails to train on the large-scale dataset due to memory, whereas RenderFormer++ trains successfully, and that training is enabled on scenes up to approximately $z_{1:K}$28K triangles (Du et al., 29 Jun 2026).

In VideoOrion, adding object tokens to the video-centric branch produces measurable gains. VideoOrion reports $z_{1:K}$29 on MVBench, $z_{1:K}$30 on EgoSchema, $z_{1:K}$31 on Perception-Test, $z_{1:K}$32 on Video-MME without/with subtitles, and $z_{1:K}$33 accuracy with $z_{1:K}$34 score on ActivityNet-QA. VideoOrion+ improves to $z_{1:K}$35 on MVBench, $z_{1:K}$36 on Perception-Test, $z_{1:K}$37 on Video-MME, and $z_{1:K}$38 on ActivityNet-QA. In matched-data ablations, adding object tokens raises MVBench from $z_{1:K}$39 to $z_{1:K}$40 and Video-MME from $z_{1:K}$41 to $z_{1:K}$42. The referring variant reports BLEU-4 $z_{1:K}$43, METEOR $z_{1:K}$44, ROUGE-L $z_{1:K}$45, CIDEr $z_{1:K}$46, and SPICE $z_{1:K}$47 in zero-shot, with finetuned variants improving further (Feng et al., 2024).

In adaptive vision tokenization, dHT reports ImageNet1k linear-probing Acc@1 of $z_{1:K}$48 for retrofitted ViT-B16 versus a DEiT3 baseline at $z_{1:K}$49, and $z_{1:K}$50 for ViT-B16 trained from scratch versus patch tokenization at $z_{1:K}$51. For semantic segmentation, $z_{1:K}$52HT-ViT-B plus MLP reports mIoU $z_{1:K}$53 on ADE20k and $z_{1:K}$54 on COCO-Stuff164k. The paper also reports zero-shot raster-to-vector conversion quality of PSNR $z_{1:K}$55, SSIM $z_{1:K}$56, and MSE $z_{1:K}$57 (Aasan et al., 4 Nov 2025).

ATC provides evidence that bottom-up hierarchical clustering can preserve object/part coherence under aggressive token reduction. The paper reports that ATC-average achieves best FID $z_{1:K}$58 at keep rate $z_{1:K}$59, beating ToMe’s best $z_{1:K}$60; on MAE-pretrained ViT-B without fine-tuning, constant schedule $z_{1:K}$61 yields $z_{1:K}$62 for ATC-complete versus $z_{1:K}$63 for ToMe, and linear schedule $z_{1:K}$64 yields $z_{1:K}$65 for ATC-average/complete versus $z_{1:K}$66 for ToMe. In COCO detection/segmentation, off-the-shelf ATC at $z_{1:K}$67 outperforms ToMe by approximately $z_{1:K}$68 and approximately $z_{1:K}$69 (Haurum et al., 2024).

The active-inference object-centric scene model shows a different benefit: better active search. Across $z_{1:K}$70 scenes, SceneCCN + AIF reaches $z_{1:K}$71 success with errors $z_{1:K}$72 rad, $z_{1:K}$73 rad, and $z_{1:K}$74 m, compared with $z_{1:K}$75 for PoseCNN + Infogain and $z_{1:K}$76 for PoseCNN. In a fixed $z_{1:K}$77-object scene, SceneCCN + AIF achieves $z_{1:K}$78 success versus $z_{1:K}$79 for LEXA (Maele et al., 2023).

6. Limitations, misconceptions, and open directions

The main limitations are tied to how object tokens are obtained. GSL depends on accurate segmentation and language grounding; failures often stem from part-level segmentation errors such as SAM2 grouping a drawer cabinet as one object rather than identifying the handle. Its current formulation assumes a single manipulated object per skill, and extreme occlusions or ambiguous instructions can break mask lifting or skill assignment (Zhao et al., 24 Oct 2025).

VideoOrion inherits the cost and brittleness of a multi-model preprocessing stack. The reliance on RAM++, GroundingDINO, SAM, and XMem adds preprocessing cost, and inaccurate masks, occlusions, fast motion, crowded scenes, or small objects can entangle token semantics or dilute features under mask pooling and temporal averaging (Feng et al., 2024).

RenderFormer++ shows that object aggregation is not free. Aggressive aggregation may lose extremely fine per-triangle details when the object-token budget $z_{1:K}$80 is too small; thin structures and heterogeneous materials inside one object can blur distinctions, especially because the current dataset shares materials within objects (Du et al., 29 Jun 2026).

Slot-VAE retains standard slot-based weaknesses. Training assumes a fixed $z_{1:K}$81 even though slot attention can generalize beyond the training count; decoder bottlenecks can limit realism; relational reasoning is implicit through $z_{1:K}$82 rather than explicit; and slot-order sensitivity remains strong enough that removing weight or initialization sharing destabilizes training (Wang et al., 2023).

dHT solves object alignment at pixel granularity but not without trade-offs. Information-criterion selection is discrete, Gaussian residual modeling may be misspecified, hard masking creates sparsity and domain shift, and throughput remains lower than patch ViTs even though the gap narrows at higher resolutions (Aasan et al., 4 Nov 2025).

ATC demonstrates that hierarchical clustering can be object-centric, but not universally. Linkage choice matters, single linkage can suffer from chaining, CPU-bound agglomeration can be slow, and object-centricity may weaken in cluttered scenes or when feature similarity aligns more with texture than with object boundaries (Haurum et al., 2024).

A broader misconception is that HOCT is only a token-reduction technique. The literature shows otherwise. In GSL it is an action interface; in Slot-VAE it is a latent-variable hierarchy; in RenderFormer++ it is a scalability mechanism for physical transport modeling; in VideoOrion it is a semantic interface to an LLM; in dHT it is a replacement for the patch tokenizer; and in active inference it is a structured scene model used for epistemic control. A plausible implication is that future HOCT research will be less about a single canonical architecture than about how object-grounded tokens can be made transferable, compositional, and compatible with the learning objective of each domain.