---
title: Hierarchical Object-Centric Tokenization
url: https://www.emergentmind.com/topics/hierarchical-object-centric-tokenization-hoct
type: topic
---

# Hierarchical Object-Centric Tokenization

Searching arXiv for recent papers on hierarchical object-centric tokenization and closely related object-centric tokenization methods.
Hierarchical Object-Centric Tokenization (HOCT) is a hierarchical, object-/region-centric tokenization paradigm in which the basic representational units are objects, object-relative attributes, or object-grounded aggregates rather than fixed patches or monolithic scene embeddings. In the recent literature, HOCT appears explicitly in some systems and is realized implicitly in others: as object-canonicalized skill primitives that bridge high-level semantic planning and low-level control in robot manipulation, as global-to-object latent hierarchies in scene generation, as triangle-to-object aggregation in neural rendering, as detect–segment–track object tokens in video-language models, and as differentiable region tokenizers for Vision Transformers [2510.21121] [2306.06997] [2606.30380] [2411.16156]. This suggests that HOCT is best understood as a representational principle: hierarchy organizes scale and composition, while object-centricity provides semantically meaningful and geometrically grounded token content.

## 1. Conceptual definition and scope

In robot manipulation, the framework "Generalizable Hierarchical Skill Learning" (GSL) realizes HOCT by making objects and object-relative skills the central tokens that bridge the high-level vision-language model and the low-level visual-motor policy. Demonstrations are parsed into object-grounded skill primitives, canonicalized in the object frame, and composed at test time as a sequence of skill–object pairs executed by a low-level controller [2510.21121].

In generative modeling, "Slot-VAE" implements hierarchical object-centric tokenization by coupling a global scene token $z_g$ with a set of object-centric slot latents $z_{1:K}$. The hierarchy is explicit: $z_g \rightarrow z_{1:K} \rightarrow$ per-slot renders $\rightarrow$ composited scene. The global token governs coherence, while the slots act as object tokens [2306.06997].

In feed-forward neural rendering, "RenderFormer++" uses HOCT to aggregate triangle-level features into compact object-level tokens via cross-attention with learnable queries. The hierarchy is triangles $\rightarrow$ object tokens $\rightarrow$ global transport tokens, with the compact global sequence feeding Physics-Informed Transport Guidance (PITG) [2606.30380].

In video-language modeling, "VideoOrion" is not presented as a formal hierarchy, but it implicitly organizes information as frame-level dense features $\rightarrow$ per-frame instance masks $\rightarrow$ track-level mask lists $\rightarrow$ temporally fused object tokens $\rightarrow$ textual reasoning in the LLM. Its object tokens are built from a detect–segment–track pipeline and aggregated spatial-temporal object features [2411.16156].

In adaptive vision tokenization, "Differentiable Hierarchical Visual Tokenization" states directly that HOCT and the paper’s Differentiable Hierarchical Tokenization ($\partial$HT or dHT) are the same concept: a hierarchical, object-/region-centric tokenizer trained end-to-end and retrofittable to pretrained Vision Transformers [2511.02652].

A common misconception is to treat HOCT as a single architecture. The literature instead uses the same organizing idea in several incompatible but related forms: latent-variable scene decomposition, token aggregation over meshes, object-centric control interfaces, track-level video tokens, bottom-up clustering, and differentiable region partitioning. The common denominator is that token boundaries are tied to objects, parts, or coherent regions rather than to a fixed grid.

## 2. Token formation and hierarchical composition

The token-construction mechanism depends on the domain, but the object-centric commitment is consistent. In GSL, an object instance is represented by a multi-view binary mask $m=\{m_v \in \{0,1\}^{H \times W}\}$ predicted per view and lifted to a 3D object point cloud $P=\{p_j \in \mathbb{R}^3\}$ using calibrated RGB-D cameras. A skill primitive is the triplet $(k,m,\tau_s)$, where $k \in \mathcal{K}$ is a discrete skill label, $m$ locates the object, and $\tau_s$ is the interaction-relevant trajectory segment. Canonicalization via a mask-as-frame operator $C_m$ yields $(k,P_c,\tau_c)$, and the low-level controller consumes the skill token $(k,P_c)$, with $k$ encoded as a learned language embedding $f_k \in \mathbb{R}^d$ [2510.21121].

In RenderFormer++, the primitive inputs are per-triangle features built from geometry, radiometric/material features, Relative Spatial Positional Encoding (RSPE), and object membership. Triangles are grouped by object, encoded into per-object triangle sequences $H_n$, and aggregated with shared learnable queries $Q \in \mathbb{R}^{K \times d}$ through cross-attention:
$$
\mathbf{F}_n = \mathrm{CrossAttn}(\mathbf{Q}, \mathbf{H}_n).
$$
This produces a fixed $K=n_q=8$ object tokens per object, regardless of object triangle count, and the global sequence is formed by concatenating $\{\mathbf{F}_n\}_{n=1}^{N}$ [2606.30380].

In VideoOrion, token construction proceeds through expert-model preprocessing. GroundingDINO proposes boxes on key frames, SAM converts them to masks, and XMem tracks masks across time, producing mask lists $M=\{M_1,\dots,M_{N_{oi}}\}$. For each tracked object $j$, patch-level CLIP or SigLIP features are spatially pooled within the object mask and then temporally pooled across the track, yielding a single object token $o_j \in \mathbb{R}^d$ [2411.16156].

In Slot-VAE, the hierarchy is latent rather than geometric. A CNN backbone and slot attention infer slot embeddings $s_{1:K}$, a shared MLP maps them to Gaussian parameters of $q(z_k \mid x)$, and a separate global encoder infers $z_g$. In the prior path, $z_g$ generates a feature map that is processed by a second slot attention module—sharing weights and random initialization with the posterior path—to parameterize $p(z_k \mid z_g)$ [2306.06997].

In dHT, tokenization begins at the pixel lattice. A similarity kernel $\kappa$ selects each vertex’s most similar neighbor,
$$
v_{\max} = \argmax_{u \in N(v)} \kappa(u,v),
$$
and connected components of the induced edge set define regions at the next level. Repeated contraction builds a hierarchy of connected partitions, after which information criteria select an optimal partition $\pi_*$. Token features are extracted by mean injection and adaptive mask blending, making the region tokens compatible with standard ViT backbones [2511.02652].

Agglomerative Token Clustering (ATC) provides a parameter-free bottom-up variant of hierarchical object-centric tokenization. Starting from singleton spatial tokens, it iteratively merges the closest clusters under single, complete, or average linkage, stopping either at a target keep rate or a distance threshold. The paper’s qualitative evidence shows that this local-first merging tends to preserve coherent semantic units such as the bird, pole, and background as separate clusters longer than partition-based alternatives [2409.11923].

## 3. Mathematical structure and learning objectives

The mathematical role of HOCT is not merely compression; it defines the conditioning structure under which learning is performed. In GSL, object-frame normalization removes global placement variance by recentering at an anchor point $t$ sampled on $P$, so that
$$
P_c = P - t, \qquad \tau_c = \tau - t.
$$
The world-frame execution mapping is translation-only:
$$
a_w^{pos} = a_c^{pos} + t_{wo}, \qquad a_w^{rot} = a_c^{rot}, \qquad a_w^{gripper} = a_c^{gripper}.
$$
The low-level controller is trained in the object frame with a diffusion noise-prediction objective,
$$
L_{\mathrm{diff}}(\theta) = \mathbb{E}_{\tau_c,t,\epsilon}\big[\|\epsilon-\hat{\epsilon}_\theta(P_c,f_k,\tau_c^{(t)},t)\|^2\big],
$$
which the paper describes as behavior cloning in the object frame [2510.21121].

Slot-VAE formalizes HOCT as a hierarchical VAE:
$$
p(x,z_g,z_{1:K}) = p(x \mid z_{1:K}) \prod_{k=1}^K p(z_k \mid z_g)\, p(z_g),
$$
with approximate posterior
$$
q(z_g,z_{1:K} \mid x) = q(z_g \mid x)\prod_{k=1}^K q(z_k \mid x).
$$
Its ELBO is
$$
L = \mathbb{E}_{q(z_g,z_{1:K}\mid x)}[\log p(x \mid z_{1:K})]
- \mathrm{KL}(q(z_g \mid x)\|p(z_g))
- \sum_k \mathbb{E}_{q(z_g \mid x)}[\mathrm{KL}(q(z_k \mid x)\|p(z_k \mid z_g))],
$$
augmented with
$$
L_{\mathrm{aux}} = - \sum_{k=1}^K \mathrm{KL}(q(z_k \mid x)\|N(0,I)).
$$
The auxiliary term regularizes slots toward object-level and attribute-level disentanglement [2306.06997].

In RenderFormer++, HOCT changes the computational regime. Triangle-level self-attention scales as $O(N_{\mathrm{tri}}^2)$ per layer, while object-centric aggregation costs approximately $O(K \cdot N_{\mathrm{tri}})$ and global attention over object tokens costs $O((N \cdot K)^2)$. For the paper’s typical values, $N_{\mathrm{tri}} \in [3{,}000,80{,}000]$, $N \in [5,70]$, and $K=8$, so triangle self-attention at $80{,}000$ triangles is approximately $6.4\times10^9$, whereas HOCT global attention at $560$ tokens is approximately $3.1\times10^5$, with aggregation cross-attention approximately $6.4\times10^5$ [2606.30380].

In dHT, hierarchical model selection is explicit. The information criterion has the form
$$
IC(\theta) = -2 \log L(\theta) + g(df_\theta),
$$
with degrees of freedom estimated from graph volume:
$$
df_{\pi_*} \propto \sum_{S \in \pi_*} \mathrm{Vol}_G(S)^{-1}.
$$
The selected region features are injected back into the image by
$$
x(v) = x(v) + Wf_*(v) - \bar{x}_*(v),
$$
and the tokenizer can be pretrained with
$$
L_{\mathrm{rec}} = \frac{1}{|V|}\sum_{v \in V} \|x(v)-Wf_*(v)\|_2^2.
$$
The pruning decision is discrete, but gradients pass through the selected features via mean injection and weighted aggregation [2511.02652].

The active-inference formulation of object-centric scene representation is also hierarchical. The generative model factorizes over objects, with identity $i_k$, allocentric translation $t_k$, pose $p_{k,t}$, object-centric crop $o_{k,t}$, image-space center $u_{k,t}$, and scale $\sigma_{k,t}$. The variational free energy decomposes into terms for pose transition, object category classification, object-centric cropping, object positions, object-centric observation likelihood, and full observation likelihood; action selection minimizes expected free energy, balancing preferences over goal tokens and information gain about object position [2302.03288].

## 4. Instantiations across research areas

The same HOCT principle serves different computational roles in different subfields.

| Work | Hierarchy | Downstream role |
|---|---|---|
| GSL [2510.21121] | skill labels, masks, canonicalized point clouds, skill embeddings | interface between high-level planning and low-level execution |
| Slot-VAE [2306.06997] | global scene token $\rightarrow$ object slots $\rightarrow$ masked components | coherent structured scene generation |
| RenderFormer++ [2606.30380] | triangle tokens $\rightarrow$ object tokens $\rightarrow$ global transport tokens | scalable global illumination rendering |
| VideoOrion [2411.16156] | patch features $\rightarrow$ masks/tracks $\rightarrow$ object tokens | video QA and video-based referring |
| dHT [2511.02652] | pixels $\rightarrow$ connected regions across hierarchy $\rightarrow$ IC-selected partition | adaptive ViT tokenization for classification and segmentation |
| ATC [2409.11923] | spatial tokens $\rightarrow$ agglomerative clusters | token reduction with object/part coherence |
| Active inference [2302.03288] | scene state $\rightarrow$ object identity/pose/translation tokens | active perception and viewpoint selection |

In manipulation, the interface is action-oriented. The high-level module predicts a sequence of pairs $(k,m)$ from task text and current scene, the mask is lifted to $P$, canonicalized to $P_c$, and EquiDiff predicts the canonical trajectory $\tau_c$ before mapping back to the world frame. Free-space motion is delegated to a motion planner, so local manipulation remains object-relative [2510.21121].

In scene generation, HOCT is coherence-oriented. Slot-VAE uses a global bottleneck to generate slots conditioned on scene structure; slot attention is run in both posterior and prior paths, with shared weights and shared random initialization values to stabilize slot order matching between $q(z_k \mid x)$ and $p(z_k \mid z_g)$ [2306.06997].

In neural rendering, HOCT is scalability-oriented. Object tokens summarize geometry, normals, materials, and emission at the object level so that PITG can model transport interactions between objects rather than between all triangles. Local geometric detail is then reintroduced by the geometry-guided decoder through G-buffer features [2606.30380].

In video-language modeling, HOCT is semantics-oriented. VideoOrion concatenates video-centric context tokens and object-centric tokens in the LLM embedding space, using the prompt template `User: <v> Here is a list of objects and instances in the video: <o_1>,<o_2>,...,<o_N> <Instruction>  Assistant: <Answer>`. The paper reports that attention on object tokens varies strongly with the question, and that the tokens naturally support video-based referring [2411.16156].

In adaptive vision tokenization, HOCT is a tokenizer replacement rather than a downstream latent interface. dHT replaces the fixed patch tokenizer with image-adaptive connected regions while remaining backward-compatible with existing architectures. ATC occupies a different point in the design space: it is inserted between multi-head self-attention and the MLP in a ViT block and merges tokens bottom-up without extra learnable parameters [2511.02652] [2409.11923].

## 5. Empirical performance and ablations

The empirical literature consistently ties HOCT-style representations to generalization, structure accuracy, or efficiency, but the measured effect depends on the task.

In GSL, the effect is most directly attributed to object-canonicalized skills. On GemBench, GSL trained with only $3$ demonstrations per task achieves a mean $82.3\%$ success rate on zero-shot testing tasks and outperforms strong baselines trained with $100$ demonstrations by $15.5\%$ on unseen tasks; on training tasks with spatial variation, it achieves $98.0\%$ mean, slightly above the best baseline at $97.6\%$. In real-world experiments with $3$ demonstrations per source task, GSL surpasses a baseline trained with $10\times$ more data across six tasks, including $9/10$ versus $6/10$ on Teabag in cup, $9/10$ versus $4/10$ on Teabag in red cup, and $6/10$ versus $3/10$ on Prepare Tea. The ablations isolate canonicalization and interface design: removing canonicalization drops mean success to $0.12$; action keyposes or heatmaps as the interface reduce performance to $0.10$–$0.17$; including the approach phase in skills reduces mean performance to $0.37$; the complete model reaches $0.83$ mean on the ablation set [2510.21121].

In Slot-VAE, the key evidence is that hierarchical linkage from global token to object tokens improves sample quality and scene structure accuracy. The model achieves ARI-FG of approximately $0.79$ on ObjectsRoom and approximately $0.80$ on ShapeStacks, and FID of $34.9$ on ObjectsRoom, $50.0$ on ShapeStacks, and $60.3$ on Arrow Room. On Arrow Room, structure accuracy (S-Acc) is $0.94$, compared with $0.11$ for GENESIS, $0.20$ for GENESIS-V2, and $0.18$ for SRI. The hierarchical ablations are severe: Slot-VAE-MLP gives FID $289$ and S-Acc $0.00$; Slot-VAE-Transformer gives FID $182.1$ and S-Acc $0.03$; removing weight sharing gives FID $215.5$ and S-Acc $0.00$; removing initialization value sharing gives FID $142.1$ and S-Acc $0.05$ [2306.06997].

In RenderFormer++, the central empirical point is scalability. On the small-scale test split, the model reports $L1 = 0.0861$, $MAPE = 0.1270$, $LPIPS = 0.0601$, and $SSIM = 0.9479$; on the large-scale test split, $L1 = 0.0899$, $MAPE = 0.3294$, $LPIPS = 0.1766$, and $SSIM = 0.7820$. The paper states that RenderFormer fails to train on the large-scale dataset due to memory, whereas RenderFormer++ trains successfully, and that training is enabled on scenes up to approximately $80$K triangles [2606.30380].

In VideoOrion, adding object tokens to the video-centric branch produces measurable gains. VideoOrion reports $58.3\%$ on MVBench, $57.6\%$ on EgoSchema, $53.4\%$ on Perception-Test, $46.1\%/48.0\%$ on Video-MME without/with subtitles, and $50.3\%$ accuracy with $3.5$ score on ActivityNet-QA. VideoOrion+ improves to $64.08\%$ on MVBench, $57.3\%$ on Perception-Test, $55.6\%/57.5\%$ on Video-MME, and $52.4\%/3.5$ on ActivityNet-QA. In matched-data ablations, adding object tokens raises MVBench from $41.9\%$ to $44.2\%$ and Video-MME from $44.1\%$ to $46.1\%$. The referring variant reports BLEU-4 $17.5$, METEOR $19.5$, ROUGE-L $43.0$, CIDEr $69.7$, and SPICE $28.4$ in zero-shot, with finetuned variants improving further [2411.16156].

In adaptive vision tokenization, dHT reports ImageNet1k linear-probing Acc@1 of $83.9$ for retrofitted ViT-B16 versus a DEiT3 baseline at $82.6$, and $83.2$ for ViT-B16 trained from scratch versus patch tokenization at $81.9$. For semantic segmentation, $\partial$HT-ViT-B plus MLP reports mIoU $53.2$ on ADE20k and $48.9$ on COCO-Stuff164k. The paper also reports zero-shot raster-to-vector conversion quality of PSNR $27.50$, SSIM $0.8541$, and MSE $0.00178$ [2511.02652].

ATC provides evidence that bottom-up hierarchical clustering can preserve object/part coherence under aggressive token reduction. The paper reports that ATC-average achieves best FID $33.22$ at keep rate $r=40\%$, beating ToMe’s best $33.57$; on MAE-pretrained ViT-B without fine-tuning, constant schedule $t=16$ yields $80.2$ for ATC-complete versus $78.5$ for ToMe, and linear schedule $t=16$ yields $67.1$ for ATC-average/complete versus $56.6$ for ToMe. In COCO detection/segmentation, off-the-shelf ATC at $r=50\%$ outperforms ToMe by approximately $4\,mAP^{box}$ and approximately $3\,mAP^{mask}$ [2409.11923].

The active-inference object-centric scene model shows a different benefit: better active search. Across $500$ scenes, SceneCCN + AIF reaches $69.0\%$ success with errors $\Delta \phi = 0.569 \pm 0.034$ rad, $\Delta \theta = 0.231 \pm 0.014$ rad, and $\Delta r = 0.067 \pm 0.004$ m, compared with $28.4\%$ for PoseCNN + Infogain and $16.2\%$ for PoseCNN. In a fixed $5$-object scene, SceneCCN + AIF achieves $62\%$ success versus $11\%$ for LEXA [2302.03288].

## 6. Limitations, misconceptions, and open directions

The main limitations are tied to how object tokens are obtained. GSL depends on accurate segmentation and language grounding; failures often stem from part-level segmentation errors such as SAM2 grouping a drawer cabinet as one object rather than identifying the handle. Its current formulation assumes a single manipulated object per skill, and extreme occlusions or ambiguous instructions can break mask lifting or skill assignment [2510.21121].

VideoOrion inherits the cost and brittleness of a multi-model preprocessing stack. The reliance on RAM++, GroundingDINO, SAM, and XMem adds preprocessing cost, and inaccurate masks, occlusions, fast motion, crowded scenes, or small objects can entangle token semantics or dilute features under mask pooling and temporal averaging [2411.16156].

RenderFormer++ shows that object aggregation is not free. Aggressive aggregation may lose extremely fine per-triangle details when the object-token budget $K$ is too small; thin structures and heterogeneous materials inside one object can blur distinctions, especially because the current dataset shares materials within objects [2606.30380].

Slot-VAE retains standard slot-based weaknesses. Training assumes a fixed $K$ even though slot attention can generalize beyond the training count; decoder bottlenecks can limit realism; relational reasoning is implicit through $z_g$ rather than explicit; and slot-order sensitivity remains strong enough that removing weight or initialization sharing destabilizes training [2306.06997].

dHT solves object alignment at pixel granularity but not without trade-offs. Information-criterion selection is discrete, Gaussian residual modeling may be misspecified, hard masking creates sparsity and domain shift, and throughput remains lower than patch ViTs even though the gap narrows at higher resolutions [2511.02652].

ATC demonstrates that hierarchical clustering can be object-centric, but not universally. Linkage choice matters, single linkage can suffer from chaining, CPU-bound agglomeration can be slow, and object-centricity may weaken in cluttered scenes or when feature similarity aligns more with texture than with object boundaries [2409.11923].

A broader misconception is that HOCT is only a token-reduction technique. The literature shows otherwise. In GSL it is an action interface; in Slot-VAE it is a latent-variable hierarchy; in RenderFormer++ it is a scalability mechanism for physical transport modeling; in VideoOrion it is a semantic interface to an LLM; in dHT it is a replacement for the patch tokenizer; and in active inference it is a structured scene model used for epistemic control. A plausible implication is that future HOCT research will be less about a single canonical architecture than about how object-grounded tokens can be made transferable, compositional, and compatible with the learning objective of each domain.

Source: https://www.emergentmind.com/topics/hierarchical-object-centric-tokenization-hoct