---
title: Object-Centric Token Merging
url: https://www.emergentmind.com/topics/object-centric-token-merging
type: topic
---

# Object-Centric Token Merging

Object-centric token merging is a paradigm in transformer-based models—both in vision and multimodal architectures—in which tokens representing spatial regions, features, or entities are aggregated specifically according to object-level or semantically meaningful groupings, rather than by generic feature similarity or patch location. This operation reduces computational and memory demands, mitigates representational redundancy, and produces more semantically aligned token sequences by mapping each perceptual object (or agent entity) to a single, representative merged token. Distinguished from classical linear patch pruning, object-centric token merging is characterized by its use of segmentation masks, attention-based object saliency, or learned slot overlap to guide merge decisions, often with adaptivity to content and end-to-end differentiability.

## 1. Principles and Theoretical Basis

The core principle behind object-centric token merging is to exploit the natural grouping of visual input into discrete entities—a process that mirrors human visual perception, which first segments a scene into objects and then focuses attention for further reasoning [2511.14072]. Conventional transformers process images or videos as uniform grids of patch tokens, resulting in quadratic growth of token count with resolution (e.g., $N=4096$ for 1024×1024 images at 16×16 patching). Operations such as self-attention, with $O(N^2)$ time/memory, become a severe bottleneck. Existing patch-centric merging (e.g., ToMe) leverages token similarity, but is prone to loss of high-level semantics or spatial structure.

Object-centric token merging remedies these failures by explicitly mapping each object, agent, or salient region into a single token via:  
- **Semantic mask aggregation:** using segmentation masks (from models like Mask2Former or SAM) to identify and pool all patch embeddings for each object [2511.14072, 2511.14169].
- **Slot-based overlap fusion:** merging tokens with highly overlapping slot-attention regions (high Soft-IoU), consolidating fragmented object representation [2603.11246].
- **Saliency-driven or agent-centric stratification:** guiding merge operations by local feature saliency, entropy sharpness, or robot gripper localization [2403.13347, 2509.23655].

The general hypothesis is that this yields more efficient, semantically coherent tokenization aligned with downstream object-centric reasoning.

## 2. Algorithmic Strategies and Implementations

Object-centric token merging encompasses a spectrum of operational strategies, differing in their source of semantic partitioning, merge algebra, and scheduling:

### Slot-based Merging (Slot Attention)
“Slot merging” addresses over-segmentation in slot-based object-centric learning (OCL), especially where multiple slots compete for the same object due to overlapping attention [2603.11246]. The protocol is as follows:
- For slots $S_i$ with attention masks $A_{:,i}$, compute soft-IoU:
  \[
  \operatorname{IoU}(p, q) = \frac{\sum_{n=1}^N p_n q_n}{\sum_{n=1}^N (p_n + q_n - p_n q_n)}
  \]
- Iteratively merge the pair $(i^*,j^*)$ of slots with maximal $I_{ij}$ above a threshold $T$ via barycentric averaging:
  \[
  S_i \leftarrow w_i S_i + w_j S_j \quad\text{where}\quad w_i = \frac{\alpha_i}{\alpha_i+\alpha_j},\ w_j = 1-w_i
  \]
  updating the mask via $A_{:,i} \leftarrow A_{:,i} + A_{:,j}$.
- The process is end-to-end differentiable due to gradient flow through the merge weights. The threshold $T$ is set data-driven by the histogram triangle method.

### Segmentation-based Merging (Object-centric Pooling)
CORE and AdaTok both employ high-quality segmentation models:
- For each mask $P_n \in [0,1]^{H\times W}$, form the merged token 
  \[
  t_n = \frac{\sum_{i} \omega_{n,i} f_i}{\sum_{i} \omega_{n,i}}
  \]
  with weights $\omega_{n,i} = P_n[i]$ for soft aggregation, or hard assignment for exclusive tokenization [2511.14072, 2511.14169].
- Merged tokens are mapped to the LLM input space; centroids $c_n$ are used for positional coherence (centroid-guided sorting).
- The number of tokens $K$ adapts per image to the number of objects, achieving adaptive-rate compression.

### Saliency and Video-centric Methods
Saliency-aware token merging (vid-TLDR and Object-Centric Diffusion) use early attention maps or predefined masks to compute object/background saliency:
- Entropy/“sharpness” of attention rows is used to score foreground tokens; only high-sharpness tokens are kept/merged [2403.13347].
- Token merging is preferentially performed in background, or within spatio-temporal blocks in video, often restricting merges to maintain fidelity in edited or important regions [2401.05735].

### Token Merging as Semantic Binding in Text-to-Image
For text-to-image diffusion, composite tokens are constructed by aggregating CLIP embeddings of entity and modifier tokens, yielding representations with shared cross-attention maps. Auxiliary losses (entropy, semantic consistency) refine the merged token for binding integrity [2411.07132].

### Action-centric and Agent-aware Tokenization
Oat-VLA introduces explicit partitioning of visual tokens into object-centric groups and a high-resolution patch window centered on the robot gripper; final sequence comprises object tokens (via pooled masks on ViT features) and agent tokens (local neighborhood) [2509.23655].

## 3. Integration and Architectural Variants

Object-centric token merging is architecturally modular. In vision-language and multimodal systems:
- **Object Pooling Modules:** Merged or pooled object tokens are generated upstream of the transformer/LLM and concatenated with textual tokens [2511.14072, 2511.14169, 2509.23655].
- **Slot Merging in OCL Pipelines:** Inserted after Slot Attention convergence but before feature reconstruction or the decoder [2603.11246].
- **Inference-time Compression:** Most methods—especially for video or text-to-image—apply merging at inference, with no retraining required [2403.13347, 2401.05735, 2411.07132].
- **Positional Restoration:** Centroid- or mask-guided token ordering maintains spatial/layout cues after compression [2511.14072].
- **Adaptivity and Control:** Algorithmic schedules support fixed-rate (target $K$ tokens/image) or adaptive-rate (one token/object) modes; hyperparameters (merge ratio $r$, saliency threshold, barycentric weights) provide fine-grained control.

## 4. Empirical Performance and Computational Efficiency

Empirical metrics consistently demonstrate that object-centric token merging:
- **Retains performance under extreme compression:** AdaTok and CORE retain >95–97% of full performance using 2–10% as many tokens across benchmarks such as POPE, MME, SQA-I, VQA, and others [2511.14072, 2511.14169].
- **Reduces FLOPs and memory by $5\times$–$16\times$:** E.g., CORE achieves a $16\times$ reduction in FLOPs and $182\times$ reduction in KV cache memory in LLM inference (POPE, 63 vs 2880 tokens) [2511.14072].
- **Accelerates training and inference:** Oat-VLA's reduction from 256 to 16 tokens allows 2× speedup and larger batch sizes, converging to the same or higher action accuracy in half the wall-clock time on LIBERO robotic manipulation [2509.23655]. Object-centric 3D ToMe yields $8\times$–$10\times$ runtime and $17\times$ memory reductions in video editing without loss of temporal consistency [2401.05735].
- **Improves object factorization:** Slot merging in OCL improves mask IoU and object boundary consistency, surpassing prior adaptive methods and patch-centric merging (DINOSAUR pipeline, VOC/COCO benchmarks) [2603.11246].
- **Semantic and spatial preservation:** Hard object masks outperform soft aggregation in highly entangled settings; centroid-guided sorting restores layout cues essential for multimodal reasoning [2511.14072].

A comparison of representative methods and their compression-performance tradeoffs (as reported):

| Method          | # Tokens | Task/Benchmark        | Performance Retention (%) | Ref      |
|-----------------|----------|----------------------|--------------------------|----------|
| AdaTok          | ∼53      | MME                  | 97.6                     | 2511.14169 |
| CORE            | 63       | POPE                 | 97.4                     | 2511.14072 |
| Oat-VLA         | 16       | LIBERO (actions)     | ≈100 (converges 2× faster) | 2509.23655 |
| Slot merging    | varies   | VOC mIoU             | +3.6 over baseline       | 2603.11246 |

## 5. Limitations and Open Challenges

Current object-centric token merging methods are subject to several constraints:
- **Dependency on segmentation:** Quality depends on the accuracy and class coverage of the segmenter (e.g., Mask2Former supports 133 classes; out-of-distribution objects can fragment into multiple tokens, raising token count rather than causing merge errors) [2511.14072].
- **Memory and bandwidth bottlenecks:** Bottleneck shifts to the vision segmentation module; operator fusion and CUDA optimization are suggested to further exploit hardware savings.
- **No end-to-end mask learning:** Most pipelines utilize frozen upstream segmenters or object detectors. Integrating the mask generation step with downstream gradient flow and updating jointly remains an active research direction [2511.14072, 2511.14169].
- **Granularity control:** Adjusting merge aggressiveness (e.g., via mask confidence $\sigma$ or slot merging threshold $T$) entails trade-offs between fidelity and efficiency.

## 6. Applications and Research Directions

Object-centric token merging is being generalized across domains:
- **Vision-Language Models**: Token compression enables scaling LVLMs and MLLMs to higher resolutions and complex queries with strict resource envelopes [2511.14072, 2511.14169].
- **Video Transformers**: Spatiotemporal merging, with object/background selectivity, provides significant acceleration in video understanding and editing, including for temporally coherent diffusion models [2403.13347, 2401.05735].
- **Robotics and Action Planning**: By segregating agent-centric and object-centric tokens, models such as Oat-VLA encode control-critical features precisely, boosting learning speed and real-world performance [2509.23655].
- **Text-to-Image Synthesis**: Semantic binding of attributes and objects via composite token merging significantly increases integrity for complex multi-object compositional generation [2411.07132].

Ongoing directions cover end-to-end training with mask learning, dynamic or hierarchical merging schedules, integration with foundation segmentation models (e.g., SAM), and expansion to cross-modal object tokenization for unified perception-action architectures.

## 7. Relation to Prior Patch-based Token Merging

Object-centric token merging represents a conceptual advance over patch-similarity-based schemes such as ToMe [2210.09461], which merge purely by local key-space similarity. While ToMe often results in object part aggregation and achieves substantial acceleration, its lack of explicit semantic priors can suffer in complex scenes or under aggressive merging, leading to mixing of distinct semantics. Object-centric approaches address these shortcomings by:
- Using segmentation or attention-based saliency to prevent cross-object merging,
- Enabling “one object, one token” guarantees,
- Preserving agent or foreground information critical for reasoning or control.

Empirically, object-centric merging methods such as AdaTok, CORE, and slot merging outperform or match the speedup and compression achieved by ToMe, maintaining higher performance in semantic-rich and compositional tasks [2511.14169, 2511.14072, 2603.11246].

---

_Object-centric token merging stands as a foundational primitive for efficient, semantically coherent, and scalable token processing in object-aware and multimodal machine learning systems._

Source: https://www.emergentmind.com/topics/object-centric-token-merging