Region-Aware Cluster Discrimination (RICE)

• Region-Aware Cluster Discrimination (RICE) is a framework that uses region-level semantics to enhance object detection, OCR, and segmentation tasks.
• It integrates a Region Transformer layer atop a ViT backbone and applies a unified loss to simultaneously optimize object and OCR recognition.
• The system scales to billions of region samples using distributed training and achieves significant performance gains over image-level contrastive models.

Region-Aware Cluster Discrimination (RICE) is a framework for large-scale visual representation learning that explicitly incorporates region-level semantics, addressing key limitations of global feature alignment in dense prediction tasks such as object grounding, optical character recognition (OCR), and segmentation. Unlike prior vision-LLMs that primarily rely on contrastive objectives at the image level, RICE introduces a regional clustering paradigm and a unified loss that strengthens both object and OCR capabilities within a single, scalable classification framework (Xie et al., 26 Jul 2025).

1. Billion-Scale Region Dataset Construction

The RICE framework is enabled by the construction of a billion-scale region dataset, sourced from LAION-2B, COYO-700M, and SAM1B. All source images are required to have a minimum edge of at least 336 pixels. Object regions are generated using the SAM model (Kirillov et al., ICCV 2023) applied to LAION and COYO, while SAM1B provides human-annotated regions. Fine-grained segmentation masks whose bounding boxes have both sides less than 128 pixels are discarded. This process yields approximately 2 billion object region proposals from about 400 million images.

For annotation, each object region $r_{ij}$ is embedded using a 512-dimensional CLIP model, $f_{ij} = \text{CLIP}(r_{ij})$. A large-scale hierarchical k-means clustering (implemented with Faiss on GPU) produces K=1,000,000 cluster centers $C = \{c_1, ..., c_K\}$. Each region receives a single pseudo-label via nearest neighbor assignment: $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$.

OCR regions are generated by running PaddleOCR on LAION-2B and COYO-700M, retaining only text detections with confidence at least 0.7. This generates about 400 million OCR regions from approximately 50 million images. Each region’s recognized text $t_{ij}$ is tokenized using a byte pair encoding (BPE) tokenizer to produce multi-token labels $y_{ij}^{ocr} = \text{tokenize}(t_{ij})$.

The resulting datasets are organized as two parallel corpora: (1) region image $\rightarrow$ single object cluster label and (2) region image $\rightarrow$ multi-token OCR label, with a combined scale of 2 billion object regions and 0.4 billion OCR regions.

2. Region Transformer Layer and Embedding Extraction

RICE introduces a Region Transformer layer atop a Vision Transformer (ViT) backbone to produce region-specific embeddings. The model first applies $L_1$ standard ViT self-attention layers to image patch tokens (including $z_{cls}$ for the global image and $f_{ij} = \text{CLIP}(r_{ij})$0 for spatial patches), yielding global embeddings $f_{ij} = \text{CLIP}(r_{ij})$1. Subsequently, $f_{ij} = \text{CLIP}(r_{ij})$2 Region Transformer layers are applied, designed to extract $f_{ij} = \text{CLIP}(r_{ij})$3 region-specific embeddings $f_{ij} = \text{CLIP}(r_{ij})$4.

The core mechanism is masked region attention. For each image, up to $f_{ij} = \text{CLIP}(r_{ij})$5 regions are sampled, each with a binary visibility mask $f_{ij} = \text{CLIP}(r_{ij})$6 that restricts attention to token indices within the corresponding region. Embedding extraction across a batch proceeds by gathering region queries ($f_{ij} = \text{CLIP}(r_{ij})$7), keys ($f_{ij} = \text{CLIP}(r_{ij})$8), and values ($f_{ij} = \text{CLIP}(r_{ij})$9), and applying:

$C = \{c_1, ..., c_K\}$0

This yields one $C = \{c_1, ..., c_K\}$1-dimensional embedding per region. Both global and regional representations can be extracted within a single forward pass.

3. Unified Region Cluster Discrimination Loss

RICE unifies region-based object recognition and OCR via a margin-based cluster-center discriminative loss. All embeddings and cluster centers are $C = \{c_1, ..., c_K\}$2 normalized; a global scale $C = \{c_1, ..., c_K\}$3 and additive angular margin $C = \{c_1, ..., c_K\}$4 are applied for positive pairs. Negative sampling is performed by randomly selecting $C = \{c_1, ..., c_K\}$5 negative centers ($C = \{c_1, ..., c_K\}$6 by default).

• Object Region Loss: Given predicted region embedding $C = \{c_1, ..., c_K\}$7 and its true cluster center $C = \{c_1, ..., c_K\}$8,

$C = \{c_1, ..., c_K\}$9

where $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$0 and $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$1 is the set of negative cluster centers.

• OCR Region Loss: For OCR regions with $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$2 positive tokens (embedding set $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$3),

$y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$4

$y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$5 contains token centers for the detected tokens, $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$6 are negative token centers.

The joint classification loss for a batch is given by $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$7, with $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$8. Both object and OCR tasks share the same cluster-center architecture, enabling fully unified training and parameter sharding through Partial-FC.

4. Vision and OCR Integration with Large-Scale Distributed Training

A balanced region selector ensures that each minibatch contains a mix of object and OCR regions, sampled up to $y_{ij}^{object} = \arg\min_k \|f_{ij} - c_k\|_2$9 regions per image. The model supports distributed data-parallel training across 64 GPUs (with a global batch of 32,000 images). Cluster center matrices ($t_{ij}$0) are sharded across GPUs and only a sampled $t_{ij}$1-fraction of negatives are communicated per GPU, following the PartialFC approach.

The Faiss hierarchical k-means clustering process for the billion-scale region set can be completed offline within approximately 10 hours on 64 GPUs. RICE supports both ViT-B/16 and ViT-L/14 architectures, with resolution schedules spanning $t_{ij}$2 up to $t_{ij}$3, and all training conducted using AdamW optimizer.

5. Empirical Evaluation and Benchmark Results

5.1 Pretraining and Downstream Benchmarks

RICE is pretrained on 13 billion region samples (from LAION-2B, COYO-700M, SAM1B). Backbones include ViT-B/16 and ViT-L/14. The system is evaluated on:

• Vision-only dense benchmarks: COCO detection/segmentation, LVIS detection/segmentation, and Roboflow100.
• Multimodal LLMs (MLLMs): LLaVA-NeXT, LLaVA-OneVision, and LISA frameworks, covering benchmarks such as OCRBench, DocVQA, InfoVQA, TextVQA, ChartQA, LiveXivVQA, AI2D, MMBench, MME, POPE, and RealworldQA.
• Referring segmentation: refCOCO, refCOCO+, refCOCOg using LISA-style two-stage training.
• Video tracking probe: OSTrack on LaSOT, TrackingNet, GOT-10k, TNL2K, using two additional transformer blocks atop the frozen backbone.

5.2 Quantitative Improvements

Task/Metric	RICE ViT-L/14-336	Baseline / SOTA	Relative Gain
InfoVQA	+9.8 pts	CLIP L/14-336	↑ notable
DocVQA	+3.98 pts	CLIP L/14-336	↑ notable
OCRBench avg	+50 pts	CLIP L/14-336	↑ large
COCO Detection AP	38.9%	SigLIP L/16	+3.9 pts
COCO Segmentation AP	31.5%	SigLIP L/16	+3.4 pts
LVIS Detection AP	26.5%	SigLIP L/16	+4.7 pts
LVIS Segmentation AP	21.4%	SigLIP L/16	+4.1 pts
Roboflow100 avg	26.5%	Baseline	+2.4 pts
refCOCO/-/g mIoU	+2.0 pts	CLIP	↑
LaSOT Succ/Prec	60.24/63.16	MLCD: 58.05/60.75	↑

Ablation studies indicate optimal settings: number of regions $t_{ij}$4 per image, cluster count $t_{ij}$5M–2M (COCO/LVIS), negative ratio $t_{ij}$6–$t_{ij}$7, and approximately 10 OCR tokens per region.

Qualitative attention visualizations (PCA-projected) show temporally consistent behavior on video tracking.

6. Limitations and Prospects

RICE’s dependence on the SAM model exposes it to error propagation from segmentation masks. OCR region quality is limited by PaddleOCR’s handling of stylized or complex text. The current region clustering is static and not updated during training; integrating learnable or online cluster centers represents an avenue for future research. Extending the approach to support spatio-temporal region proposals for video or 3D point-cloud regions is a natural direction for expansion.

7. Code, Pretrained Models, and Reproducibility

Code, pretraining and finetuning scripts, model weights, and reproducibility recipes are publicly released at https://github.com/deepglint/MVT. The repository includes:

• make_regions.py for region curation (SAM and PaddleOCR pipeline)
• train_rice.py with ViT and Region Transformer integration and PartialFC loss
• Downstream task configurations for Detectron2, LLaVA-NeXT, LISA, and OSTrack

All required hyperparameters, library versions (Python, PyTorch, Faiss), and dependencies (SAM and PaddleOCR via Docker) are specified, ensuring direct replication of reported results.

RICE represents an overview of billion-scale region clustering with an efficient transformer-based regional embedding framework, advancing both vision and vision-language tasks through unified object and OCR recognition losses, while remaining scalable and computationally tractable at the largest data and compute regimes to date (Xie et al., 26 Jul 2025).