---
title: 'OpenLVD200M: 200M-Image Corpus for Vision Distillation'
url: https://www.emergentmind.com/topics/openlvd200m
type: topic
---

# OpenLVD200M: 200M-Image Corpus for Vision Distillation

OpenLVD200M is a 200 million-image training corpus constructed explicitly to optimize multi-teacher distillation workflows for vision foundation models. Generated through an aggressive hierarchical clustering and balanced sampling strategy from large-scale public datasets, OpenLVD200M addresses the inefficiencies of naïvely sampled web-scale data by ensuring uniform concept coverage, thereby enabling lower-cost, more data-efficient representation learning. The corpus is entirely unsupervised, containing only RGB images without provided captions or labels, and is designed for integration into distillation and self-supervised learning pipelines, notably underpinning AMoE (Agglomerative Mixture-of-Experts Vision Foundation Models) [2512.20157].

## 1. Definition and Rationale

OpenLVD200M is defined as a 200 million-image dataset curated for efficient multi-teacher distillation, where a single student encoder learns joint representations from multiple frozen teacher models (such as SigLIP2 for image-text alignment and DINOv3 for dense visual geometry). Standard random sampling of large-scale web data is empirically insufficient: it leads to overrepresentation of head concepts and cannot affordably expose students to the tail of rare or fine-grained concepts. OpenLVD200M solves this by leveraging hierarchical k-means clustering and balanced sampling, ensuring uniform coverage across the semantic space and improving sample efficiency, convergence speed, and downstream accuracy relative to random subsets [2512.20157].

## 2. Source Datasets and Corpus Composition

OpenLVD200M is drawn from a 2.3 billion-image superset comprising two major sources:
- LAION-5B, contributing approximately 2 billion CC-BY licensed images.
- DFN (Data Filtering Networks), adding around 300 million images from a CC0-like custom license.

The final OpenLVD200M set contains exactly 200,000,000 RGB images in mixed aspect ratios. It is strictly an unsupervised corpus with no accompanying textual labels or captions. Images span multiple resolutions, reflecting the original sources.

| Source         | Images Contributed | License                       |
|----------------|-------------------|-------------------------------|
| LAION-5B       | ~2,000,000,000    | CC-BY 4.0                     |
| DFN            | ~300,000,000      | Custom (CC0-like)             |
| OpenLVD200M    | 200,000,000       | CC-BY 4.0 (most permissive)   |

## 3. Hierarchical Clustering and Balanced Sampling Pipeline

The construction of OpenLVD200M applies a four-level hierarchical k-means clustering pipeline to flatten the dataset’s long-tailed concept distribution:

- **Level 1:** $K_1 = 20,000,000$ centroids
- **Level 2:** $K_2 = 500,000$ centroids
- **Level 3:** $K_3 = 50,000$ centroids
- **Level 4:** $K_4 = 20,000$ centroids

**Pipeline steps:**
1. All 2.3B candidate images are first embedded using the DINOv3-ViT-B encoder; 1B are extracted for clustering.
2. K-means is computed at each level $\ell$ to find centroids $\{\mu_{\ell, j}\}_{j=1..K_\ell}$, minimizing $\sum_{i \in I} \min_{j \in [1..K_\ell]} \|x_i - \mu_{\ell,j}\|^2$.
3. The remaining 1.3B images are assigned to the nearest Level 1 centroids.
4. Balanced hierarchical sampling is employed: each level’s clusters are sampled uniformly, then images within each cluster are sampled uniformly, yielding a final corpus with near-uniform representation of all Level-4 concepts.

The final per-image sampling weight is:
$$
w_i \propto \prod_{\ell=1}^4 \frac{1}{|C_\ell(\text{cluster}(i))|}
$$
Images are drawn without replacement until 200M are selected. This approach systematically overcomes the long-tailed distribution challenge endemic to web-scraped datasets [2512.20157].

## 4. Preprocessing and Training Integration

OpenLVD200M’s design explicitly supports advanced batching and training strategies:
- **Token-Balanced Batching:** Each accelerator processes image sequences as packed tokens; sequences combine images of different resolutions within an overall token budget (e.g., $C_\text{max}=20,000$) using FlexAttention masks that prevent inter-image attention. This maintains uniform load across GPUs, prevents over-weighting high-resolution examples, and normalizes losses per image by patch count.
- **Two-Stage Distillation:** 
  - *Stage 1*: All images resized to a maximum of $256 \times 256$ pixels, retaining aspect ratio and relying on token batching for padding/packing.
  - *Stage 2*: Joint mixing of:
    - OpenLVD200M at $256 \times 256$
    - Natural-size images between $256 \times 256$ and $384 \times 384$
    - High-resolution images downsampled to both $256 \times 256$ and $512 \times 512$

Standard random cropping, horizontal flips, and color jitter are the only augmentations applied. No extra augmentations are required—most images are used in their native form.

| Image Resolution | Fraction of Corpus |
|------------------|-------------------|
| 256×256          | 60%               |
| 384×384          | 20%               |
| 512×512          | 15%               |
| >512             | 5% (Stage 2 only) |

## 5. Licensing, Distribution, and Usage Guidelines

OpenLVD200M inherits its licensing from the underlying datasets, resolving to CC-BY 4.0 to maximize permissivity. The corpus is distributed as a list of 200M URLs and corresponding SHA-256 hashes, supporting automated download and data integrity verification. No official validation split is provided; users are advised to carve out a 1M uniformly sampled hold-out set, stratified across clusters, for in-distribution evaluation.

Recommended best practice limits OpenLVD200M use to unsupervised distillation and self-supervised learning. Training supervised classifiers directly with this corpus requires compliance with the originating image licenses.

## 6. Empirical Impact and Benchmark Results

In the AMoE framework, OpenLVD200M enables highly sample-efficient training:
- AMoE trained on 200M OpenLVD images ($\sim$230B tokens) achieves:
  - State-of-the-art zero-shot image-text classification: macro-avg 84.13 (vs. 82.26 for RADIOv2.5, which used 1B random images and $\sim$1.1T tokens).
  - Competitive segmentation and linear probe transfer.
  - Superior Recall@1 for retrieval on MSCOCO5k and Flickr30k.

Ablation evidence:
- Substituting OpenLVD200M for a random 200M image subset reduces image-text classification by $\sim$4.15\% and kNN macro-avg by $\sim$2.42\%.
- For fine-grained long-tail benchmarks such as FGVC-Aircraft, OpenLVD200M delivers gains exceeding +18\% in top-1 accuracy, underscoring the effectiveness of hierarchical balancing for rare concepts [2512.20157].

## 7. Loading, Data Access, and Practicalities

OpenLVD200M’s manifest is partitioned into shards of 1M URLs each. Cluster assignments are stored in an accompanying file (e.g., "openlvd_clusters.json"). Sampling and batching logic, as used in experimental pipelines, is illustrated by the following procedural outline:

```python
import random, json, requests
clusters = json.load(open("openlvd_clusters.json"))
lvl4_index = defaultdict(list)
for img_id, (c1,c2,c3,c4) in clusters.items():
    lvl4_index[c4].append(img_id)
sampled = []
per_leaf = 200_000_000 // len(lvl4_index)
for leaf, img_ids in lvl4_index.items():
    sampled_ids = random.sample(img_ids, per_leaf)
    sampled.extend(sampled_ids)
with open("train_200M.txt", "w") as f:
    for img_id in sampled:
        f.write(img_id + "\n")
class OpenLVDLoader(torch.utils.data.Dataset):
    def __init__(self, manifest, transforms):
        self.ids = open(manifest).read().splitlines()
        self.transforms = transforms
    def __getitem__(self, idx):
        url = self.ids[idx]
        img = load_and_decode(url)     # implement caching & retry
        return self.transforms(img)
loader = DataLoader(OpenLVDLoader("train_200M.txt", tfs),
                    batch_size=None,  # handled by token packer
                    collate_fn=token_packer)
```
Collation uses the token-balanced packer from Dehghani et al. (2023) or FlexAttention code. Images are loaded, decoded, and transformed for training, with batching handled at the token level for optimal throughput.

OpenLVD200M thus serves as a drop-in replacement for random web-scraped datasets in large-scale vision foundation model training, offering superior coverage, efficiency, and empirical performance in multi-teacher distillation settings [2512.20157].

Source: https://www.emergentmind.com/topics/openlvd200m