MVCap Dataset: Multi-view Captioning for VLP

• MVCap is a large-scale multi-view image–text dataset featuring 4.6M pairs, 94,600 objects, and high spherical viewpoint diversity for VLP.
• It integrates synthetic and real-world images with automated captioning aligned to 1,600 semantic categories to ensure robust viewpoint invariance.
• The dataset supports advanced techniques like Omniview-Tuning for cross-view alignment, making it pivotal for enhancing vision-language models.

The Multi-View Caption (MVCap) dataset is a large-scale resource designed to address the challenge of training and evaluating Vision-Language Pre-training (VLP) models under significant 3D viewpoint variation. Developed to support advances in viewpoint-invariant representation learning, MVCap comprises over 4.6 million image–text pairs representing 94,600 distinct objects and covering more than 1,600 semantic categories. Notably, its construction emphasizes high spherical diversity of viewpoints, systematic object coverage, and automated caption generation consistent with strict category guidance. MVCap is central to methodologies such as Omniview-Tuning for robust cross-view alignment in VLPs (Ruan et al., 2024).

1. Dataset Scope and Composition

MVCap is composed of 4,600,000 multi-view image–text pairs, representing a curated taxonomy of 1,600 object categories and 94,600 unique objects. The dataset features both synthetic 3D objects and real-world video-frame objects. For synthetic objects, each instance is represented by 100 randomly sampled viewpoints from the upper hemisphere, rendered in Blender; real-world instances, derived from MVImgNet, are represented with approximately 30 distinct viewpoints per object. The dataset draws 3D assets from sources including Objaverse, IM3D, and MVImgNet, with 24,495 virtual objects filtered via semantic-embedding similarity (OpenShape) to ensure broad category coverage. The spherical diversity of viewpoints is classified as “high,” corresponding to a maximum rating (★★★).

Property	Value	Notes
Image–text pairs	4.6 million	Combination of synthetic and real pairs
Unique objects	94,600
Object categories	1,600	Semantic taxonomy
Views per object	100/30	Synthetic/real

2. Data Acquisition and Automated Captioning

Collection of synthetic multi-view images proceeds by downloading 3D meshes (from Objaverse and IM3D), filtering for semantic clarity, and rendering 100 viewpoint images per object in Blender, with azimuth and elevation angles sampled uniformly over the upper hemisphere. Real views are extracted from video frames in MVImgNet, requiring at least 30 camera angles per object. Captions are generated automatically using a Vision-Language Large Model (VLLM), specifically InstructBLIP-flant5xl, leveraging a category-guided prompting protocol:

$T_{ij} = \mathcal{G}[I_{ij}, \text{Prompt}(c_i)]$

$$\text{Prompt}(c_i) = \text{``Write a short description for the image, noting that the main instance of the image is a } \langle c_i\rangle\text{.''}$$

This process enforces caption consistency with the ground-truth category. There is no human annotation; quality assurance is achieved through prompt design and embedding-based filtering applied to the object set.

3. Structural Organization and Metadata

Files are presumably arranged by object ID ($i$) and viewpoint index ($j$), with paired image files ($I_{ij}$.jpg) and caption text files ($T_{ij}$.txt), though the precise file/folder schema is not specified. Each datum is associated with at least the following metadata: object ID, viewpoint index, and caption. Rendering angles for synthetic views are logged but are not included in the described published schema. No train/val/test splits are introduced; the entire MVCap dataset is used for model fine-tuning, frequently in conjunction with the ImageNet-1K training set.

4. Statistical Properties and Training-Related Metrics

Key statistical characteristics include the number of objects, categories, and overall image–caption pairs, as summarized above. The dataset is notable for its high spherical viewpoint coverage and diversity. During model training, MVCap is used to enable learning of viewpoint-invariant representations, with key losses defined as follows:

• Multi-View Caption Generation (Eq. 1):

$T_{ij} = \mathcal{G}[I_{ij}, \text{Prompt}(c_i)]$

• Viewpoint Consistency Loss ($L_{VC}$, Eq. 7):

$$L_{VC} = \sum_{i=1}^N \sum_{j \neq j'} d(z^I_{ij}, z^I_{ij'})$$

where $d(\cdot, \cdot)$ denotes cosine distance between image embeddings from distinct viewpoints.

• Full Fine-Tuning Objective:

$$\text{Prompt}(c_i) = \text{``Write a short description for the image, noting that the main instance of the image is a } \langle c_i\rangle\text{.''}$$0

Here, $$\text{Prompt}(c_i) = \text{``Write a short description for the image, noting that the main instance of the image is a } \langle c_i\rangle\text{.''}$$1 is the standard image–text contrastive loss and $$\text{Prompt}(c_i) = \text{``Write a short description for the image, noting that the main instance of the image is a } \langle c_i\rangle\text{.''}$$2 weights the viewpoint consistency contribution. These objectives are relevant to downstream training using MVCap, not properties of the dataset per se.

5. Data Integrity, Category Taxonomy, and Quality Control

Category coverage is enforced by semantic-embedding similarity using OpenShape, ensuring more than 1,600 distinct semantic categories. For synthetic data, 3D assets are filtered for semantic clarity before rendering. Caption alignment with object categories is controlled via a prompt injected with ground-truth category. No human oversight occurs; instead, automated techniques—prompt engineering and embedding-based filtering—are the only quality assurance mechanisms. This approach eliminates annotator bias but may introduce systematic biases from the underlying VLLM and semantic filters.

6. Access, Release Status, and Licensing

As of publication (Ruan et al., 2024), MVCap is not accompanied by a public download URL or API. The authors state that the dataset will be released for research use either concurrent with, or following, the paper's publication. Specific licensing details are not provided in the source material. A plausible implication is that release may be staggered or gated by additional approval processes.

7. Applications and Research Implications

MVCap directly enables research on viewpoint-invariant vision-language representation learning by providing dense, category-aligned multi-view captioned images with high spherical viewpoint diversity. Its use in the Omniview-Tuning framework establishes new standards for VLP robustness to 3D viewpoint shift, notably by allowing the definition and optimization of cross-viewpoint alignment objectives and parameter-efficient model fine-tuning (Ruan et al., 2024). The design of MVCap suggests a paradigm for synthetic–real hybrid multi-view data construction, automated VLLM-based captioning at scale, and quality control absent manual annotation. This positions MVCap as a pivotal resource for future investigations requiring comprehensive, taxonomically structured multi-view imagery paired with consistent language supervision.