---
title: 'NFT1000: A Benchmark for NFT Visual-Text Retrieval'
url: https://www.emergentmind.com/topics/nft1000-dataset
type: topic
---

# NFT1000: A Benchmark for NFT Visual-Text Retrieval

NFT1000, introduced as the “NFT Top1000 Visual-Text Dataset,” is a large-scale cross-modal benchmark for Non-Fungible Token retrieval centered on ERC-721 profile-picture collections on Ethereum. It contains 7.56 million image–text pairs drawn from the top 1000 PFP NFT collections by sales volume, with one consolidated caption per image derived from trait metadata, and is designed for retrieval under extreme fine-grained similarity, where images often differ only in small spatial regions and captions differ only in small combinations of trait words [2402.16872].

## 1. Scope, domain, and motivation

NFT1000 is motivated by a Web 3.0 and Metaverse retrieval problem: locating a desired NFT within marketplaces containing tens or hundreds of thousands of visually and semantically similar items. The paper notes that, by late 2023, more than 1.4 billion NFT tokens had been minted across blockchains, while the abstract states that by the end of March 2024 more than 1.7 billion NFTs had been minted across various blockchain platforms. Within that setting, PFP NFTs are singled out because they represent more than 60% of NFT market cap and because each token typically exposes a JSON metadata file containing trait attributes that can be aligned naturally with visual content [2402.16872].

The benchmark is restricted to Ethereum and specifically to ERC-721 collections. Ethereum is described as the “birthplace of NFTs” and the most active chain for NFT crypto-art, while ERC-721 is treated as the canonical NFT standard. The focus on PFP collections is methodologically consequential: these collections are usually generated by fixed-layer permutation and combination, so tokens within a project share the same base character and style and differ only in a small set of layers such as background color, accessories, clothing, or facial attributes.

From a retrieval standpoint, the dataset formalizes a regime of simultaneous high regional similarity and high semantic similarity. The regional aspect arises because many tokens share almost identical pixel regions, differing only in a small patch such as a hat or glasses. The semantic aspect arises because descriptions within a collection repeatedly recombine the same trait names. NFT1000 is therefore positioned as a benchmark for large-scale, high-similarity cross-modal retrieval bridging vision–language modeling and blockchain-based NFT applications.

## 2. Construction pipeline and representation

The collection process begins from 1250 Ethereum PFP projects ranked by sales volume; after filtering, 1000 collections remain. “Most famous” is operationalized explicitly as top collections by sales volume on Ethereum. Data acquisition combines on-chain smart-contract information with Web3 infrastructure components including Alchemy, NFTScan, and IPFS. For each token, the workflow enumerates tokens in a collection, queries the metadata URI, downloads the JSON metadata file, resolves the image or media link from that JSON, and downloads the media resource [2402.16872].

Filtering removes three classes of collections: projects with completely duplicated media data, collections with fewer than 500 NFTs, and collections lacking usable metadata, including absent JSON or JSON without substantive semantic information. The retained corpus consists of 1000 collections with approximately 6600 NFTs on average, yielding 7.56 million image–text pairs.

Each ERC-721 NFT is associated with a JSON metadata file containing key–value trait pairs such as `"trait_type": "Hat", "value": "Commie Hat"` and `"trait_type": "Mouth", "value": "Bored bubblegum"`. NFT1000 leverages this inherent image–text pairing but does not keep the raw trait list as the final textual field. Instead, it consolidates traits into one primary caption per image.

Two caption-generation strategies are described. The first uses ChatGPT and LLAMA-13B to convert trait lists into natural sentences through prompt engineering. The second uses sentence templates to concatenate traits deterministically into a single sentence. For a sample of 10,000 images, the authors generated 30,000 LLM-based descriptions and 10,000 template-based captions, then evaluated both with CLIP-ViT-L in zero-shot retrieval. LLM captions gave slightly better retrieval performance, but because the gain was small and generation over 7.56 million items would be expensive and time-consuming, the full dataset adopts the template-based method.

Media are standardized aggressively. Static formats such as `.jpg`, `.png`, `.svg`, and `.webp` are converted to PNG; for dynamic media such as `.gif` and `.mp4`, one representative frame is randomly selected and converted to PNG. The standard width is 512 pixels, with height scaled proportionally to preserve aspect ratio. This standardization reduces storage from about 14 TB to about 1.75 TB.

## 3. Internal structure, statistics, and dataset-specific difficulty

NFT1000 combines strong within-project homogeneity with substantial cross-project diversity. Within a collection, images are generated by fixed-layer permutation and combination over attributes such as background, body, clothes, hat, and glasses. After cover removal there are no exact duplicates of final images, but many tokens remain extremely close neighbors in both pixel space and metadata space. The paper illustrates the strength of spatial alignment by averaging multiple images per project: the resulting mean image still shows clear contours, indicating that tokens are strongly aligned in space [2402.16872].

The textual side is similarly structured yet nontrivial. Trait lists are repetitive within a collection, but the corpus still contains 595,504 unique descriptive phrases. Moreover, artists often use abstract, subjective, or idiosyncratic naming conventions that do not align with ordinary natural-language usage. This produces a domain-specific semantic regime in which captions can be syntactically regular while remaining lexically opaque to models pretrained on generic web corpora.

Across collections, the benchmark spans 356 distinct themes, including 3D rendered avatars, 2D flat illustrations, pixel art, NPC characters, real photographs, and other artistic styles. The resulting combination is unusual: each individual collection is highly homogeneous internally, yet the overall dataset remains visually diverse at the project level.

These properties make NFT1000 difficult in a way not captured by standard image–text corpora. Intra-collection differences may reduce to a single trait; captions are highly structured and repetitive; and retrieval must scale to nearly one million items in the test split alone. The paper interprets the resulting weakness of zero-shot performance as evidence that NFT-style imagery and captions are underrepresented in existing pretraining distributions.

## 4. Benchmark protocol and evaluation design

NFT1000 is designed primarily for cross-modal retrieval, especially text-to-image retrieval, which aligns with the use case of searching for NFTs via textual descriptions. The benchmark also supports image-to-text retrieval. In the standard setup, each query caption corresponds to exactly one ground-truth image, and the gallery contains one caption per image [2402.16872].

If an image encoder \(f_I\) and a text encoder \(f_T\) produce normalized embeddings, similarity is defined as

$$
s(x,y)=\langle f_I(x), f_T(y)\rangle,
$$

which is equivalent to cosine similarity when both embeddings are \(L_2\)-normalized. For each query text, similarities to all candidate images are computed, the gallery is ranked by descending score, and performance is summarized with Top-\(K\) accuracy.

For Top-1 accuracy, the paper gives

$$
\text{Top-1 Acc}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\text{rank}(g_i)=1\},
$$

and for Top-\(K\),

$$
\text{Top-K Acc}=\frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\text{rank}(g_i)\leq K\},
$$

where \(g_i\) is the ground-truth image for query \(i\). The reported evaluation metrics are Top-1, Top-5, and Top-10 retrieval accuracy; the manuscript excerpt does not present mAP or Recall@\(\!K\) formulas.

To avoid data leakage arising from shared layers and repeated traits inside a collection, data splitting is performed at the project level rather than the item level.

| Split | Projects | Image–text pairs |
|---|---:|---:|
| Training | 800 | 6,178,429 |
| Validation | 50 | 383,916 |
| Test | 150 | 1,000,838 |

This project-level partition is central to the benchmark’s validity because token-level random splitting would permit models to see near-identical variants of a collection during training and evaluation.

## 5. Comprehensive Variance Index

A central analytical contribution of the benchmark is the Comprehensive Variance Index (CVI), a scalar measure of image–text retrieval difficulty derived from similarity distributions in a batch. The core intuition is that low variance in image–image, text–text, and text–image similarities indicates uniformity in feature space: images look alike, captions are semantically similar, and correct matches are correspondingly hard to separate from incorrect ones. Lower variance therefore corresponds to higher similarity and higher retrieval difficulty [2402.16872].

Let image features be \( \mathbf{I} \in \mathbb{R}^{N \times M} \) and text features be \( \mathbf{T} \in \mathbb{R}^{N \times M} \). The similarity matrices are

$$
\mathbf{S}_{II}=\mathbf{I}\cdot \mathbf{I}^{\top},
$$

$$
\mathbf{S}_{TT}=\mathbf{T}\cdot \mathbf{T}^{\top},
$$

$$
\mathbf{S}_{TI}=\mathbf{T}\cdot \mathbf{I}^{\top}.
$$

If \( \mathbf{S}_{II_i}, \mathbf{S}_{TT_i}, \mathbf{S}_{TI_i} \) denote the \(i\)-th rows of these matrices, the CVI is defined as

$$
CVI=\frac{1}{2N}\left(
\alpha \sum_{i=1}^{N}\mathrm{var}(\mathbf{S}_{II_i})
+
(1-\alpha)\sum_{i=1}^{N}\mathrm{var}(\mathbf{S}_{TT_i})
+
\sum_{i=1}^{N}\mathrm{var}(\mathbf{S}_{TI_i})
\right),
$$

where \( \alpha \in [0,1] \) controls the relative weighting between visual and textual similarity. Larger \( \alpha \) emphasizes image–image similarity; smaller \( \alpha \) emphasizes text–text similarity.

The paper validates CVI by computing, for batches from the NFT1000 test set under CLIP-ViT-L zero-shot retrieval, both CVI and Top-\(K\) accuracies from Top-1 to Top-20, then applying \(L_1\) normalization and comparing the resulting distributions via Jensen–Shannon divergence (JSD):

$$
JSD(P\parallel Q)=\frac{1}{2}D(P\parallel M)+\frac{1}{2}D(Q\parallel M),
$$

with \( M=\frac{1}{2}(P+Q) \). The reported finding is that JSD is minimized around \( \alpha \approx 0.7 \), implying that an image-weighted CVI best matches empirical retrieval difficulty. The manuscript interprets this as evidence that visual information is more informative than textual information for difficulty estimation in this dataset. It further reports that, after normalization, the CVI curve and Top-\(K\) accuracy curve have similar shapes: lower CVI corresponds to more difficult subsets, while higher CVI corresponds to easier subsets.

## 6. Baselines, discrepancies in the methodological record, access, and outlook

The benchmark evaluates four pretrained vision–language models: CLIP-ViT-B/32, CLIP-ViT-L/14, META-CLIP-ViT-L/14 (`metaclip_400m`), and EVA02-CLIP-ViT-L/14 (`merged2b_s4b_b131k`). The reported results compare zero-shot inference on the NFT1000 test split with fine-tuning on NFT1000 training data [2402.16872].

| Model | Zero-shot Top-1 / Top-5 / Top-10 | Fine-tuned Top-1 / Top-5 / Top-10 |
|---|---|---|
| CLIP-ViT-B/32 | 0.0061 / 0.0204 / 0.0330 | 10.6310 / 20.3156 / 25.1922 |
| META-CLIP-ViT-L/14 | 0.0029 / 0.0091 / 0.0173 | 13.0598 / 23.6794 / 28.8138 |
| EVA02-CLIP-ViT-L/14 | 3.2305 / 7.9090 / 10.9149 | 11.7849 / 22.2515 / 27.4309 |
| CLIP-ViT-L/14 | 0.0593 / 0.2453 / 0.4232 | 15.3625 / 27.5525 / 33.2599 |

The paper characterizes zero-shot performance as extremely low for most models and fine-tuning performance as substantially improved, which it interprets as evidence that NFT imagery and trait-based descriptions are underrepresented in broad-domain pretraining corpora. It also reports a data-scale study using three training regimes: NFT100_mini with 40,000 pairs, NFT1000_mini with about 794,698 pairs, and the full NFT1000 training set with 6,178,429 pairs. Increasing scale improves retrieval, but gains saturate, consistent with the dataset’s high redundancy.

A notable textual discrepancy concerns dynamic masking. The abstract states that, using CLIP-series models as a foundation, the paper proposes a dynamic masking fine-tuning scheme that yields a 7.4% improvement in Top-1 accuracy while using 13% of the training data, comparing 0.79 million with 6.1 million pairs. However, the manuscript excerpt summarized in the supporting material does not include an explicit section describing that method, its equations, or an algorithmic specification. The methodological record available in the excerpt therefore consists of standard CLIP-style fine-tuning, model comparisons, data-scale analysis, and CVI-based difficulty assessment. A plausible implication is that any treatment of dynamic masking should be read directly from the full paper or associated repository rather than inferred from the excerpt alone.

The dataset is announced as open source, and the abstract points to `https://github.com/ShuxunoO/NFT-Net.git`. The paper states that release is intended to include the dataset and code, potentially including scripts for download and reconstruction, preprocessing, and baseline CLIP training. At the same time, the benchmark raises nontrivial copyright and licensing issues. NFT images and metadata are often copyrighted by their original creators, and token ownership or public metadata availability does not automatically imply permission to redistribute images or use them commercially. The paper therefore frames NFT1000 primarily as a research resource and notes practical caution around dataset licensing and collection-specific IP terms.

The benchmark’s stated limitations are equally important. Coverage is confined to Ethereum ERC-721 PFP collections and excludes other chains such as Solana, Polygon, BNB Chain, and Klaytn. The top-1000-by-sales-volume criterion introduces market bias toward highly traded, popular collections. Most traits are in English or English-coded forms, and abstract or subcultural naming conventions may be poorly represented in generic language models. Future work proposed in the paper includes reducing redundancy, expanding to additional blockchains, scaling toward hundreds of millions of NFTs, exploring generative modeling, and developing richer difficulty metrics beyond the present CVI formulation.

Source: https://www.emergentmind.com/topics/nft1000-dataset