Fastdup: Deduplication and Quality Analysis Tool
- Fastdup is a dataset quality analysis tool that detects duplicates, outliers, and defects in image pipelines using deep feature extraction and cosine similarity.
- It employs a pre-trained ONNX model to extract embeddings and computes brightness and blur scores, enabling automatic thresholding in conjunction with pixel-level methods.
- Fastdup's approach is analogous to deduplication strategies in text and telemetry, demonstrating its relevance in fast, large-scale data quality workflows.
Searching arXiv for papers on Fastdup and closely related usage in data quality analysis. Fastdup is a tool used for dataset quality analysis, particularly in image pipelines, where it serves as a mechanism for detecting invalid images, duplicate image pairs, outliers, dark, bright, and blurry images, and image clusters. In the image-domain workflow examined in recent data-centric research, Fastdup is positioned as a latent-space complement to pixel-level tools: it uses a pre-trained ONNX feature extractor, computes cosine similarity between feature vectors, identifies near-duplicate pairs with similarity greater than 0.96, and groups them through a connected components algorithm on the similarity graph (Chen et al., 29 Sep 2025). The same work treats Fastdup not as a complete data-quality solution in isolation, but as a building block within a broader quality-assessment pipeline that combines semantic redundancy detection, simple scalar quality scores, and automatic threshold selection (Chen et al., 29 Sep 2025). A distinct but related usage of the term also appears in large-scale text deduplication, where FED is described as a “fastdup for LLM-scale corpora,” emphasizing that the underlying problem is rapid exact and near-duplicate detection at very large scale (Son et al., 2 Jan 2025).
1. Definition and Scope
In the image setting, Fastdup is used for four overlapping functions. First, it detects duplicate and near-duplicate images by operating in latent space rather than relying only on pixel-level hashes. Second, it ranks images by brightness and darkness using mean pixel value derived from RGB channel means. Third, it measures blur through Laplacian variance. Fourth, it exposes feature vectors that can be used for clustering and outlier detection, although reported results indicate that clustering directly over Fastdup features was unsatisfactory on the CIFAKE dataset studied in the cited work (Chen et al., 29 Sep 2025).
This operational profile places Fastdup within a broader class of deduplication systems that emphasize scalable similarity search over large collections. The image-quality paper explicitly contrasts Fastdup’s semantic or latent-level similarity with CleanVision’s MD5- and pHash-based pixel-level mechanisms, recommending the combination of both approaches rather than substituting one for the other (Chen et al., 29 Sep 2025). This suggests a useful distinction between two forms of redundancy: pixel-level redundancy, where images are nearly identical in visual encoding, and semantic redundancy, where images are similar in embedding space despite lower pixel-wise similarity.
A broader interpretation of “fastdup” also appears in other domains. In large-scale corpus curation for LLMs, FED is described as dataset deduplication at multi-trillion-token scale, engineered around GPU-optimized MinHash LSH and a custom rolling hash design (Son et al., 2 Jan 2025). In software telemetry, stack trace deduplication is framed as “Fastdup” for crash and performance reports, using a contrastive embedding model, approximate nearest neighbor search, and a reranker (Shibaev et al., 2024). These usages do not define the Fastdup image library itself, but they show that the term has become associated with the general task of fast, large-scale deduplication across modalities.
2. Core Technical Mechanisms in Image Pipelines
Fastdup’s near-duplicate mechanism, as described in the image-quality study, begins with a pre-trained ONNX model that extracts a feature vector for each image. A nearest-neighbor search is then run so that, for each image, its two most similar neighbors are identified in embedding space. Similarity is computed using cosine similarity, normalized to the interval , and pairs with similarity greater than 0.96 are considered near-duplicates (Chen et al., 29 Sep 2025).
Grouping proceeds by constructing a graph in which nodes are images and edges connect near-duplicate pairs. Duplicate groups are then obtained with a connected components algorithm. For deduplication, one representative is retained per connected component (Chen et al., 29 Sep 2025). The paper does not provide a more detailed public description of the underlying ONNX model, and explicitly notes that the feature extractor is proprietary and that details are not public (Chen et al., 29 Sep 2025).
For scalar quality analysis, Fastdup provides mean brightness values and blur scores. The mean brightness is verified in the paper as
where , , and are per-channel means. Dark images are ranked from lowest mean to highest, and bright images from highest mean to lowest (Chen et al., 29 Sep 2025). Blur is measured by Laplacian variance on the grayscale image: lower variance indicates greater blur, and images are ranked accordingly (Chen et al., 29 Sep 2025).
The same study emphasizes that Fastdup operates in latent space and can capture semantic similarity that pixel-level hashing such as pHash may miss (Chen et al., 29 Sep 2025). A plausible implication is that Fastdup is better suited to cases where similarity is preserved at the object or scene level despite perturbations that alter low-level pixel structure.
3. Role in Integrated Data-Quality Workflows
The image-quality pipeline described in the literature integrates Fastdup with CleanVision. Duplicate and near-duplicate detection are divided between the two systems: CleanVision uses MD5 for exact duplicates and pHash for near-duplicates, while Fastdup uses deep features plus cosine similarity for semantic or latent-level similarity (Chen et al., 29 Sep 2025). The paper recommends combining them so that pixel-level redundancy is handled by CleanVision and semantic-level redundancy by Fastdup (Chen et al., 29 Sep 2025).
For non-duplication defects such as brightness, darkness, blur, and low-information images, the workflow can use scores from either Fastdup or CleanVision. These scalar scores are then passed to automatic thresholding algorithms rather than using manually selected cutoffs (Chen et al., 29 Sep 2025). The cited work identifies hard-coded thresholds as a weakness because thresholds that perform adequately on one dataset may not generalize to another (Chen et al., 29 Sep 2025).
The pipeline’s main methodological extension is therefore not a modification of Fastdup’s internals, but an additional threshold-selection layer placed on top of its outputs. Histogram-based methods including Otsu’s method, Kittler–Illingworth Minimum Error Thresholding, Li’s Minimum Cross-Entropy Thresholding, Maximum Entropy Thresholding, Generalized Histogram Thresholding, Modified Valley Emphasis, and a gamma mixture model are applied to distributions of scalar quality scores to estimate thresholds directly from data (Chen et al., 29 Sep 2025). This makes the workflow model-agnostic in the sense stated in the paper: it depends only on the distribution of scalar quality scores, not on the underlying model or the specific dataset (Chen et al., 29 Sep 2025).
Within that combined workflow, Fastdup specifically contributes semantic near-duplicate detection, feature generation, and simple quality scores. The paper reports that Fastdup’s default clustering and thresholding are not always satisfactory on the studied dataset, but still treats Fastdup as a useful component when embedded in a broader process that includes automatic thresholding and pixel-level deduplication (Chen et al., 29 Sep 2025).
4. Quantitative Behavior and Reported Performance
The cited image-quality study evaluates low-quality image detection as a binary classification problem, with the positive class defined as images that truly contain a target defect and the negative class defined as normal images. Precision, recall, and F1 are used as evaluation metrics (Chen et al., 29 Sep 2025). For near-duplicate detection, the same binary-classification framing is used (Chen et al., 29 Sep 2025).
For near-duplicate detection on a test set in which 12% of training images were replaced with augmented versions, the reported F1 scores are 0.4579 for CleanVision, 0.6466 for Fastdup, and 0.7928 for the proposed pHash plus hierarchical clustering method (Chen et al., 29 Sep 2025). This establishes two points. First, Fastdup outperforms the default CleanVision near-duplicate configuration in this experimental setting. Second, Fastdup alone is not the best-performing deduplication strategy on that dataset, since the improved pHash-clustering method achieves a higher F1 (Chen et al., 29 Sep 2025).
For low-quality image detection more broadly, the paper reports that automatic thresholding improves the F1 score from 0.6794 to 0.9468 under single perturbations and from 0.7447 to 0.8557 under dual perturbations (Chen et al., 29 Sep 2025). These aggregate results are presented for the integrated workflow rather than Fastdup alone, but they directly inform practical Fastdup usage because the thresholding layer is described as applicable to Fastdup scores as well as CleanVision scores (Chen et al., 29 Sep 2025).
The paper also connects data-quality detection to downstream model behavior. On CIFAKE, convolutional neural networks are reported as robust to moderate brightness changes, grayscale conversion, and moderate resolution changes, but fragile to strong blur, severe downscaling, and extreme low-information images (Chen et al., 29 Sep 2025). This suggests that Fastdup-derived blur and brightness rankings are not equally consequential for training outcomes: blur-related decisions may matter more than moderate brightness variation in the studied setting.
5. Near-Duplicate Detection, Similarity Semantics, and Common Misconceptions
A common misconception is to treat all duplicate detection as a single problem. The cited work instead distinguishes exact duplicates, pixel-level near-duplicates, and semantic near-duplicates (Chen et al., 29 Sep 2025). Exact duplicates are handled by hashing such as MD5. Pixel-level near-duplicates can be handled by pHash and Hamming-distance-based clustering. Semantic near-duplicates are the domain in which Fastdup is explicitly positioned, through embedding extraction and cosine similarity (Chen et al., 29 Sep 2025).
Another misconception is that pixel-level methods and embedding-based methods are interchangeable. The paper argues the opposite by recommending both: CleanVision pHash plus clustering for pixel-level near-duplicates, and Fastdup latent features plus cosine similarity for semantic near-duplicates (Chen et al., 29 Sep 2025). This suggests that different notions of image similarity are operationally complementary rather than redundant.
A further misconception is that Fastdup’s default thresholds should be treated as dataset-independent decision rules. The paper explicitly criticizes hard-coded thresholds and proposes automatic thresholding because score distributions vary across datasets (Chen et al., 29 Sep 2025). In the studied workflow, Fastdup’s quality scores are therefore most effective when interpreted through a dataset-specific threshold inferred from the histogram of observed values (Chen et al., 29 Sep 2025).
The paper also reports that clustering directly in Fastdup’s embedding space, using HDBSCAN, hierarchical clustering, PCA, and normalization, produced unsatisfactory performance on the CIFAKE dataset (Chen et al., 29 Sep 2025). This does not imply that Fastdup’s embeddings are ineffective for similarity search; rather, in the reported experiments, they were less effective than the improved pHash-based clustering method for pixel-level near-duplicate grouping on that dataset (Chen et al., 29 Sep 2025).
6. Relation to Fast Deduplication in Other Domains
The term “fastdup” is also used descriptively beyond the image library itself. In LLM data curation, FED is described as a “fastdup for LLM-scale corpora,” meaning a system for rapidly detecting and removing exact and near-duplicate documents at extremely large scale (Son et al., 2 Jan 2025). FED is based on MinHash plus Locality-Sensitive Hashing, but differs from CPU and earlier GPU baselines through a GPU-optimized rolling, non-cryptographic hash, custom CUDA kernels for all-pairs bucket comparison, double buffering, and out-of-core processing (Son et al., 2 Jan 2025). It reportedly deduplicates 1.2 trillion tokens in approximately 5.1 hours on 16 GPUs while maintaining Jaccard agreement of at least 0.95 with full MinHash on evaluated subsets (Son et al., 2 Jan 2025).
In software engineering, stack trace deduplication is similarly framed as fast, large-scale grouping of reports that correspond to the same underlying issue. The reported system uses byte-pair encoding for stack frames, a biLSTM embedding model trained with InfoNCE, FAISS approximate nearest neighbor retrieval, and a reranker that emphasizes repeated frames between traces (Shibaev et al., 2024). The resulting pipeline achieves state-of-the-art attach accuracy across several datasets while operating at 8.7 ms per report for embedding-only inference and 144.5 ms per report with reranking on Ubuntu (Shibaev et al., 2024).
These cross-domain examples are not implementations of the Fastdup image tool, but they illuminate an underlying pattern: fast deduplication generally combines a compressed representation, an efficient similarity search mechanism, and a grouping rule such as connected components or union-based clustering. In image pipelines, Fastdup instantiates that pattern with embeddings and cosine similarity; in text corpora, the analogous mechanism is MinHash LSH; in stack traces, it is contrastive embeddings plus ANN retrieval (Chen et al., 29 Sep 2025, Son et al., 2 Jan 2025, Shibaev et al., 2024).
7. Limitations, Practical Use, and Research Directions
The main published limitations for Fastdup usage arise from the image-quality study rather than from a dedicated Fastdup systems paper. The experiments are conducted only on the FAKE subset of CIFAKE, which is synthetic, generated by Stable Diffusion, and relatively clean aside from discovered grayscale and duplicate images (Chen et al., 29 Sep 2025). The paper therefore states that results may differ on real-world, noisy datasets with watermarks, more complex noise, and label errors (Chen et al., 29 Sep 2025).
A second limitation is that the paper examines at most two concurrent perturbations, whereas real-world datasets may contain multiple interacting issues such as blur, low resolution, and compression artifacts (Chen et al., 29 Sep 2025). A third limitation concerns thresholding hyperparameters: Li’s and Otsu’s methods are used with default parameters from scikit-image, while other methods are re-implemented from original papers; no exhaustive ablation is reported for bin counts, initialization, or numerical stability, and Li’s method is noted as sensitive to initialization (Chen et al., 29 Sep 2025).
For practical use, the cited workflow recommends running Fastdup to extract features and similarity pairs, and to generate brightness and blur statistics; running CleanVision to obtain pixel-level anomaly and hashing signals; applying automatic thresholding to the scalar scores; and using both pHash-based clustering and Fastdup cosine similarity to identify distinct forms of near-duplicate structure (Chen et al., 29 Sep 2025). The retention policy within duplicate groups is not formalized in the paper, though it mentions retaining one representative per connected component in Fastdup evaluation (Chen et al., 29 Sep 2025).
Future work directions identified in the literature include applying the integrated pipeline to real-world datasets, exploring a broader range of degradations, conducting ablations on thresholding hyperparameters and Li initialization, and investigating alternative feature representations and evaluation metrics that could improve robustness and generality, including potential improvements to Fastdup’s feature usage (Chen et al., 29 Sep 2025). This suggests that Fastdup’s current research role is best understood as a useful semantic-similarity subsystem within a data-centric quality pipeline rather than as a universally sufficient standalone solution.
