EMMa: Extensible Multimodal, Multi-task Dataset
- EMMa is an Amazon-derived, object-centric dataset offering extensible multimodal and multi-task learning with rich attribute annotations.
- It features a hand-curated hierarchical taxonomy of 182 material types and enables combinatorial prediction tasks across images, text, and structured data.
- Synthetic label completion and Smart Labeling strategies boost prediction accuracy for diverse properties such as price, mass, ratings, and materials.
EMMM is not the formal name of an established benchmark in the relevant paper. In this arXiv context, it most plausibly functions as an informal shorthand for the combination of extensibility, multimodality, multi-task learning, and materials-centric annotation realized by EMMa, the dataset introduced in (Standley et al., 2023). The formal dataset name is EMMa, not EMMM: the title capitalizes Extensible, Multimodal, and Multi-task, while the final “a” comes from the dataset name rather than another M-word. EMMa is an Amazon-derived, object-centric corpus in which each listing may include images, text, mass, price, ratings, category information, materials, and other attributes, with the explicit design principle that each attribute can serve either as a model input or as a prediction target (Standley et al., 2023).
1. Nomenclature and conceptual framing
EMMa was introduced to address three linked problems. The first is the dataset-creation bottleneck in modern machine learning, especially when practitioners need labels that are expensive, ambiguous, or difficult to specify to crowd workers. The second is the lack of a very large, fine-grained, object-centric materials dataset. The third is the absence of a benchmark in which multiple aligned attributes can be recombined into many input-output task configurations across computer vision and NLP (Standley et al., 2023).
The work is explicitly object-centric rather than scene-centric. That choice matters because the target attributes are attached to product listings keyed to a single object instance, which makes cross-modal reasoning direct: listing text, listing images, category placement, price, mass, ratings, and materials are all aligned to the same product record. The authors therefore describe the space of supported tasks as combinatorial. Example tasks mentioned include predicting category from listing text, predicting mass and price from image, predicting ratings and price from text, and predicting materials from all other attributes (Standley et al., 2023).
This framing also explains why the informal label “EMMM” maps closely onto EMMa. The core idea is not merely multimodal supervision, but a reusable object record in which extensibility and multi-task recombination are first-class design goals.
2. Corpus construction and modality structure
EMMa is assembled from Amazon product listings and contains 2,883,698 instances after filtering. The merged raw pool reached about 18 million entries, drawn from three ASIN-keyed sources: roughly 2 million items from the Amazon Selling Partner API (ASPA), about 3 million listings from the raw image2mass dataset, and about 15 million listings from the UCSD Amazon product rating/review dataset. The final split is 2,806,806 train, 26,535 validation, 26,941 test, and 23,416 calibration (Standley et al., 2023).
The dataset is multimodal and attribute-rich. It contains 7,389,213 images total, averaging 2.56 images per listing, at original 500×500 resolution. Every listing has a title; 88% have bullet-point product features; 63% have a description. The text input is the concatenation of title, features, and description. The released embeddings include image representations obtained by summing ConvNeXT-Large features over listing images and text representations from all-mpnet-base-v2. Structured attributes include mass/weight for 77% of listings, price for 69%, materials for 30%, category for 70%, and at least one review/rating for 75%. Most listings also have some size information, and 85% have package weight. Product type is included with 1,296 unique seller-facing types. The inferred category hierarchy has 11,592 nodes and maximum depth 6 (Standley et al., 2023).
The corpus is heavily cleaned. Duplicates and near-duplicates were removed using both NLP and image embeddings; entries with no title or no image were removed; any entry lacking all of mass, materials, and category was dropped; mismatched image/text listings were detected and removed; nonphysical or unhelpful listings were filtered; and ASIN collisions caused by sellers overwriting listings over time were heuristically detected and discarded. Validation and test were chosen from objects with all attributes filled in, with more than one image and more than one review, and were deduplicated more aggressively against train; image2mass Amazon test items were excluded from training for fair comparison (Standley et al., 2023).
3. Hierarchical material annotation
A central contribution of EMMa is its materials annotation scheme. Raw Amazon materials are seller-entered strings, so the authors clean and map them into a hand-curated hierarchical taxonomy of 182 material types. The taxonomy is explicitly hierarchical, with examples such as Plastic → Thermoplastic → Acrylic, and branches including metal subtypes such as steel and aluminum, as well as wood, fabric, leather, diamond, marble, food, electronics, and liquid (Standley et al., 2023).
The taxonomy is described as designed “in a data-driven way” to capture what sellers actually write. It intentionally includes entries that may not be “pure materials” in a chemistry sense, including plush, microfiber, leather, and wood, because these imply useful parent classes and matter for downstream physical and economic properties. Each object may have multiple materials; on average, each object has 4.9 material labels. Because seller input is often coarse, labels may attach to internal nodes rather than leaves: an object may be known to be plastic without specifying acrylic versus polypropylene, or metal without specifying steel versus aluminum (Standley et al., 2023).
Material quality control is unusually explicit. 11% of the calibration set was manually labeled, focusing on difficult instances, and that calibration set is used for training and calibrating material probabilities. The Smart Labeling procedure was also applied to 19 material nodes to refine labels across the full dataset: nylon plastic, nylon fabric, cardboard, cardstock, diamond, liquid, electronics, food, gold, genuine leather, lace, marble, faux-fur, cherry, leather, fabric, wood, metal, and plastic. Coverage is substantial: 78 nodes have more than 10k instances, and 91% of nodes have more than 500 instances (Standley et al., 2023).
This hierarchical structure has direct modeling implications. Material prediction is not limited to leaf classes: models predict a probability distribution over all 182 material nodes, even when the original annotation is only at an internal node. A listing labeled only as metal can therefore induce probabilities over children such as steel and aluminum. An iterative consistency algorithm is then applied so hierarchical probabilities obey the constraint that a parent probability must lie between its highest child probability and the sum of its children’s probabilities; because an object may contain several children under a parent, the parent is not constrained to equal the sum of its children (Standley et al., 2023).
4. Synthetic completion and multitask learning
EMMa treats missing-label completion as a predictive problem over aligned object attributes. For each missing attribute except size—price, mass, materials, categories, ratings distribution—models are trained to predict that attribute from all the others. The architecture is a late-fusion system: a vision model, ConvNeXT-Base pretrained on ImageNet-22k, is fine-tuned to jointly predict all tasks from images, and a text model, all-mpnet-base-v2, is fine-tuned to do the same from listing text. Both produce 768-d embeddings, and task-specific Everything models take these standardized embeddings together with other available attributes. The authors describe the overall process as “a form of expectation-maximization”: generation-1 models are trained on original data, their predictions fill in missing labels, and generation-2 models are retrained on the completed dataset (Standley et al., 2023).
The paper defines task-specific metrics and losses. For price and mass, it uses minimum ratio error (MnRE),
where is the prediction and is the ground truth. For materials, the loss is the mean of binary cross-entropy across all 182 material types, with descendants excluded from the mean whenever a training label is an internal node, so the model is not penalized for being more specific than the annotation. For categories, the loss is a cross-entropy at each level of the taxonomy, combined by a weighted mean over levels; higher taxonomy levels are treated as more important, and only predictions under the correct parent branch are considered at deeper levels. For ratings, the loss is KL-divergence, weighted by the log of the number of reviews. Task losses are combined using scale-balancing weights: 5.0 for price, 5.0 for mass, 140.0 for materials, 1.0 for categories, and 1.0 for ratings (Standley et al., 2023).
The benchmark results show the effect of multimodality and synthetic completion. On the test set, Single Image Gen 2 achieved price 0.653 MnRE, mass 0.649, materials 72.7 F1, category 84.5 Acc, and ratings 0.153 KL-d. Text Gen 2 achieved 0.704, 0.695, 85.3, 91.9, and 0.147. Everything Gen 2 reached 0.743, 0.788, 86.0, 92.2, and 0.136. Generation-2 generally improves over generation-1, which the paper takes as support for the synthetic-label completion strategy (Standley et al., 2023).
The same modeling stack also supports external comparison. For mass prediction, the image model outperformed the prior image2mass model on the public image2mass Amazon test set: 0.709 vs 0.672 MnRE on their Amazon test set, and 0.694 vs 0.675 on their household test set. The paper further notes that text performs surprisingly well even on physical properties such as mass, a result that underscores how much information is encoded in seller-authored listings (Standley et al., 2023).
5. Smart Labeling and extensibility
The dataset’s extensibility is operationalized through Smart Labeling, a framework for adding a new binary property to all objects with very little manual effort. The method trains a linear logistic regression model over a compact multimodal object embedding, optionally augmented with task-specific hand-authored features. The workflow begins by collecting a seed set through word search, ideally around 50 positive and 50 negative examples, then uses uncertainty sampling active learning to label informative examples. It also includes a correction mode that targets false positives or false negatives by inspecting selected predicted-probability ranges, and a review mode that uses 20-fold cross-validation to rank likely mislabeled examples (Standley et al., 2023).
The underlying object representation is a 2048-dimensional bottleneck embedding trained from a 4542-dimensional input vector containing learned image and text embeddings, the original pretrained image and text embeddings, price, mass, materials, ratings, and 128-bit embeddings for product type and category. The embedding network is trained as an autoencoder-like reconstruction model with an L2 loss on the full input. It achieves average standardized L2 of 0.0054, interpreted by the authors as reconstructing each input feature to within 7.3% of its variance on average. From this embedding they report reconstruction of category at 99.8% accuracy, materials at 98.45 F1, mass with , and price with (Standley et al., 2023).
The framework is scalable because only hundreds to a couple thousand annotations are needed before probabilistic labels can be propagated across the remaining 2.8M+ objects. In practice, the paper adds five new properties: Sharp, Transparent, Deformable, Articulating, and Electronic. On manually curated test sets sampled uniformly from all products, with ambiguous cases discarded, the reported results are: Sharp with train 1331, positives 633; test 1504, positives 25; precision 92.0, recall 92.0, F1 92.0, accuracy 99.7. Transparent with train 2217, positives 877; test 706, positives 46; precision 93.5, recall 93.5, F1 93.5, accuracy 99.1. Deformable with train 636, positives 350; test 253, positives 150; precision 96.7, recall 97.3, F1 97.0, accuracy 96.4. Articulating with train 1399, positives 817; test 403, positives 148; precision 93.3, recall 94.6, F1 94.0, accuracy 95.5. Electronic with train 525, positives 248; test 301, positives 63; precision 96.8, recall 95.2, F1 96.0, accuracy 98.3 (Standley et al., 2023).
This design places EMMa closer to a mutable object database than to a fixed-label benchmark. A plausible implication is that the “extensible” component of the informal EMMM reading is not a marginal feature but the main systems-level claim.
6. Benchmark role, comparative position, and limitations
EMMa is positioned as a benchmark for multimodal, multi-task learning in which each attribute can serve either as model input or as prediction target. Its practical support for this claim comes from two sources: first, every record bundles heterogeneous modalities and metadata in aligned form; second, the release includes pretrained embeddings and joint multi-task image and text encoders that predict all major tasks simultaneously. The Everything models then fuse these modality-specific embeddings with tabular attributes, allowing raw images and text, released embeddings, synthetic labels, and Smart-Labeled attributes to be used together (Standley et al., 2023).
Relative to prior datasets, the paper argues that EMMa combines several properties rarely present together: 2.8M+ objects, object-centric multimodality, multiple prediction targets per instance, a 182-node material taxonomy, and a built-in mechanism for adding more labels. Compared with materials datasets such as OpenSurfaces and MINC, it is described as much larger in object count and richer in material granularity, while also including materials that may not be visible on the surface, such as copper inside a wire. Compared with Amazon-based datasets such as ABO or image2mass, it is described as broader in attribute coverage and task diversity. The authors explicitly position it as the first work to exploit the bulk of information in Amazon product listings for machine learning in this integrated way (Standley et al., 2023).
The paper is also explicit about noise and limitations. Amazon metadata are incomplete, seller-entered, and temporally inconsistent. Materials originate as free-form seller strings and are therefore noisy. Prices can be missing for inactive products and can differ across sources, in which case the median is used. Categories and materials are absent in many records before synthetic completion. Some listings use generic images, contain mismatched text and images, or represent nonphysical items; seller behavior can change listing contents under the same ASIN over time. Smart Labeling is sensitive to annotation mistakes because active learning magnifies the impact of mislabeled points, and property definitions themselves can be ambiguous. For Smart Labeling test sets, examples that could not be labeled confidently were discarded, so reported results do not cover the hardest ambiguous cases. Synthetic labels are useful but imperfect: in a sanity check on 100 initially unlabeled-category products, 82% of synthetic category labels were judged perfect or very good, versus 86% on a sample with Amazon categories already present (Standley et al., 2023).
Taken as a whole, EMMa provides the clearest referent for an “EMMM” query understood as extensible multimodal multi-task learning with materials. The formal acronym is different, but the conceptual correspondence is close: a large-scale Amazon-derived object dataset designed for extensible labeling, multimodal prediction, combinatorial task definition, and large-scale hierarchical material annotation (Standley et al., 2023).
