---
title: 'EMMa: Extensible Multimodal, Multi-task Dataset'
url: https://www.emergentmind.com/topics/emmm
type: topic
---

# EMMa: Extensible Multimodal, Multi-task Dataset

EMMM is not the formal name of an established benchmark in the relevant paper. In this arXiv context, it most plausibly functions as an informal shorthand for the combination of **extensibility**, **multimodality**, **multi-task learning**, and **materials-centric annotation** realized by **EMMa**, the dataset introduced in [2305.14352]. The formal dataset name is **EMMa**, not EMMM: the title capitalizes **E**xtensible, **M**ultimodal, and **M**ulti-task, while the final **“a”** comes from the dataset name rather than another M-word. EMMa is an Amazon-derived, object-centric corpus in which each listing may include images, text, mass, price, ratings, category information, materials, and other attributes, with the explicit design principle that each attribute can serve either as a model input or as a prediction target [2305.14352].

## 1. Nomenclature and conceptual framing

EMMa was introduced to address three linked problems. The first is the dataset-creation bottleneck in modern machine learning, especially when practitioners need labels that are expensive, ambiguous, or difficult to specify to crowd workers. The second is the lack of a very large, fine-grained, object-centric materials dataset. The third is the absence of a benchmark in which multiple aligned attributes can be recombined into many input-output task configurations across computer vision and NLP [2305.14352].

The work is explicitly object-centric rather than scene-centric. That choice matters because the target attributes are attached to product listings keyed to a single object instance, which makes cross-modal reasoning direct: listing text, listing images, category placement, price, mass, ratings, and materials are all aligned to the same product record. The authors therefore describe the space of supported tasks as **combinatorial**. Example tasks mentioned include predicting category from listing text, predicting mass and price from image, predicting ratings and price from text, and predicting materials from all other attributes [2305.14352].

This framing also explains why the informal label “EMMM” maps closely onto EMMa. The core idea is not merely multimodal supervision, but a reusable object record in which extensibility and multi-task recombination are first-class design goals.

## 2. Corpus construction and modality structure

EMMa is assembled from Amazon product listings and contains **2,883,698 instances** after filtering. The merged raw pool reached about **18 million entries**, drawn from three ASIN-keyed sources: roughly **2 million** items from the **Amazon Selling Partner API (ASPA)**, about **3 million** listings from the raw **image2mass** dataset, and about **15 million** listings from the **UCSD Amazon product rating/review** dataset. The final split is **2,806,806 train**, **26,535 validation**, **26,941 test**, and **23,416 calibration** [2305.14352].

The dataset is multimodal and attribute-rich. It contains **7,389,213 images total**, averaging **2.56 images per listing**, at original **500×500** resolution. Every listing has a title; **88%** have bullet-point product features; **63%** have a description. The text input is the concatenation of title, features, and description. The released embeddings include image representations obtained by summing **ConvNeXT-Large** features over listing images and text representations from **all-mpnet-base-v2**. Structured attributes include **mass/weight** for **77%** of listings, **price** for **69%**, **materials** for **30%**, **category** for **70%**, and at least one **review/rating** for **75%**. Most listings also have some size information, and **85%** have package weight. Product type is included with **1,296** unique seller-facing types. The inferred category hierarchy has **11,592 nodes** and maximum depth **6** [2305.14352].

The corpus is heavily cleaned. Duplicates and near-duplicates were removed using both NLP and image embeddings; entries with no title or no image were removed; any entry lacking all of mass, materials, and category was dropped; mismatched image/text listings were detected and removed; nonphysical or unhelpful listings were filtered; and ASIN collisions caused by sellers overwriting listings over time were heuristically detected and discarded. Validation and test were chosen from objects with all attributes filled in, with more than one image and more than one review, and were deduplicated more aggressively against train; image2mass Amazon test items were excluded from training for fair comparison [2305.14352].

## 3. Hierarchical material annotation

A central contribution of EMMa is its materials annotation scheme. Raw Amazon materials are seller-entered strings, so the authors clean and map them into a hand-curated **hierarchical taxonomy of 182 material types**. The taxonomy is explicitly hierarchical, with examples such as **Plastic → Thermoplastic → Acrylic**, and branches including metal subtypes such as **steel** and **aluminum**, as well as **wood**, **fabric**, **leather**, **diamond**, **marble**, **food**, **electronics**, and **liquid** [2305.14352].

The taxonomy is described as designed “in a data-driven way” to capture what sellers actually write. It intentionally includes entries that may not be “pure materials” in a chemistry sense, including **plush**, **microfiber**, **leather**, and **wood**, because these imply useful parent classes and matter for downstream physical and economic properties. Each object may have **multiple materials**; on average, each object has **4.9 material labels**. Because seller input is often coarse, labels may attach to internal nodes rather than leaves: an object may be known to be **plastic** without specifying acrylic versus polypropylene, or **metal** without specifying steel versus aluminum [2305.14352].

Material quality control is unusually explicit. **11% of the calibration set** was manually labeled, focusing on difficult instances, and that calibration set is used for training and calibrating material probabilities. The Smart Labeling procedure was also applied to **19 material nodes** to refine labels across the full dataset: **nylon plastic, nylon fabric, cardboard, cardstock, diamond, liquid, electronics, food, gold, genuine leather, lace, marble, faux-fur, cherry, leather, fabric, wood, metal, and plastic**. Coverage is substantial: **78 nodes** have more than **10k** instances, and **91%** of nodes have more than **500** instances [2305.14352].

This hierarchical structure has direct modeling implications. Material prediction is not limited to leaf classes: models predict a **probability distribution over all 182 material nodes**, even when the original annotation is only at an internal node. A listing labeled only as **metal** can therefore induce probabilities over children such as steel and aluminum. An **iterative consistency algorithm** is then applied so hierarchical probabilities obey the constraint that a parent probability must lie between its highest child probability and the sum of its children’s probabilities; because an object may contain several children under a parent, the parent is not constrained to equal the sum of its children [2305.14352].

## 4. Synthetic completion and multitask learning

EMMa treats missing-label completion as a predictive problem over aligned object attributes. For each missing attribute except size—**price, mass, materials, categories, ratings distribution**—models are trained to predict that attribute from all the others. The architecture is a late-fusion system: a vision model, **ConvNeXT-Base pretrained on ImageNet-22k**, is fine-tuned to jointly predict all tasks from images, and a text model, **all-mpnet-base-v2**, is fine-tuned to do the same from listing text. Both produce **768-d** embeddings, and task-specific **Everything** models take these standardized embeddings together with other available attributes. The authors describe the overall process as “a form of expectation-maximization”: generation-1 models are trained on original data, their predictions fill in missing labels, and generation-2 models are retrained on the completed dataset [2305.14352].

The paper defines task-specific metrics and losses. For price and mass, it uses **minimum ratio error (MnRE)**,
$$
\mathrm{MnRE}=\min\left(\frac{p}{t},\frac{t}{p}\right),
$$
where \(p\) is the prediction and \(t\) is the ground truth. For materials, the loss is the mean of **binary cross-entropy** across all 182 material types, with descendants excluded from the mean whenever a training label is an internal node, so the model is not penalized for being more specific than the annotation. For categories, the loss is a **cross-entropy at each level of the taxonomy**, combined by a weighted mean over levels; higher taxonomy levels are treated as more important, and only predictions under the correct parent branch are considered at deeper levels. For ratings, the loss is **KL-divergence**, weighted by the log of the number of reviews. Task losses are combined using scale-balancing weights: **5.0** for price, **5.0** for mass, **140.0** for materials, **1.0** for categories, and **1.0** for ratings [2305.14352].

The benchmark results show the effect of multimodality and synthetic completion. On the test set, **Single Image Gen 2** achieved price **0.653 MnRE**, mass **0.649**, materials **72.7 F1**, category **84.5 Acc**, and ratings **0.153 KL-d**. **Text Gen 2** achieved **0.704**, **0.695**, **85.3**, **91.9**, and **0.147**. **Everything Gen 2** reached **0.743**, **0.788**, **86.0**, **92.2**, and **0.136**. Generation-2 generally improves over generation-1, which the paper takes as support for the synthetic-label completion strategy [2305.14352].

The same modeling stack also supports external comparison. For mass prediction, the image model outperformed the prior image2mass model on the public image2mass Amazon test set: **0.709 vs 0.672** MnRE on their Amazon test set, and **0.694 vs 0.675** on their household test set. The paper further notes that text performs surprisingly well even on physical properties such as mass, a result that underscores how much information is encoded in seller-authored listings [2305.14352].

## 5. Smart Labeling and extensibility

The dataset’s extensibility is operationalized through **Smart Labeling**, a framework for adding a **new binary property** to all objects with very little manual effort. The method trains a **linear logistic regression** model over a compact multimodal object embedding, optionally augmented with task-specific hand-authored features. The workflow begins by collecting a seed set through word search, ideally around **50 positive and 50 negative** examples, then uses **uncertainty sampling** active learning to label informative examples. It also includes a correction mode that targets false positives or false negatives by inspecting selected predicted-probability ranges, and a review mode that uses **20-fold cross-validation** to rank likely mislabeled examples [2305.14352].

The underlying object representation is a **2048-dimensional** bottleneck embedding trained from a **4542-dimensional** input vector containing learned image and text embeddings, the original pretrained image and text embeddings, price, mass, materials, ratings, and **128-bit** embeddings for product type and category. The embedding network is trained as an autoencoder-like reconstruction model with an **L2 loss** on the full input. It achieves average standardized L2 of **0.0054**, interpreted by the authors as reconstructing each input feature to within **7.3% of its variance on average**. From this embedding they report reconstruction of category at **99.8% accuracy**, materials at **98.45 F1**, mass with **\(R^2 = 0.998\)**, and price with **\(R^2 = 0.960\)** [2305.14352].

The framework is scalable because only hundreds to a couple thousand annotations are needed before probabilistic labels can be propagated across the remaining **2.8M+** objects. In practice, the paper adds five new properties: **Sharp**, **Transparent**, **Deformable**, **Articulating**, and **Electronic**. On manually curated test sets sampled uniformly from all products, with ambiguous cases discarded, the reported results are: **Sharp** with train **1331**, positives **633**; test **1504**, positives **25**; precision **92.0**, recall **92.0**, F1 **92.0**, accuracy **99.7**. **Transparent** with train **2217**, positives **877**; test **706**, positives **46**; precision **93.5**, recall **93.5**, F1 **93.5**, accuracy **99.1**. **Deformable** with train **636**, positives **350**; test **253**, positives **150**; precision **96.7**, recall **97.3**, F1 **97.0**, accuracy **96.4**. **Articulating** with train **1399**, positives **817**; test **403**, positives **148**; precision **93.3**, recall **94.6**, F1 **94.0**, accuracy **95.5**. **Electronic** with train **525**, positives **248**; test **301**, positives **63**; precision **96.8**, recall **95.2**, F1 **96.0**, accuracy **98.3** [2305.14352].

This design places EMMa closer to a mutable object database than to a fixed-label benchmark. A plausible implication is that the “extensible” component of the informal EMMM reading is not a marginal feature but the main systems-level claim.

## 6. Benchmark role, comparative position, and limitations

EMMa is positioned as a benchmark for multimodal, multi-task learning in which **each attribute can serve either as model input or as prediction target**. Its practical support for this claim comes from two sources: first, every record bundles heterogeneous modalities and metadata in aligned form; second, the release includes pretrained embeddings and joint multi-task image and text encoders that predict all major tasks simultaneously. The **Everything** models then fuse these modality-specific embeddings with tabular attributes, allowing raw images and text, released embeddings, synthetic labels, and Smart-Labeled attributes to be used together [2305.14352].

Relative to prior datasets, the paper argues that EMMa combines several properties rarely present together: **2.8M+ objects**, object-centric multimodality, multiple prediction targets per instance, a **182-node** material taxonomy, and a built-in mechanism for adding more labels. Compared with materials datasets such as **OpenSurfaces** and **MINC**, it is described as much larger in object count and richer in material granularity, while also including materials that may not be visible on the surface, such as **copper inside a wire**. Compared with Amazon-based datasets such as **ABO** or **image2mass**, it is described as broader in attribute coverage and task diversity. The authors explicitly position it as the first work to exploit the bulk of information in Amazon product listings for machine learning in this integrated way [2305.14352].

The paper is also explicit about noise and limitations. Amazon metadata are incomplete, seller-entered, and temporally inconsistent. Materials originate as free-form seller strings and are therefore noisy. Prices can be missing for inactive products and can differ across sources, in which case the median is used. Categories and materials are absent in many records before synthetic completion. Some listings use generic images, contain mismatched text and images, or represent nonphysical items; seller behavior can change listing contents under the same ASIN over time. Smart Labeling is sensitive to annotation mistakes because active learning magnifies the impact of mislabeled points, and property definitions themselves can be ambiguous. For Smart Labeling test sets, examples that could not be labeled confidently were discarded, so reported results do not cover the hardest ambiguous cases. Synthetic labels are useful but imperfect: in a sanity check on 100 initially unlabeled-category products, **82%** of synthetic category labels were judged perfect or very good, versus **86%** on a sample with Amazon categories already present [2305.14352].

Taken as a whole, EMMa provides the clearest referent for an “EMMM” query understood as **extensible multimodal multi-task learning with materials**. The formal acronym is different, but the conceptual correspondence is close: a large-scale Amazon-derived object dataset designed for extensible labeling, multimodal prediction, combinatorial task definition, and large-scale hierarchical material annotation [2305.14352].

Source: https://www.emergentmind.com/topics/emmm