FETA: Benchmark for Expert Vision–Language Tasks
- FETA is a benchmark dataset for expert vision–language tasks, focusing on technical manuals and catalogs with specialized imagery and detailed text.
- It uses an automatic PDF parsing pipeline via IBM DeepSearch to extract layout-aware images and text, addressing noise and domain-specific challenges.
- The dataset supports multimodal retrieval tasks through standardized splits and includes both automated and human-curated annotations for robust evaluation.
FETA is a benchmark dataset for specializing vision–language foundation models on expert, long-tail applications, rather than on the generic web-style imagery and colloquial captions that dominate standard evaluation suites. In the formulation of "FETA: Towards Specializing Foundation Models for Expert Task Applications," it is built around technical documentation and catalog material, with the central objective of teaching foundation models to match graphical illustrations and language descriptions in domains where out-of-the-box performance is weak, such as technical car manuals and consumer catalogs (Alfassy et al., 2022).
1. Motivation and problem setting
FETA was created to evaluate foundation models on long-tail distribution data such as technical manuals or consumer catalogs, where the input distribution differs sharply from common internet images and short captions (Alfassy et al., 2022). The benchmark targets a recurring limitation of large pretrained models: strong zero-shot and out-of-domain behavior on generic tasks does not translate reliably to expert documentation, where both the language and the visual content are specialized.
The motivating failure modes are specific. The underlying documents contain technical terminology such as part names and product identifiers; diagrams that do not resemble natural photographs, including exploded views and wiring-like schematics; and dense page layouts in which relevant text appears in blocks, headings, and structured fragments rather than as a single caption. This makes FETA a benchmark for grounded multimodal understanding under domain shift, rather than merely another image–caption corpus (Alfassy et al., 2022).
The benchmark is also motivated by practical industrial workflows. The datasheet associates the target problem with tasks such as retrieving the right diagram or maintenance step in a car repair manual, retrieving a product image in an IKEA catalog from a textual description, and navigating multimodal technical documents. This suggests a deliberate shift from common-object benchmarking toward professional documentation search and assistance systems (Alfassy et al., 2022).
2. Corpus, sources, and dataset structure
FETA consists of documents drawn from car manuals and IKEA catalogs. The materials originate as PDF files and include technical illustrations, product imagery, and associated text such as descriptions, instructions, parts information, and headings. The full version of the dataset contains approximately 56,000 images and approximately 89,000 pieces of extracted image-related text (Alfassy et al., 2022).
| Source | Visual content | Associated text |
|---|---|---|
| Car manuals | Technical illustrations, mechanical diagrams, exploded views | Instructions, parts information, headings |
| IKEA catalogs | Product and catalog images | Descriptions, headings, surrounding text |
The dataset schema is page-centric. Each instance contains an extracted image, one or more text pieces extracted from the same page, and metadata linking images and texts by page location. The datasheet states that "each image is associated with a set of texts on the same page," and that the metadata includes page-level and location information connecting the two modalities (Alfassy et al., 2022).
This structure makes text-to-image retrieval and image-to-text retrieval natural benchmark tasks. A text query may be a contiguous block describing an operation or product, while the target is an image on the same page; conversely, an image query may retrieve the page text that describes the depicted component or item. The datasheet further notes that "any image and/or text learning tasks" can be used, although the benchmark is explicitly tailored to vision–language retrieval (Alfassy et al., 2022).
Recommended train/validation/test splits are provided in the supplementary material or the accompanying paper, indicating an intended standardized evaluation setup even though split details are not reproduced in the datasheet excerpt (Alfassy et al., 2022).
3. Automatic annotation extraction and preprocessing pipeline
A defining feature of FETA is its automatic construction pipeline. PDFs were manually downloaded and processed using IBM's DeepSearch system, which parses each document to extract images, text boxes, and bounding-box or spatial metadata. This makes FETA a layout-derived multimodal benchmark rather than a manually captioned dataset in the conventional sense (Alfassy et al., 2022).
The preprocessing pipeline has four main stages. First, PDF files are acquired from the source collections. Second, DeepSearch performs document parsing and returns images, layout-aware text blocks, and positional information. Third, the extracted text is cleaned by removing bad character artifacts from PDF parsing, filtering improbable or failed text boxes, and merging spatially close text boxes into larger, consecutive chunks of text. Fourth, image–text linkage is established primarily through page location, with proximity-based merging used to preserve multi-line captions or associated instructions as coherent text units (Alfassy et al., 2022).
The annotation process is explicitly automatic and explicitly noisy. The datasheet states that the data are automatically annotated and that there is "a significant amount of noise." This is a central characteristic of the benchmark rather than an incidental flaw: page-level and proximity-based associations scale to large document collections, but they do not guarantee perfect semantic alignment between every image and every linked text chunk (Alfassy et al., 2022).
The same pipeline is intended to be extensible beyond the initial domains. Because the procedure depends on PDFs, extracted images, extracted text boxes, and their positions, the methodology is not specific to car manuals or IKEA catalogs. The paper details are said to support extension to additional documentation types and application domains, and the dataset authors explicitly welcome expansion of the dataset (Alfassy et al., 2022).
4. Tasks, evaluation protocol, and benchmarking role
FETA is organized around multimodal retrieval in expert documentation, especially text-to-image and image-to-text retrieval. Its role is to test whether models that perform well on standard multimodal benchmarks can also ground language and vision in technical domains. The benchmark therefore functions both as an evaluation set and as a specialization resource for finetuning foundation models on expert tasks (Alfassy et al., 2022).
The abstract reports an important methodological component beyond raw annotation extraction: the automatic annotation process yields an automated performance metric that is shown to be consistent with metrics computed on human-curated annotations. It also states that human-curated annotations are released. At the same time, the datasheet excerpt does not enumerate exact metric names, formulas, or validation statistics, and it places the detailed evaluation protocol in the accompanying paper and supplementary materials (Alfassy et al., 2022).
The same distinction appears in the treatment of baselines. The abstract states that the work provides multiple baselines and analysis of popular foundation models on FETA, whereas the datasheet excerpt does not list the evaluated models, training regimes, or numerical results. A common misconception would be to infer that the benchmark is fully defined by a fixed leaderboard; the provided materials instead emphasize the benchmark design, the automatic extraction pipeline, and the need for explicit expert-domain evaluation of foundation models (Alfassy et al., 2022).
Technically, FETA is best understood as a benchmark for multimodal retrieval under layout noise, technical vocabulary shift, and diagrammatic visual content. That combination distinguishes it from datasets centered on common objects, natural photos, or short caption pairs.
5. Research uses, distribution, and limitations
FETA fills a gap between generic foundation-model benchmarks and real industrial use cases. It can be used to evaluate readiness for technical-documentation search, to measure the effect of domain-specific finetuning, and to study transfer across two distinct expert-domain types: car manuals and IKEA catalogs. A plausible implication is that it supports both capability assessment and specialization strategy design for multimodal systems intended for professional settings (Alfassy et al., 2022).
The benchmark is distributed under the GNU/GPL license. The datasheet states that it is available for download as a tarball on IBM cloud, hosted and maintained by IBM Research, with IBM cloud file versioning preserving older versions. It also notes that this is the first version of the dataset and that future versions will be updated and posted on GitHub. The raw car-manual files can be downloaded separately from workshopservicemanual.com, and the software used to preprocess, clean, and label the instances is reported as available (Alfassy et al., 2022).
Several limitations are explicit. The first is annotation noise from automatic page-based and proximity-based association. The second is domain scope: the current version focuses on car repair or maintenance documentation and IKEA product catalogs, rather than a wider set of industrial or multilingual technical corpora. The third is alignment granularity: the datasheet emphasizes automatic associations rather than fine-grained human relevance labels for subregions of diagrams or tightly localized text–image correspondences (Alfassy et al., 2022).
These constraints matter scientifically. They mean that FETA is not merely a clean benchmark for retrieval accuracy; it is also a testbed for robustness to document-layout noise, imperfect supervision, and expert-domain distribution shift. That is precisely why it is relevant to the study of foundation-model specialization.
6. Terminological scope and related acronym usage
In arXiv usage, the acronym is not unique. The FETA discussed here is the expert-task benchmark introduced in "FETA: Towards Specializing Foundation Models for Expert Task Applications" (Alfassy et al., 2022). It is unrelated to FeTA, the "Fetal Tissue Annotation" challenge and dataset for fetal brain MRI segmentation (Payette et al., 2022), to FeTA as the "Frequency Attention Network" for graph transformers (Bastos et al., 2022), or to FeTa as the "Fast and Efficient Trimming Algorithm" for pruning fully connected neural networks (Pitas et al., 2018).
This terminological overlap is consequential in literature search. In multimodal foundation-model research, FETA denotes a benchmark for technical documentation understanding; in medical imaging, graph learning, and pruning, the same or nearly identical acronym denotes different datasets or algorithms. For expert readers, disambiguation by title or arXiv identifier is therefore essential.