Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy

Abstract: Document classification forms the backbone of modern enterprise content management, yet existing benchmarks remain trapped in oversimplified paradigms -- single domain settings with flat label structures -- that bear little resemblance to the hierarchical, multi-modal, and cross-domain nature of real-world business documents. This gap not only misrepresents practical complexity but also stifles progress toward industrially viable document intelligence. To bridge this gap, we construct the first Multi-level, Multi-domain, Multi-modal document classification Benchmark (MMM-Bench). MMM-Bench includes (1) a deeply hierarchical taxonomy spanning five levels that capture the authentic organizational logic of business documentation; and (2) 5,990 real-world multi-modal documents meticulously curated from 12 commercial domains in Alibaba. Each document is manually annotated with a complete hierarchical path by domain experts. We establish comprehensive baselines on MMM-Bench, which consists of open-weight models and API-based models. Through systematic experiments, we identify four fundamental challenges within MMM-Bench and propose corresponding insights. To provide a solid foundation for advancing research in multi-level, multi-domain document classification, we release all of the data and the evaluation toolkit of MMM-Bench at https://github.com/MMMDC-Bench/MMMDC-Bench.

• The paper introduces MMM-Bench, the first benchmark incorporating a 5-level hierarchical taxonomy for multi-domain, multimodal document classification.
• It presents a comprehensive methodology with expert annotations and baseline evaluations using state-of-the-art models across diverse document types.
• Experimental results highlight challenges in fine-grained semantic discrimination, domain-specific performance, cross-modal fusion, and long-tailed class distributions.

Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy

Introduction and Motivation

Automatic document classification underpins numerous enterprise content management systems, supporting automated archiving, intelligent retrieval, and workflow automation. Despite major advances in representation learning and document understanding, existing benchmarks for document classification remain limited by three constraints: flat (non-hierarchical) label sets, single-domain focus, and restricted or single-modal content. These limitations diverge from the authentic complexity of real-world enterprise documents, which typically require hierarchical multi-label assignment, integrate visual and textual modalities, and span heterogeneous business domains.

The paper introduces MMM-Bench, the first benchmark that systematically captures these real-world complexities. MMM-Bench delivers a robust evaluation framework comprising a 5-level taxonomy, documents sourced across 12 domains, and rich multi-modal annotations including text, tabular data, layout structures, and visual elements. The corpus consists of 5,990 expert-annotated documents, each assigned a full path within the hierarchical taxonomy.

Hierarchical Taxonomy Design

Central to MMM-Bench is its deeply hierarchical taxonomy, modeled after real business logic in Alibaba's enterprise environment. The taxonomy is organized into five discrete levels:

• L1 (Business Function): Assigns the document's macro-purpose (e.g., Transaction Voucher, Invoice, Contract).
• L2 (Business Object): Identifies the type of entity involved (e.g., Goods, Payment).
• L3 (Business Stage): Encodes transactional temporality (e.g., Order, Receipt).
• L4 (Business Form): Specifies the template/schema (e.g., Purchase Order).
• L5 (Document Class): Provides the finest technical class.

This taxonomy is strictly hierarchical, supporting fine-grained evaluation and reflecting realistic document structures.

Figure 1: Example depicting the five-level hierarchical taxonomy spanning business function, object, stage, form, and class.

Multi-modal Multi-domain Corpus Construction

The dataset is curated from authentic internal business repositories, spanning 12 domains: Finance, Taxation, Investment, Legal, Human Resources, Procurement, Corporate Governance, Real Estate, Technology, Construction, Logistics, and Marketing. Each document is multi-modal, containing OCR-extracted and hand-verified text, tabular features, logos, stamps, signatures, and complex layouts. Textual content is the dominant modality, with ~99.8% coverage, complemented by substantial tabular and visual information. The language profile is primarily Chinese, with minor samples in English, Spanish, and Portuguese.

Annotation quality is ensured via a rigorous multi-stage expert protocol involving initial guideline construction, triple-expert blind annotation, and final adjudication by senior experts to resolve conflicts.

Experimental Protocol and Baseline Evaluation

MMM-Bench supports multi-level hierarchical classification where models predict a path from root to leaf. The benchmark integrates 10+ SOTA open-weight and API-based models, including Qwen2.5-VL, Qwen3-VL, Qwen-3.5, Claude-Opus-4.5/4.6, GPT-5.2/5.4, Gemini-3.1-Pro, MiniMax-M2.7, GLM-5, and Kimi-K2.5.

Evaluation comprises per-level accuracy, macro-F1, and an aggregate Hierarchical F1-score (HF1) capturing path-level sequence correctness. All models are profiled under standardized zero-shot, few-shot, and supervised fine-tuning protocols to assess both intrinsic and adaptable classification capabilities.

Analysis and Challenges Unveiled

Granular Performance Differentiation

There is a marked performance drop from coarse (L1/L2) to fine-grained levels (L3-L5) irrespective of model family. For instance, Claude-Opus-4.5 and Gemini-3.1-Pro achieve high accuracy at L1 and L2 (above 95% and 83%, respectively), yet accuracy at L5 typically falls to 80% or lower. Open-weight models like Qwen-3.5-27B report 93.4% (L1) but only 53.3% (L5).

Figure 2: Accuracy progression across hierarchy levels for evaluated models, illustrating performance decay at finer granularity.

This evidences the semantic discrimination difficulty at fine granularity, where class boundaries are less distinct. Models struggle with subtle cues required for accurate L4/L5 classification.

Domain Sensitivity

Model performance exhibits significant domain-specific variability. Highest macro-F1 is observed in legal and corporate governance documents, attributable to high structural regularity and consistent terminological cues. In contrast, marketing and construction domains yield the lowest scores, reflecting domain heterogeneity and multimodal ambiguity.

Figure 3: Per-domain accuracy (%) highlights differential model performance, with structured domains (e.g., LEGAL) consistently outperforming unstructured ones (e.g., MARKETING).

This domain fragility motivates research into adaptive, domain-aware modeling and transfer learning.

Cross-modal Fusion Failure

Surprisingly, ablation studies establish that text-only inputs outperform multi-modal ones for leading vision-LLMs; the addition of visual and layout signals fails to improve (and often degrades) classification accuracy under current fusion and SFT approaches. This demonstrates that state-of-the-art models cannot currently exploit multimodality in document classification when presented with heterogeneous visual features, revealing a critical research gap for cross-modal representation and fusion strategies.

Long-tailed Class Distribution

The corpus exhibits a pronounced long-tailed distribution at all taxonomy levels; the majority of classes are severely under-represented, especially at L4/L5. Models display significant accuracy and F1-score drops on tail categories (average macro-F1 60.2%) compared to head (92.1%) and middle (87.7%) groups.

Figure 4: Long-tailed distribution of training samples in L1 categories, underscoring class imbalance and its impact on classification.

This distributional skew exposes limitations of current architectures in few-shot tail adaptation and highlights the necessity for hierarchical knowledge transfer and data augmentation.

Insights and Implications for AI Research

The analysis yields several actionable insights with direct implications for future AI research:

• Joint Hierarchical Training: Optimizing for all taxonomy levels simultaneously enables better representation sharing and gradient flow, essential for robust fine-grained classification.
• Domain-adaptive Modeling: Techniques such as domain-adversarial training, domain-specific mixture-of-experts, and meta-learning are indicated for improving cross-domain transfer and generalization.
• Cross-modal Attention Architectures: Naïve feature concatenation is insufficient; cross-modal gated attention and hierarchy-aware fusion mechanisms are required to exploit inter-modality complementarity adaptively.
• Hierarchical Knowledge Transfer: Approaches such as hierarchical parameter sharing, contrastive embedding learning, and taxonomy-informed data augmentation are crucial for addressing sample imbalance and improving tail accuracy.

Conclusion

MMM-Bench defines a new standard for document classification evaluation, introducing multi-level, multi-domain, and multimodal complexity in a single benchmark. The systematic experimental analysis with diverse large models reveals four fundamental challenges: semantic discrimination at fine-grained levels, domain-sensitive performance, cross-modal fusion limitations, and sample distribution imbalance. These results both demonstrate the current limitations of state-of-the-art models and chart explicit technical trajectories for future research in hierarchical, multi-domain, and multimodal document intelligence (2605.10550).