TorchUMM: Unified Multimodal Codebase

• TorchUMM is a unified multimodal model framework that standardizes benchmarking across diverse architectures and tasks for reproducible evaluation.
• It employs a modular design with a common BackboneAdapter interface, enabling consistent evaluation of models in understanding, generation, and editing.
• TorchUMM integrates post-training methods to analyze trade-offs and performance shifts, offering actionable insights for comparative research.

TorchUMM is a unified, modular PyTorch codebase and benchmark suite for unified multimodal models (UMMs), defined as models that can understand, generate, and edit images using natural language within a common experimental framework. It was introduced as the first unified codebase for comprehensive evaluation, analysis, and post-training across diverse UMM backbones, tasks, and datasets, with the explicit goal of making comparisons fair and reproducible despite substantial heterogeneity in architectures, tokenizations, inference engines, and training paradigms (Luo et al., 12 Apr 2026).

1. Scope and conceptual role

TorchUMM addresses a specific fragmentation problem in UMM research. UMMs differ along multiple axes: some are autoregressive decoder-only transformers, some are diffusion-based, some are hybrid AR + diffusion or flow systems, and others use modular orchestration rather than a highly unified backbone. In the pre-TorchUMM setting, each family typically shipped with its own evaluation scripts, its own preprocessing and decoding assumptions, and its own narrow benchmark focus. The resulting comparisons were not directly comparable, even when reported on ostensibly similar tasks (Luo et al., 12 Apr 2026).

The codebase therefore serves three tightly connected purposes: evaluation, analysis, and post-training. Evaluation refers to fair, reproducible benchmarking of heterogeneous UMM architectures across many datasets and tasks. Analysis refers to diagnostic tools for probing capabilities, trade-offs, and the effects of architectural unification. Post-training refers to a unified environment for applying and comparing methods such as SFT, reasoning-augmented training, reconstruction alignment, and self-play. A central premise is that improvements in one capability often degrade others, and that this should be made visible rather than hidden behind benchmark-specific pipelines (Luo et al., 12 Apr 2026).

TorchUMM is positioned as a framework for UMMs analogous, in concept, to toolkits such as OpenCompass, VLMEvalKit, or LMMS-Eval for LLMs and VLMs, but extended to cover all three multimodal dimensions—understanding, generation, and editing—while also supporting heterogeneous architectures and post-training utilities. This suggests that TorchUMM is not merely an evaluation harness, but an infrastructure layer for controlled comparative research on multimodal unification.

2. Architecture and supported model taxonomy

TorchUMM is organized into four layers: an infrastructure layer, a core functionality layer, a task and execution layer, and an application and API layer (Luo et al., 12 Apr 2026). The infrastructure layer provides common utilities and integration with external libraries such as vLLM, GPU management, and logging. The core functionality layer contains UMM backbones, benchmark datasets, and post-training methods. The task and execution layer implements unified pipelines for understanding, generation, editing, and cross-task evaluation. The application and API layer exposes the configuration system, CLI and Python APIs, and reporting tools.

The key software abstraction is the BackboneAdapter protocol. Each supported model implements a standardized interface with load(cfg) and generate(batch, gen_cfg), so model-specific quirks are hidden behind a task-agnostic API. The load method handles weights, device placement, sharding, and tokenizer or image-tokenizer setup. The generate method handles standardized requests containing prompts, images, task type, and decoding hyperparameters. This design allows the same inference and evaluation logic to be reused across backbones with very different internals (Luo et al., 12 Apr 2026).

TorchUMM supports 14 models spanning major UMM design paradigms. These include Bagel (7B, 14B), OmniGen2 (7B), Emu3 (8B), Emu3.5 (34B), Janus (1.3B), JanusFlow (1.3B), Janus-Pro (7B), Show-o (1.3B), Show-o2 (1.5B, 7B), BLIP3-o (4B), TokenFlow (7B), DeepGen 1.0 (5B), and MMaDA (8B) (Luo et al., 12 Apr 2026). The supported taxonomy spans autoregressive decoder-only transformers, diffusion-based models, hybrid AR + diffusion or flow models, discrete visual-token systems, latent or CLIP-feature generators, and varying degrees of architectural unification from highly unified to modular orchestration.

A concise view of TorchUMM’s task-facing design is as follows:

Component	Function	Examples from TorchUMM
BackboneAdapter	Standardized backbone interface	Bagel, Emu3, OmniGen2, Janus, Show-o2
YAML configs	Experiment specification	inference, evaluation, post-training
Unified pipelines	Task execution	understanding, generation, editing
Registries and runners	Extensibility	new models, benchmarks, post-training methods

Because evaluation logic is decoupled from model internals, adding a new model typically requires implementing an adapter, registering it, and supplying configs rather than modifying the benchmark pipeline itself (Luo et al., 12 Apr 2026).

3. Task coverage and standardized evaluation

TorchUMM organizes benchmarking around three core task dimensions: multimodal understanding, multimodal generation, and multimodal editing (Luo et al., 12 Apr 2026). For understanding, it includes MME, MMMU, MMBench, MM-Vet, and MathVista. These benchmarks cover perception, OCR, cognition, domain-specific knowledge, general multimodal reasoning, robustness, and visual mathematical reasoning. For generation, it includes DPG-Bench, GenEval, WISE, and UEval, focusing on object correctness, attribute binding, spatial arrangement, compositionality, world-knowledge consistency, and structurally constrained generation. For editing, it includes GEdit-Bench and ImgEdit, covering attribute modification, object addition or removal, style transfer, text editing, and multi-turn editing (Luo et al., 12 Apr 2026).

Dimension	Representative benchmarks	Evaluated aspects
Understanding	MME, MMMU, MMBench, MM-Vet, MathVista	perception, cognition, knowledge, reasoning
Generation	DPG-Bench, GenEval, WISE, UEval	compositionality, fidelity, world knowledge
Editing	GEdit-Bench, ImgEdit	semantic correctness, perceptual quality, multi-turn consistency

The evaluation protocol is standardized end to end. Benchmark runners convert raw datasets into a canonical format of images, prompt text, options, and gold answers. Inference uses fixed decoding or sampling parameters per benchmark and model class, with the same number of samples per prompt for fairness. Scoring uses official scripts where available and wraps external judge models such as Qwen2.5-VL-72B for WISE, GEdit, ImgEdit, and UEval so that outputs, logs, and reporting formats remain uniform (Luo et al., 12 Apr 2026).

Several benchmark-specific metrics are explicitly normalized by TorchUMM’s runners. MME uses perception and cognition scores. MMMU, MMBench, and MathVista use accuracy, with MathVista further split into multiple-choice and free-form. WISE uses WiScore in $[0,1]$, judged by Qwen2.5-VL-72B. GEdit-Bench uses SC, PQ, and an overall score defined as the geometric mean,

$O = \sqrt{\text{SC} \times \text{PQ}}$

as stated in the TorchUMM description (Luo et al., 12 Apr 2026).

A significant feature is support for unified and cross-task benchmarks such as Uni-MMMU and UEval. These are designed to expose failure modes in tightly coupled multimodal workflows that require perception, reasoning, and controlled generation in sequence rather than in isolation. This suggests that TorchUMM’s notion of “unified evaluation” is not restricted to aggregating task-specific scores; it also probes procedural and stateful multimodal behavior.

4. Post-training framework and ecosystem modules

TorchUMM includes a dedicated post-training subsystem under src/umm/post_training/, and the base paper reports unified support for five post-training methods: SFT, IRG, UniCoT, RecA, and UniGame (Luo et al., 12 Apr 2026). SFT is standard supervised fine-tuning on multimodal instruction-response pairs. IRG encourages explicit reasoning before image generation. UniCoT promotes chain-of-thought across text and vision. RecA aligns understanding and generation through reconstruction and contrastive terms. UniGame uses self-play supervision. The defining feature is not simply that these methods are implemented, but that their checkpoints are evaluated under the same benchmark and scoring protocol as the base models, which makes cross-method trade-offs visible.

The TorchUMM ecosystem was subsequently extended with additional post-training recipes. LatentUMM is implemented in src/umm/post_training/LatentUMM as a post-training framework that explicitly refines the shared latent space of a pretrained UMM. Its stated goal is to fix functional inconsistency between understanding and generation by combining dual latent alignment with latent dynamics stabilization, while remaining architecture-agnostic across backbones such as Bagel, Janus-Pro, and Harmon (Luo et al., 18 May 2026). UniPath is implemented in src/umm/post_training/unipath as a planner-executor post-training and routing layer for TorchUMM-compatible UMMs, specifically BAGEL in the reported experiments. Rather than enforcing a single coordination pattern, it introduces path-conditioned reasoning behaviors and a lightweight planner that chooses among paths such as direct answering, explicit understanding, textual reasoning, visual-thought construction, and hypothesis exploration (Bai et al., 12 May 2026).

These extensions clarify the design philosophy of TorchUMM. The codebase is not limited to generic fine-tuning loops; it supports post-training strategies that alter latent alignment, reasoning structure, routing, or multimodal coordination while reusing the same evaluation harness. A plausible implication is that TorchUMM functions as an experimental substrate for comparing not only backbones, but also different theories of what “unification” should mean in practice: shared latent geometry, explicit reasoning paths, reconstruction alignment, or self-play.

5. Empirical findings and analytical use

TorchUMM’s comparative experiments show that no single model dominates all capabilities (Luo et al., 12 Apr 2026). In generation, DeepGen (5B) achieves GenEval 86.59 and DPG 87.44, making it the strongest compositional generator in the reported setup, while Emu3.5 (34B) reaches WISE 0.633, the best world-knowledge alignment score. Bagel (7B) and OmniGen2 (7B) are reported as strong all-round generators, with DPG 84.11 and 84.51 respectively. In understanding, Bagel (14B) is the strongest model in the benchmark, with MME P/C 1691.5 / 695.4, MMMU 0.519, MMBench 0.843, MM-Vet 65.9, and MathVista 71.6. In editing, Emu3.5 leads the models with native editing support, including GEdit-EN O 7.56 and ImgEdit ST/MT/UGE 4.24 / 4.89 / 4.88 (Luo et al., 12 Apr 2026).

The framework also makes systematic trade-offs visible. The base paper reports that naive SFT is unstable and often harmful, with Bagel’s WISE score dropping from 0.399 to 0.227 under SFT and TokenFlow’s DPG dropping from 71.29 to 22.16. Method effectiveness is backbone-dependent: RecA improves Bagel’s DPG from 84.11 to 85.20 and GenEval from 78.81 to 83.05, but MathVista drops from 71.6 to 51.6. UniGame improves Bagel’s GenEval from 78.81 to 85.8 while DPG drops from 84.11 to 65.77. These results are central to TorchUMM’s research value because they show that post-training typically shifts capability mass rather than uniformly boosting performance (Luo et al., 12 Apr 2026).

TorchUMM also supports model-internal and behavior-level analysis. One case study on UEval argues that architectural unification degree is not a reliable proxy for capability: the paper informally treats MMaDA as most unified, Show-o2 as intermediate, and OmniGen2 as least unified, yet only OmniGen2 consistently attempts certain tightly coupled, stepwise visual-textual tasks, and UEval scores on example cases favor OmniGen2 over the more unified alternatives (Luo et al., 12 Apr 2026). Another analysis examines robustness to meaning-preserving query variations on MathVista by comparing output-embedding similarity and internal hidden-state similarity across rephrasings. OmniGen2 remains close to its Qwen2.5-VL-3B backbone, while Show-o2 departs more substantially from its Qwen2.5-VL-7B backbone. This is used to argue that aggressive unification training can deform inherited backbone behavior (Luo et al., 12 Apr 2026).

These analytical results address two common misconceptions. The first is that stronger architectural unification automatically yields stronger multimodal coordination; TorchUMM’s UEval case study explicitly argues against that simplification. The second is that post-training gains should be interpreted benchmark by benchmark; TorchUMM instead emphasizes holistic evaluation because improvements in one dimension often coincide with regressions in another.

6. Extensibility, limitations, and related distinctions

TorchUMM is explicitly designed for extensibility. Adding a new model requires implementing a BackboneAdapter subclass under src/umm/backbones/, registering it, and supplying YAML configs. Adding a new benchmark requires a dataset loader and runner plus configs and CLI hooks. Adding a new post-training method requires a training loop under src/umm/post_training/ and method registration (Luo et al., 12 Apr 2026). This modularity is reinforced by the fact that evaluation logic is separated from model internals, so new backbones can participate in the existing benchmark suite without rewriting task runners.

The current scope remains bounded. The framework focuses on vision-language UMMs; audio, video, and 3D are not integrated in the reported version. Even with broad benchmark coverage, unexplored domains remain, including multi-image reasoning, interactive agents, and richer world-model tasks. Post-training diversity is also partial: the paper notes no explicit RLHF pipelines and only limited preference-based optimization in the base release. The authors further characterize their unification analysis as observational rather than causal, since backbone, data, tokenizer, and training recipe are not all controlled simultaneously (Luo et al., 12 Apr 2026).

A practical clarification concerns naming. TorchUMM is unrelated to XBTorch, despite superficial similarity in nomenclature. XBTorch, or CrossBarTorch, is a PyTorch-native framework for modeling crossbar-based in-memory accelerators built from emerging memory technologies such as FeFET and ReRAM, with emphasis on device-level modeling, hardware-aware training, and inference-time fault tolerance rather than unified multimodal modeling (Yousuf et al., 11 Jan 2026). This distinction matters because “TorchUMM” in the multimodal literature refers specifically to a unified multimodal model codebase, not to a unified memory-modeling framework.

The stated future directions for TorchUMM include integrating more modalities, adding larger and more diverse UMMs, developing controlled experiments that vary unification degree while holding other factors fixed, extending post-training to more advanced alignment methods such as RLHF-like and reflection-based RL approaches, and expanding analysis modules for safety and robustness (Luo et al., 12 Apr 2026). Taken together, these directions position TorchUMM as an evolving research infrastructure for multimodal systems rather than a closed benchmark release.