Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes

Abstract: Human cognition does not separate understanding and generation. A teacher at a whiteboard speaks and draws $\textit{together}$, each modality reshapes the other. In this paper, we bring this coupled loop to artificial systems. Masked Diffusion Models (MDMs) are ideally suited to this task, yet existing samplers either decode text and image interleavedly or independently update them in parallel branches that share only previous-step history, but not the other modality's latest decisions $\textit{within}$ the same step; combined with MDMs' inability to remask, cross-modal contradictions are neither detected nor repaired. We introduce $\textbf{Self-Correcting Coupled Markov Jump Processes (SC-CMJP)}$, a framework in which one modality's transition rates are functionals of the other modality's confidence score, as weighted by cross-modal attention. Furthermore, a remasking jump retracts commitments the moment cross-modal evidence turns against them. In conjunction with SC-CMJP, we introduce $\texttt{CO}\texttt{2}\texttt{Jump}$ (Self-$\underline{\text{CO}}$rrecting $\underline{\text{CO}}$upled $\underline{\text{Jump}}$), a novel training-free single-pass sampler for joint multimodal geneneration. For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora: $\text{JEdit-1M}$, $\text{JMaze-200K}$, $\text{JNono-200K}$, with matching in- and out-of-distribution benchmarks. $\texttt{CO}\texttt{2}\texttt{Jump}$ achieves best joint performance for image understanding and editing as well as visual reasoning (maze and nonogram solving). The performance of the sampler scales monotonically with the number of denoising steps, evidence that the benefits of cross-modal coupling $\textit{compound}$ across the trajectory. Project page: https://coupled-jump.github.io

• The paper introduces SC-CMJP and the CO sampler, a self-correcting framework that couples image and text generation to enforce cross-modal consistency.
• It utilizes a single-pass, training-free inference with dynamic remasking and cross-modal negotiations to overcome limitations of conventional MDM pipelines.
• Empirical results on large-scale benchmarks like JEdit-1M, JMaze-200K, and JNono-200K demonstrate monotonic improvements and robust state-of-the-art performance.

Self-Correcting Coupled Markov Jump Processes for Joint Image Understanding and Generation

Overview

This paper introduces Self-Correcting Coupled Markov Jump Processes (SC-CMJP) and the CO sampler, a modality-agnostic, training-free, single-pass inference algorithm for unified joint text-image generation using Masked Diffusion Models (MDMs) (2607.13188). The formulation captures the interplay between modalities typically neglected by conventional MDM samplers, enforcing stepwise cross-modal negotiation and immediate self-correction via remasking. The formulation is instantiated on three large-scale datasets—JEdit-1M (image editing and understanding), JMaze-200K (maze solving), and JNono-200K (nonogram reasoning)—and is shown to empirically outperform state-of-the-art samplers across all settings with monotonic improvement as the number of denoising steps increases.

Figure 1: CO in action: text and image trajectories co-evolve in image editing, maze, and nonogram tasks, converging to solutions through joint stepwise commitments and cross-modal self-correction.

Motivation and Problem Formulation

Autoregressive and standard parallel pipelines for multimodal generation suffer from sequential or weakly coupled update protocols. Specifically, current MDM-based samplers either:

1. Decode modalities in a fixed order, preventing immediate correction when a contradiction is detected.
2. Interleave updates without intra-step cross-modal feedback.
3. Fail to revisit previously committed tokens (no remasking).

This leads to persistent cross-modal contradictions—e.g., text describing an object not present in the image, or image edits unsupported by text. The need is clear for a single-pass stochastic process where modalities are tightly coupled and can self-correct.

The SC-CMJP framework recasts multimodal masked diffusion as a joint Markov jump process, where each modality’s transition rates adapt to the other's per-token confidence using cross-modal attention. Furthermore, a remasking ("death") jump is introduced so that tokens can be re-masked (and thus revised), driven by cross-modal evidence—crucially, in a training-free manner within the standard MDM backbone.

The SC-CMJP and CO Sampler Algorithm

Theoretical Structure

• Self-Correcting Coupled MJP: Each modality’s birth (unmask) and death (remask) rates are functionals of the self-confidence of the other, weighted through attention. This makes each update mutually responsive rather than conditionally independent.
• Remasking: Based on the generalization of ReMDM, death (remask) steps are triggered by low cross-modal confidence, allowing for continuous revision as the denoising trajectory advances.
• Cross-modal Attention and Gated Negotiation: The model’s hidden states for each modality are used to extract cross-modal attention matrices, which in turn blend self-confidence with incoming evidence from the other branch conditioned on local uncertainty.

  Figure 2: CO sampler schematic: self-confidence in both modalities and attention-based cross-modal coupling drive parallel birth-death updates in a single denoising step.

Implementation

• Single Forward Pass: Unlike naively coupled systems requiring repeated backbone evaluation, CO achieves coupling via a single forward pass per step.
• Chain-Rule Decomposition: Text updates depend only on the current state, while image updates blend self and text-side confidence, and are additionally modulated by an entropy-based gate determining when to trust cross-modal signals.
• Score Normalization: Different vocabulary sizes for text and image are normalized using percentile ranks to allow effective gating and blending.
• Birth-Death Protocol: At each step, the lowest-scoring committed tokens are remasked, and the top-scoring masked tokens are unmasked, with quotas driven by the diffusion schedule.

Datasets, Tasks, and Benchmarks

To evaluate SC-CMJP and CO, the authors curate three benchmarks, each with a large training split and rigorous in-/out-of-distribution (OOD) test splits. All conform to a joint schema for parallel multimodal supervision. The data curation process involves extensive annotation and trace generation.

Figure 3: Dataset curation pipeline for JEdit-1M, JMaze-200K, and JNono-200K, with annotation via strong MLLMs and algorithmic oracles for reasoning datasets.

• JEdit-1M: Extends ImgEditBench to include detailed scene graphs and logic-based reasoning traces, enabling both grounded editing and concurrent understanding. mAP@0.5:0.95 is used for evaluation.
• JMaze-200K: Each maze instance contains ground-truth path (as an explicit coordinate sequence) and a solution rendered as an overlay.
• JNono-200K: Each sample is a nonogram with clues and both textual and image solutions, with a holistic constraint-propagation reasoning trace.

  Figure 4: Example JEdit-1M sample: prompt, source/target images, scene graph annotations, and thinking trace, all aligned for joint training.

Experimental Results and Analysis

Image Editing and Grounded Understanding

• The CO sampler achieves the highest joint mAP (0.369 overall) and target-mAP, outperforming even ground-truth-grounded Qwen3-VL-8B, illustrating effective concurrent generation and understanding.
• Numeric result: The overall ImgEditBench score rises monotonically from 1.72 to 1.93 as the number of denoising steps increases, in contrast to baselines that plateau or degrade.

  Figure 5: CO uniquely shows monotonic improvement in both image-editing fidelity (left) and overall joint mAP (right) as the number of diffusion steps grows.

Visual Reasoning: Maze and Nonogram Solving

• CO uniformly leads in joint text+image accuracy on in-distribution and OOD splits. For nonogram OOD, the relative gap to baselines grows wider, indicating robust generalization.
• Qualitative results confirm that CO aligns text explanations and image outputs, ensuring, for instance, Nonogram solutions where all clues are satisfied (whereas baselines drift).

  Figure 6: Qualitative maze and nonogram comparisons—only CO’s joint trajectory satisfies all constraints concurrently; baselines violate alignment or clues.

Ablation and Mechanism Analysis

• Removal of shared rank normalization, entropy gating, or remasking significantly harms joint accuracy and mAP.
• Entropy-driven gates adjust reliance on modal confidences as uncertainty shifts during the trajectory.
• Cross-modal remasking events (death jumps) cluster in regions of low joint support, but also serve to revoke overconfident local commits contradicted by the other modality.

  Figure 7: (a) Gating $\lambda$ rises as text commits, shifting image’s trust to text-side evidence; (b) remasking events cluster where self and cross signals are low, with a smaller hotspot for cross-modal conflict resolution.

Qualitative Illustrations

• Image Editing: The model localizes edits per grounded scene graph, transferring only the necessary modifications to the image and preserving context.
• Maze Solving: The cross-modal trace allows the path rendered in the image to exactly match the coordinate sequence committed in text.
• Nonogram Reasoning: CO’s reasoning trace embodies bidirectional propagation, and the image solution satisfies every constraint commensurate with the text-side answer.

  Figure 8: Joint image editing—plan and realized edit are congruent, localized, and supported by per-panel scene graph alignment.

  

  Figure 9: CO’s maze solution—global reasoning, explicit coordinate path in text, and image rendering are isomorphic.

  

  Figure 10: Nonogram solution—parallel bidirectional inference, text trace, cell-wise grid filling, and full clue satisfaction.

Implications and Future Directions

Practical: By breaking the separation between understanding and generation, this approach enables robust, consistent, and efficient joint multimodal models that are more interpretable (with explicit parallel reasoning traces) and in principle more reliable for tasks requiring cross-domain verification.

Theoretical: SC-CMJP exposes a flexible framework for controlled self-correction, applicable beyond image/text pairs. It suggests new directions for non-autoregressive, high-fidelity joint modeling, especially relevant for settings with strong cross-modal priors or strict logical requirements (e.g., visual programming, grounded robotics).

Tractability: The ability to achieve stepwise cross-modal coupling in a training-free, single-pass protocol greatly facilitates deployment and scaling.

Future Extensions: SC-CMJP is modality-agnostic—applications to video, audio, or code (with joint source/diagram/documentation generation/understanding) are direct extensions. Further, theoretical investigation into the optimality and convergence of joint coupled diffusion in high-dimensional, multi-modal spaces is warranted.

Conclusion

This paper establishes SC-CMJP and the CO sampler as a significant advance in unified multimodal masked diffusion. By coupling modalities at each denoising step, enabling self-correction via remasking, and maintaining efficiency through single-pass inference, CO sets a new empirical standard for concurrent image understanding and generation, with robust cross-modal consistency and tractable sampling scaling. The formulation bridges non-autoregressive diffusion with the operational patterns observed in human explanation and revision, motivating further research toward generalized, modality-agnostic joint generation frameworks.