MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

Abstract: VLA models have emerged as a powerful paradigm for transferring semantic knowledge from web-scale data to physical robotic control. However, current single-frame architectures suffer from intrinsic limitations: temporal myopia that discards historical dynamics, reasoning gaps between high-level instructions and low-level motor commands, and inference inefficiency due to autoregressive scalar decoding. In this work, we propose MIRTH, a unified framework designed to address these challenges. MIRTH augments a pretrained VLA backbone with three key innovations: (1) dual-scale temporal memory hubs that compress long-term scene evolution and short-term motion trends into compact embeddings; (2) latent reasoning tokens optimized via a mutual-information objective carving out a semantic plan space to align multimodal context with action trajectories; and (3) a parallel action decoding scheme that replaces autoregressive generation with vector-wise prediction to maximize control throughput. Extensive evaluations on the LIBERO simulation benchmark and a real-world LeRobot platform demonstrate that MIRTH achieves state-of-the-art performance and exhibiting emergent error recovery capabilities. The codes and collected datasets are released at http://github.com/kiva12138/mirth.

• The paper introduces a unified MIRTH framework that integrates dual-scale temporal memory hubs, mutual-information optimized latent tokens, and parallel action decoding for robust VLA control.
• It demonstrates significant performance gains on benchmarks, achieving up to 98.1% mean success rate in multi-stage tasks, confirming its superiority over SOTA models.
• The methodology offers actionable insights for overcoming temporal limitations and bridging semantic gaps, setting a new baseline for long-horizon robotic manipulation.

MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents

Introduction

"MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents" (2606.31167) introduces a unified architecture addressing fundamental design limitations in Vision-Language-Action (VLA) agents, specifically temporal myopia, reasoning bottlenecks between high-level instructions and low-level control, and the inefficiency of conventional autoregressive decoding for continuous control. The proposed MIRTH framework synergizes dual-scale temporal memory hubs, mutual-information-optimized latent reasoning tokens, and a fully parallelizable action decoding paradigm atop a pretrained VLA backbone to substantively advance multi-step, history-dependent robotic manipulation from complex, multimodal instructions.

MIRTH Architecture

Dual-Scale Temporal Memory Hubs

The temporal integration in MIRTH is realized through two specialized memory hubs: a long-term workspace memory and a short-horizon memory. The workspace hub maintains exponential moving averages (EMA) of scene embeddings across multiple timescales, efficiently compressing extended scene histories and adapting to the temporal granularity required to robustly track object states—even under full occlusion. The short-term hub aggregates recent motion cues using per-patch attention over a local observation queue, enabling the agent to attend dynamically to high-frequency events pertinent to task execution. These hubs are fused via a patchwise gating mechanism, supporting context-dependent mixing of slow and fast temporal features, while allowing deployment within the context window constraints of VLA architectures and without quadratic scaling of attention.

Figure 1: MIRTH overcomes temporal myopia of standard VLA models using dual-scale temporal memory hubs and reasoning tokens to effectively track objects through occlusion.

Latent Reasoning via Mutual Information Maximization

To bridge the semantic gap between multimodal context and action space, MIRTH introduces latent reasoning tokens situated between perceptual and action tokens in the input sequence. Unlike approaches relying on explicit textual annotation, these tokens are optimized to maximize mutual information with both the context and the action embedding using a symmetric InfoNCE loss. This design compels the latent codes to capture task-relevant plans aligned to both the semantic and dynamic facets of the current episode without the supervision overhead or ambiguity intrinsic to fine-grained action-language decomposition. The t-SNE analysis of the resulting embeddings demonstrates clear semantic clustering by task type.

Figure 2: The t-SNE projection reveals that MIRTH’s latent reasoning embeddings organize tasks into coherent semantic clusters, supporting structured plan formation.

Parallel Action Decoding

Contrasting the scalar autoregressive tokenization in canonical VLAs, MIRTH employs a global, vector-wise parallel decoding strategy. Each action step in a predicted chunk is represented by a dedicated token; all action vectors in a sequence are decoded via a single forward pass through a projection head. This architectural choice, empirically validated to improve training convergence without sacrificing control precision, enables real-time throughput crucial for embodied deployment while retaining seamless integration with transformer-style token interfaces.

Figure 3: The MIRTH pipeline integrates dual memory hubs, mutual-information-based reasoning tokens, and parallel (vector-wise) action decoding for efficient VLA control.

Empirical Evaluation

Simulation Benchmarks (LIBERO)

On the LIBERO benchmark suite, MIRTH achieves top-tier results, notably outperforming SOTA open-source models including Diffusion Policy, Octo, and OpenVLA across spatial, object, and especially long-horizon (LIBERO-long) manipulation suites. MIRTH demonstrates a mean success rate of 98.1%, exceeding all baselines, with a 95.3% success rate on the long-horizon suite—a marked improvement attributable to robust history integration and action plan consistency provided by the temporal and reasoning modules. Linear probing confirms that MIRTH’s encoded features are significantly more linearly predictive for both pose and velocity estimation versus single-frame models, verifying the effective encoding of temporally grounded dynamics.

Figure 4: On LeRobot, MIRTH decisively surpasses all baselines, especially in complex reasoning/multi-step tasks across five evaluation groups.

Real-World Robotic Deployment (LeRobot)

Evaluations on the LeRobot platform confirm MIRTH’s sim-to-real transferability and its emergent capabilities in error recovery and closed-loop re-planning. On complex, unseen test instructions requiring semantic generalization (e.g., abstract category grouping or multi-stage tool use), MIRTH maintains a high success rate and achieves an order of magnitude higher action throughput relative to heavier, less efficient decoding schemes. Error recovery analysis further reveals that MIRTH, owing to its structured reasoning representations, can detect and recover from execution failures at a far greater rate (12.1%) than single-frame or ablated variants devoid of explicit reasoning pathways.

Figure 5: Visualized rollouts on LeRobot and LIBERO confirm MIRTH’s robust multi-step execution and sim-to-real transfer consistency.

Analysis and Ablations

Component ablations indicate that both the workspace and short-horizon memory hubs provide additive gains for long-horizon tasks, confirming their complementary temporal roles. The latent reasoning tokens, under the mutual-information loss, substantially enhance both direct instruction following and error recovery; removing these components leads to significant performance degradation, emphasizing the necessity of structured semantic-action alignment.

Ablations on decoding paradigms demonstrate that the adopted global vectorwise projection method achieves faster convergence and better efficiency versus standard or compressed schemes; chunk size analysis reveals optimal trade-offs between computational throughput and control accuracy, with a 10-step chunk preferred for 10Hz closed-loop control.

Practical and Theoretical Implications

MIRTH’s approach establishes that compact, adaptive temporal summarization and information-theoretic latent reasoning are critical for long-horizon, multi-stage scene manipulation under multimodal instructions. The dual-hub architecture is highly transferable to other embodied agents requiring efficient, high-frequency reasoning over long and complex episodic interaction histories. Mutual-information-driven reasoning tokens provide a scalable alternative to costly and ambiguous step-by-step textual supervision, with emergent, interpretable clustering on diverse task families. Parallel decoding resolves the inference efficiency bottleneck that has long plagued transformer-based VLA controllers, enabling new classes of responsive, real-time robotic agents.

Future Directions

This work directly motivates further research into the extension of MIRTH to multi-arm and mobile manipulation scenarios, scaling mutual-information reasoning to even more abstract and compositional task decompositions, and integrating additional sensory modalities (e.g., tactile signals) for closed-loop interaction with occlusions or dynamic environments. Improving the interpretability of latent reasoning tokens—potentially through hybrid textual-latent objectives—remains an open challenge to facilitate safety and auditing in safety-critical applications.

Conclusion

MIRTH introduces a rigorous framework combining efficient temporal integration, unsupervised semantic reasoning, and high-throughput action decoding for vision-language-action agents. The demonstrated performance gains and emergent behaviors establish MIRTH as a compelling baseline for long-horizon, real-world semantic robot control, with design principles directly relevant to future research and deployment in robust embodied AI systems.