POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs

Abstract: Multimodal LLMs (MLLMs) have recently demonstrated remarkable capabilities in cross-modal understanding and generation. However, the rapid growth of visual token sequences--especially in long-video and streaming scenarios--poses a major challenge to their scalability and real-world deployment. Thus, we introduce POINTS-Long, a native dual-mode MLLM featuring dynamic visual token scaling inspired by the human visual system. The model supports two complementary perception modes: focus mode and standby mode, enabling users to dynamically trade off efficiency and accuracy during inference. On fine-grained visual tasks, the focus mode retains the optimal performance, while on long-form general visual understanding, the standby mode retains 97.7-99.7% of the original accuracy using only 1/40-1/10th of the visual tokens. Moreover, POINTS-Long natively supports streaming visual understanding via a dynamically detachable KV-cache design, allowing efficient maintenance of ultra-long visual memory. Our work provides new insights into the design of future MLLMs and lays the foundation for adaptive and efficient long-form visual understanding.

• The paper introduces a dual-mode architecture with Focus and Standby modes, enabling high-fidelity reasoning alongside scalable visual token compression.
• It employs a two-stage post-training strategy to maintain fine-grained image performance while integrating efficient temporal modeling for long video streams.
• Experimental results show up to a 40× increase in effective video context and a 6.2× generation speedup with minimal accuracy loss.

POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs

Introduction

The introduction of POINTS-Long addresses fundamental scalability and efficiency limitations observed in Multimodal LLMs (MLLMs) when handling long-duration and high-frame-rate visual streams. Existing video MLLMs inherently suffer from quadratic growth in compute and memory demands due to visual tokenization, severely constraining their deployment in real-world, long-form visual comprehension scenarios. While previous work on visual token compression exists, pragmatic limitations—including compression efficacy, architectural rigidity, and incompatibility with modern inference frameworks—prevent widespread adoption.

POINTS-Long establishes a new paradigm, inspired by principles of human visual cognition, integrating a dynamic dual-mode visual processing mechanism within MLLMs. Specifically, it introduces "Focus Mode," which preserves high-fidelity reasoning, and "Standby Mode," which provides dramatically compressed representations optimized for long-term and streaming inference.

Figure 1: The POINTS-Long design is inspired by adaptive human perception, introducing Focus and Standby modes for a flexible trade-off between accuracy and efficiency.

Architecture and Training

Dual-Mode Visual Processing

At the core of POINTS-Long is a native dual-path modification of a strong multilingual MLLM backbone (POINTS1.5-8B-Instruct), which itself builds on Qwen2/3 family architectures. The model appends a set of learnable tokens (standby tokens) to the patchified visual sequence. These tokens, significantly fewer than the original sequence, are processed by duplicated, learnable MLP layers parallel to the main vision trunk, enabling independent refinement and aggregation.

To allow efficient, scalable video understanding, a temporal modeling block—implemented as a causal temporal attention mechanism—is introduced in the last layers of the vision transformer. This facilitates spatio-temporal information distillation, ensuring that frame-level redundancy is effectively compressed without loss of causal structure needed for streaming.

A key architectural innovation is an asymmetric attention mask: original (focus) tokens attend only to each other, while standby tokens can aggregate globally. This achieves full backward compatibility and deployability in frameworks supporting FlashAttention and other efficient inference primitives.

Figure 2: POINTS-Long architecture supports dual visual token paths, temporal modeling for streaming, and a two-stage post-training to integrate Standby mode without loss in Focus fidelity.

Two-Stage Post-Training Strategy

Integration of Standby mode is accomplished via a carefully controlled two-stage adaptation protocol:

• Stage 1 (Visual Distillation/Alignment): The base model is frozen; only newly introduced parameters (learnable tokens, duplicated MLPs, temporal attention) are updated to distill the information from the original sequence to the standby tokens.
• Stage 2 (Mode Adaptation): The LLM is fine-tuned with both Focus (full sequence) and Standby (compressed) inputs via a two-pass training step, ensuring preservation of the original fine-grained reasoning capacity.

This strategy precludes destructive interference between adaptation and previously acquired capabilities, enforcing loss preservation in both operational regimes.

Streaming and Long Video Understanding

One of the principal limitations of extant MLLMs is a fixed, short visual memory window constrained by context length. POINTS-Long mitigates this via mode-switching and a detachable Key-Value (KV) cache mechanism. In streaming inference, recent frames are maintained in high fidelity (Focus), while longer-term context is stored as a compact memory bank (Standby).

Figure 3: Standard MLLMs discard old visual KV-caches as limits are reached, whereas POINTS-Long migrates compact standby tokens into a growing memory bank, emulating hierarchical memory.

Empirically, this hybrid memory strategy allows POINTS-Long to extend effective video context by up to 40× with minimal loss, enabling, for instance, 6 seconds of high-fidelity working memory and up to 30 minutes of compressed recall within a standard 32k token context.

Experimental Results

POINTS-Long demonstrates robust empirical advances over both the baseline and existing compression and streaming models:

• Compression/Accuracy Tradeoff: Standby Mode achieves 97.7–99.7% of baseline SOTA accuracy on the OpenCompass video benchmark with only 2.5–10% of visual tokens, enabling high-throughput inference. (Results in Table 1 of main manuscript.)
• Scalability: POINTS-Long maintains or improves accuracy when the number of input frames increases, whereas standard architectures plateau or degrade. This suggests enhanced long-context modeling and immunity from long-range attention decay typical of LLMs.
• Fine-Grained Image Performance: Focus Mode performance is strictly preserved post-adaptation, with no degradation on high-level image benchmarks.
• Streaming Recall: In long video streaming VQA, POINTS-Long outperforms baselines by leveraging a dynamic memory bank, with the sliding window limitation alleviated.

POINTS-Long is compatible with modern inference frameworks (e.g., SGLang, vLLM), realizing practical throughput benefits: up to 6.2× generation speedup via increased batch parallelism and drastically lower LLM-side compute requirements.

Visualization and Analysis

Visualization of attention in the final ViT layer reveals that standby tokens learn a strong spatially local aggregation bias, further validating the distillation design.

Figure 4: Standby token attention maps cluster around spatial neighborhoods, facilitating effective local-to-global information distillation.

Ablation studies show that all architectural components—including parallel MLPs, temporal modeling, and two-stage adaptation—are essential for achieving optimal compression and accuracy.

Analysis of failure cases reveals that Standby Mode can underperform in spatially fine-grained tasks, while the baseline model is more likely to fail in temporal or holistic understanding—consistent with the intended design trade-offs.

Figure 5: Qualitative comparison: Standby mode has residual failures in spatial/fine-grained tasks, baseline fails in long-range coherence.

Implications and Future Directions

POINTS-Long demonstrates that a human-inspired, dual-mode architecture enables decoupling of the long-standing accuracy-efficiency trade-off in visual MLLMs. The methodology allows a single model to function as both a high-fidelity image reasoner and a scalable, long-context video engine. The native support for streaming and efficient memory management sets a foundation for industrial deployment, particularly in resource-constrained or high-throughput environments.

Theoretically, the architectural pattern prompts new inquiries: dynamic mode selection or adaptive video "thinking" strategies, where models autonomously allocate perception and reasoning resources depending on the task or input content. Future extensions may incorporate learnable switching policies (rather than pre-defined windows), or leverage the Standby pathway for scalable curriculum learning in temporally rich datasets.

Conclusion

POINTS-Long introduces a principled, deployable solution for adaptive visual processing in MLLMs. Through dual-mode design and robust post-training strategies, it achieves state-of-the-art visual compression efficacy, maintains full-fidelity reasoning where needed, and substantially enhances streaming and long-term video understanding. By reconciling efficiency and accuracy in a unified framework, POINTS-Long paves the way for future AI systems capable of dynamic, human-like perception and memory over visual streams.