AutoNeural: Co-Designing Vision-Language Models for NPU Inference (2512.02924v1)

Abstract: While Neural Processing Units (NPUs) offer high theoretical efficiency for edge AI, state-of-the-art Vision--LLMs (VLMs) tailored for GPUs often falter on these substrates. We attribute this hardware-model mismatch to two primary factors: the quantization brittleness of Vision Transformers (ViTs) and the I/O-bound nature of autoregressive attention mechanisms, which fail to utilize the high arithmetic throughput of NPUs. To bridge this gap, we propose AutoNeural, an NPU-native VLM architecture co-designed for integer-only inference. We replace the standard ViT encoder with a MobileNetV5-style backbone utilizing depthwise separable convolutions, which ensures bounded activation distributions for stable INT4/8/16 quantization. Complementing this, our language backbone integrates State-Space Model (SSM) principles with Transformer layers, employing efficient gated convolutions to achieve linear-time complexity. This hybrid design eliminates the heavy memory I/O overhead of Key-Value caching during generation. Our approach delivers substantial efficiency gains, reducing quantization error of vision encoder by up to 7x and end-to-end latency by 14x compared to conventional baselines. The AutoNeural also delivers 3x decoding speed and 4x longer context window than the baseline. We validate these improvements via a real-world automotive case study on the Qualcomm SA8295P SoC, demonstrating real-time performance for cockpit applications. Our results highlight that rethinking model topology specifically for NPU constraints is a prerequisite for robust multi-modal edge intelligence.

• The paper introduces a co-designed architecture that integrates a MobileNet-based vision encoder with a hybrid SSM-transformer language backbone to optimize NPU inference.
• The methodology achieves up to 7× reduction in quantization error and 14× latency improvement over ViT-transformer models by deploying integer-only operations.
• The system is validated on real automotive tasks, demonstrating enhanced throughput, longer context windows, and robust quantization performance on edge NPUs.

AutoNeural: NPU-Native Vision-LLM Design for Efficient Edge Inference

Introduction

AutoNeural targets the inefficiencies of state-of-the-art Vision-LLMs (VLMs) when deployed on Neural Processing Units (NPUs) by advocating a fundamental architectural co-design. Conventional VLMs, typically based on Vision Transformers (ViTs) and transformer-based LLMs, are optimized for floating-point GPU computation, leading to substantial accuracy degradation and high inference latency on NPUs, especially under INT4/8/16 quantization. AutoNeural proposes a hybrid topology: a MobileNet-based vision encoder and a hybrid Transformer-State Space Model (SSM) language backbone, specifically engineered for robust, low-latency integer-only inference on edge NPUs. The system is evaluated in the context of real-time automotive applications, emphasizing design principles that prioritize both quantization stability and memory efficiency.

Architecture and NPU-Centric Innovations

AutoNeural replaces the ViT encoder with a MobileNetV5-style backbone, leveraging depthwise separable convolutions and bounded activations, yielding robustness under INT4/8/16 quantization. This architectural choice eliminates the quadratic compute and quantization brittleness typical of ViTs. The MobileNet encoder operates at an input resolution of $768 \times 768$ and produces 256 high-fidelity visual tokens through a Multi-Scale Fusion Adapter (MSFA).

The language backbone employs a hybrid arrangement: it alternates between gated-convolution sequence modeling layers based on liquid SSM principles and standard transformer self-attention layers—10 gated-convolution and 6 self-attention across 16 total layers in the presented 1.2B parameter Liquid AI model. This topology enables linear-time decoding, eliminates the Key-Value cache overhead (a dominant I/O bottleneck in transformer generation), and significantly reduces quantization error propagation through constrained memory footprints.

Figure 1: Architecture overview of AutoNeural, showing MobileNet-based vision encoding with MSFA, an MLP connector, and a hybrid 16-layer Liquid AI backbone.

A lightweight, two-layer MLP (sans normalization) aligns vision output features with LLM embeddings, exploiting NPU-friendly per-tensor quantization. The absence of normalization layers in the connector further mitigates quantization calibration drift.

Benchmark Results and Latency-Accuracy Trade-off

The robustness of AutoNeural under aggressive quantization is confirmed through extensive benchmarking on the Qualcomm SA8295P NPU. The MobileNet-based vision encoder exhibits up to 7$\times$ reduction in quantization error versus ViT-based encoders and maintains accuracy even under W8A16 quantization.

End-to-end system latency is reduced by up to 14$\times$ compared to state-of-the-art ViT-Transformer baselines. In automotive cockpit tasks, this translates into real-time throughput and enables larger image resolutions (up to $768 \times 768$) unattainable by ViT-based models due to NPU memory constraints.

Figure 2: On-device performance and latency plots for AutoNeural showing stable accuracy under mixed-precision quantization (W8A16/W4A16).

Latency profiling on real hardware shows that at $512 \times 512$ resolution, MobileNet-based encoders achieve inference in 101.7 ms (vs. 1.4 s with ViT-300M), and are the only approach capable of real-time operation at higher resolutions on NPU. The hybrid language backbone achieves 2.9$\times$ decode throughput improvement and supports context windows 4$\times$ longer than transformer baselines, directly benefiting user-perceived responsiveness in interactive VLM applications.

Figure 3: Vision encoder latency scaling with input resolution, illustrating orders-of-magnitude efficiency gaps between MobileNet and ViT architectures on NPU.

Task Performance and Automotive Dataset Integration

AutoNeural’s efficacy is verified on five representative multimodal benchmarks (MMStar, HallusionBench, MathVista_MINI, AI2D_TEST, OCRBench), where it outperforms same-size InternVL2-1B and approaches 2B-3B scale models, despite operating with fewer parameters and an NPU-centric topology. Notably, ablation studies indicate that the MobileNet encoder not only matches the accuracy of state-of-the-art ViTs but delivers compliant quantization robustness and dramatic latency reduction.

A custom, in-domain automotive dataset (200k samples) further tailors the model to tasks such as driver/passenger monitoring, vehicle security, parking localization, and safety event identification. Domain-specific quantization-aware fine-tuning and balanced synthetic data integration ensure high annotation fidelity and robustness across demographic and environment diversity inherent to real-world cockpit deployments.

Theoretical and Practical Implications

This model substantiates that architectural choices made to satisfy floating-point GPU throughput objectives are largely orthogonal or detrimental to edge inference on NPUs. By rooting both vision and language pathways in operator classes naturally amenable to quantized, memory-efficient execution, AutoNeural demonstrates that accuracy-latency trade-offs can be materially reshaped—contradicting the common assumption that quantized edge models must sacrifice capacity or input resolution for deployability.

The SSM-Transformer hybrid approach in language modeling is specifically shown to alleviate memory bandwidth as the primary system bottleneck and to generalize to longer context windows, hinting at further opportunities in NPU-native instruction tuning and compressed state management. The results advocate for a paradigm shift from post-hoc quantization and pruning to up-front, hardware-guided architecture search and co-design, for both vision and sequence modeling layers.

Outlook and Future Directions

While the empirical focus is on automotive intelligence, the principles delineated extend to any latency- and power-constrained multimodal application (e.g., mobile assistants, robotics, AR). Future directions highlighted include automated architecture optimization conditioned on target NPU operator/topology profiles, refinement of mixed-precision quantization strategies, and validation on additional edge NPU platforms to confirm generalizability.

Further, the adoption of linear-complexity state-space models mixed with shallow self-attention may become the normative language modeling baseline for edge-centric foundation models, replacing the pure transformer stack in this regime.

Conclusion

AutoNeural provides a concrete demonstration of the merit in holistic NPU-native vision-LLM co-design. Through MobileNet-based visual encoding, SSM-augmented language modeling, and avoidance of normalization-induced quantization brittleness, the architecture achieves substantial improvements in quantization error, end-to-end latency, throughput, and context capacity over transformer-centric baselines, all validated in real-world edge automotive scenarios. The work marks a decisive step toward practical, robust, and responsive multi-modal AI inference on next-generation edge NPUs.