EdgeFM: Efficient Edge Inference for Vision-Language Models

Abstract: Vision-LLMs (VLMs) have demonstrated strong applicability in edge industrial applications, yet their deployment remains severely constrained by requirements for deterministic low latency and stable execution under resource limitations. Existing frameworks either rely on bloated general-purpose designs or force developers into opaque, hardware-specific closed-source ecosystems, leading to hardware lock-in limitation and poor cross-platform adaptability. Observing that modern AI agents can efficiently search and tune configurations to generate highly optimized low-level kernels for standard LLM operators, we propose EdgeFM, a lightweight, agent-driven VLM/LLM inference framework tailored for cross-platform industrial edge deployment. EdgeFM removes non-essential features to reduce single-request latency, and encapsulates agent-tuned kernel optimizations as a modular library of reusable skills. By allowing direct invocation of these skills rather than waiting for closed-source implementations, it effectively closes the performance gap long dominated by proprietary toolchains. The framework natively supports mainstream platforms including x86 and NVIDIA Orin SoCs, and represents the first end-to-end VLA deployment on the domestic Horizon Journey platform, enhancing cross-platform portability. In most cases, it yields clearly better inference performance than conventional vendor-specific toolchains, achieving up to 1.49 times speedup over TensorRT-Edge-LLM on the NVIDIA Orin platform. Experimental results show that EdgeFM delivers favorable end-to-end inference performance, providing an open-source, production-grade solution for diverse edge industrial scenarios.

• The paper presents a lightweight, agent-driven framework that achieves deterministic, low-latency inference on resource-constrained edge platforms.
• It employs modular design with shape-aware kernels, configurable KV cache pipelines, and hardware-tuned optimizations across x86, NVIDIA Orin, and Horizon Journey SoCs.
• Empirical results demonstrate significant latency improvements, such as up to 1.49× speedup and 34.35% decode latency reduction, validating its industrial applicability.

EdgeFM: Efficient Edge Inference for Vision-LLMs

Introduction and Motivation

EdgeFM proposes a lightweight, agent-driven framework explicitly designed for deploying vision-LLMs (VLMs) and LLMs in resource-constrained edge industrial environments. Unlike cloud-centric inference systems, edge scenarios prioritize deterministic, low-latency execution and stable performance under stringent memory and compute limitations. Existing solutions rely heavily on proprietary, vendor-specific toolchains, imposing ecosystem lock-in and diminished flexibility. EdgeFM circumvents these pitfalls by employing modular, agent-tuned kernel libraries, supporting mainstream hardware including x86, NVIDIA Orin, and domestic Horizon Journey SoCs. This approach delivers consistent performance advantages while remaining adaptable and open-source.

System Architecture and Design Principles

EdgeFM is architected with a thin runtime and heavy kernel design philosophy. The C++ runtime core orchestrates request dispatch and memory management, while operator backends—including CUDA-optimized attention and GEMM kernels—handle computational bottlenecks. The framework is configuration-driven, facilitating rapid profiling and deployment through single-JSON execution schemas. Core architectural components include:

• Scheduling Layer: Request dispatch and batch matching (single-request, deterministic).
• KVManager: KV cache pre-allocation, compression, and slot-based prefix reuse.
• Base Computing Layer: Shape-specific operator optimization, high-performance kernels, and vocabulary pruning.

  Figure 1: Layered architecture of EdgeFM with explicit operator decoupling and cache management, emphasizing edge-optimized scheduling and memory efficiency.

Prefill and decode phases are handled via two-phase execution. Shape- and stage-aware operator implementation tables enable fine-grained kernel selection, particularly important on edge devices with fixed tensor dimensionality.

KV Cache Pipelines and Optimizations

EdgeFM employs slot-based KV cache design. Fixed-prefix pre-solification and offline prefix computation yield constant-time cache reuse, minimizing prefill and decode latency. KV cache compression (e.g., SageAttention, FlashMLA) is configurable and separable, evaluated for memory savings and inference quality. CUDA Graphs and dedicated decode kernels further lower kernel launch overhead and improve end-to-end responsiveness.

Figure 2: EdgeFM serving pipeline illustrating end-to-end request processing and slot-based KV cache reuse for deterministic latency.

Operator Dispatch and Kernel Abstractions

The operator implementation table acts as a dispatch layer, regulating kernel selection over dimensions such as model class, hardware profile, operator kind, layer role, execution phase, and shape signature. Defaults are overridden by tuned entries for shape-specific and stage-specific execution; auto-tuning modules and external JSON overrides allow hardware-aligned optimization without modifying runtime code. This abstraction ensures maintainability and high adaptability across hardware fragmentation.

Speculative Decoding Integration

EdgeFM integrates EAGLE3-style speculative decoding. A lightweight draft model proposes blocks of candidate tokens, validated token-wise against the base model. KV cache management is unified, enabling speculative pipeline coexistence with graph acceleration and fixed-prefix cache reuse. This substantially improves throughput for decode-bound workloads.

Performance Evaluation

Cross-Platform Latency Analysis

EdgeFM demonstrates robust cross-platform performance on both x86-based server accelerators and embedded SoCs.

Figure 3: Cross-platform performance overview—(a) x86 latency decomposition, (b) latency gap comparison with vendor toolchains, (c) Horizon BPU deterministic prefill and ultra-low decode latency.

On the NVIDIA A800 x86 platform, EdgeFM achieves lower end-to-end latency than the official TensorRT-Edge-LLM baseline for Qwen2.5 and related models, maintaining stable throughput across short and long contexts. Linear scaling of latency with decode length is observed for both frameworks, but EdgeFM consistently yields better latency profiles.

On the NVIDIA Orin platform, EdgeFM delivers up to 1.49× speedup over TensorRT-Edge-LLM under 512-token prefill. The decode phase exhibits up to 34.35% reduction in latency, and prefill latency scales linearly with prompt length.

On the domestic Horizon Journey platform, EdgeFM enables the first full VLA deployment, supporting a SmolVLA-base model under native 5GB BPU memory constraints. Deterministic latency in prefill and ultra-low decoding latency support high-frequency control loops for embodied AI applications, validating cross-platform portability and efficiency.

Implementation Insights and Deployment Constraints

EdgeFM’s Horizon BPU deployment requires explicit stage partitioning and custom operator substitutions for numerical stability under INT16 quantization. Loop-wise calibration across denoising steps improves robustness for Flow Matching models, preventing trajectory divergence. The inference pipeline is designed for modular initialization, contextual encoding, iterative denoising, and action generation, maximizing cache reusability and hardware utilization.

Implications and Future Outlook

EdgeFM’s agent-driven, modular approach establishes a paradigm for lock-in-free industrial edge deployment of VLMs and LLMs. The findings demonstrate the viability of open-source frameworks in outperforming vendor-specific toolchains on diverse edge platforms. The deterministic, low-latency profile meets operational requirements for mission-critical applications, such as autonomous vehicles and robotics.

Going forward, further extension is planned for:

• End-to-end optimization for multimodal and speech-centric models
• Expanded support for full-range Horizon Journey chips with dedicated kernel adaptation
• Integration of dynamic inference scheduling for efficient multi-model concurrent deployments

The modular abstraction of operator dispatch and auto-tuning mechanisms provide fertile ground for future developments in edge AI, including on-the-fly configuration profiling and real-time scheduling.

Conclusion

EdgeFM provides a production-grade, open-source LLM/VLM inference stack tailored for industrial edge deployment, breaking free from proprietary hardware lock-in and delivering consistent latency improvements across platforms. The system’s layered architecture, agent-optimized kernel library, and unified cache management design enable deterministic, low-latency inference suitable for mission-critical edge workloads. The framework’s extensibility and adaptability serve as a foundation for future AI deployment paradigms focused on efficiency, portability, and operational independence.