FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

Abstract: Deploying Vision Transformer (ViT) models on edge platforms remains challenging due to their high computational demands and the architectural heterogeneity of modern hybrid ViT models, which incorporate both fully connected and convolutional layers. This heterogeneity leads to significant variation in tensor shapes, requiring flexible and efficient FPGA-based acceleration. In this paper, we present FlexViT, a reconfigurable FPGA accelerator for efficient ViT inference on resource-constrained edge devices. Built on the SECDA-TFLite framework, FlexViT employs a hardware-software co-design approach that maps both fully connected and convolutional layers onto a unified high-throughput INT8 GEMM engine using a runtime im2col transformation. To efficiently support diverse layer configurations, we propose a dual-mode dataflow that dynamically switches between input and weight reuse by reconfiguring the compute array at runtime. We further introduce a depth-first tiling strategy that completes accumulation in a single pass, eliminating off-chip partial-sum transfers and reducing memory bandwidth requirements. We implement FlexViT on a PYNQ-Z2 FPGA and evaluate it across a representative set of ViT models. FlexViT achieves up to 2.74x speedup on accelerator-executed layers, translating into up to 1.40x end-to-end speedup compared to CPU-only execution. The code is available at: https://github.com/gicLAB/FlexViT

• The paper presents a unified FPGA accelerator that flexibly supports both standard and hybrid Vision Transformer models by mapping all layers onto a common GEMM engine.
• It introduces a dual-mode dataflow with runtime im2col transformations that optimizes execution across fully-connected and convolutional layers for edge deployment.
• Empirical evaluation on a PYNQ-Z2 board shows up to 2.74× speedup and high resource utilization, proving its practical benefits in latency and energy efficiency.

FlexViT: A Flexible FPGA-based Accelerator for Edge Vision Transformers

Motivation and Problem Statement

Edge deployment of Vision Transformers (ViTs) presents substantial challenges due to their high computational complexity, heavy memory requirements, and growing architectural divergence. Recent hybrid variants combine convolutional and fully-connected layers, increasingly complicating hardware support for efficient inference on resource-constrained FPGAs. Existing accelerator designs generally optimize for either fully-connected or convolutional workloads, but lack the flexibility required to support the diverse spectrum of modern ViT architectures.

Unified Hardware-Software Co-Design

FlexViT leverages the SECDA-TFLite methodology to deliver a unified, reconfigurable FPGA accelerator for both standard and hybrid ViTs. The design pivots on a hardware-software co-design philosophy, mapping all computational layers—regardless of type—onto a common high-throughput GEMM engine. The framework utilizes runtime im2col transformations to linearize convolutional inputs, thus normalizing both FC and CONV layers into two-dimensional GEMMs. A dual-mode dataflow is introduced to dynamically switch between input and weight reuse depending on layer configuration and resource utilization.

Figure 1: Vision Transformer architecture, illustrating the tokenization of image patches and the subsequent multi-head attention and MLP processing.

Architectural Overview and Optimizations

The FlexViT hardware comprises six specialized units: scheduler, three read units, GEMM engine, and post-processing unit. Layer execution is controlled at tile granularity, with configurable per-layer mode (Input-Broadcast or Weight-Broadcast) determined by analytical and heuristic drivers. Single-pass accumulation is achieved through depth-first tiling, sized according to the constraints of the PYNQ-Z2 FPGA (BRAM, DSP, LUT). The GEMM engine is instantiated as three vector processing cores, leveraging both LUTs and DSPs via hybrid arithmetic split. Banking and cyclic partitioning strategies in the memory hierarchy further optimize bandwidth and parallel access.

Figure 2: FlexViT hardware accelerator showing the scheduler, read units, parallel GEMM cores, and post-processing pipeline.

Empirical Results: Latency, Energy, and Utilization

Evaluation on five quantized ViT models (ViT-T, DeiT-T, Swin-T, MobileViT-S, EfficientViT-b1) demonstrates up to 2.74× accelerator-layer speedup and 1.40× end-to-end speedup over CPU-only execution. The observed speedup is strongly correlated with the layer composition of the underlying models; standard ViTs with dense FC layers benefit more from hardware offloading, while hybrid models' fragmented execution graphs impose upper limits on achievable system-level gains (dictated by Amdahl’s law). FlexViT delivers consistent end-to-end latency improvements with negligible or positive energy implications, except for heterogeneous models where the dynamic power overhead of the FPGA fabric marginally exceeds CPU-only baseline.

Resource utilization is near saturation on PYNQ-Z2: 83.9% BRAM, 71.8% DSP, 69.3% LUT. This attests to an effective mapping and maximization of compute density under edge constraints.

Figure 3: FlexViT resource utilization on PYNQ-Z2 board, showing high-density mapping of BRAM, DSP, and LUT resources.

Comparative Analysis

FlexViT distinguishes itself from prior work in several domains:

• End-to-end verifiability: All metrics are measured on a deployed hardware platform, encompassing actual DRAM, AXI bus, and CPU-FPGA synchronization overheads.
• Resource efficiency under edge constraints: Competing designs often target large-scale FPGAs with less emphasis on practical edge deployment. FlexViT’s hardware footprint fits resource-constrained platforms.
• Architectural coverage: The unified GEMM-based design supports both FC and CONV layers, encompassing standard and hybrid ViT models without architectural modifications.

The design demonstrates superior flexibility compared to model-specialized accelerators and achieves practical edge deployment of modern Vision Transformers.

Practical and Theoretical Implications

FlexViT establishes a hardware architecture capable of supporting the evolving landscape of Vision Transformer models on edge devices. Practically, this paves the way for real-time, low-latency vision inference in cost- and power-sensitive deployments, such as surveillance, smart IoT, and autonomous systems. Theoretically, the flexible GEMM-based scheduling and depth-first tiling may inform dataflow designs for future accelerators targeting increasingly heterogeneous workloads.

Furthermore, the results emphasize the utility of hardware-software co-design strategies and runtime adaptiveness in overcoming layer and tensor shape variance—an insight relevant to the wider class of transformer-based architectures.

Future Directions

Potential future enhancements include exploring system-level optimizations to alleviate memory bottlenecks beyond layer-by-layer scheduling, scaling parallelism within tighter constraints, and extending support to broader classes of transformer variants. Investigating energy-proportional execution via dynamic power gating for non-compute phases could mitigate FPGA power overhead in hybrid models.

Conclusion

FlexViT demonstrates a flexible, resource-efficient FPGA accelerator architecture for Vision Transformer inference on edge platforms (2606.31938). It achieves strong throughput and latency improvements across both standard and hybrid ViT models by unifying layer execution under a GEMM-based compute paradigm, effectively addressing architectural heterogeneity, dimensional variance, and edge resource limitations. The system’s empirical results and deployability directly inform practical and theoretical approaches to edge AI and hardware accelerator co-design.