FPGN: Redefining Ultra-Fast Programmable Gate-based Neural Acceleration with Differentiable LUTs

Abstract: Achieving nanosecond-scale inference latency for deep neural networks (DNNs) has become a primary architectural concern for latency-critical applications. While Field-Programmable Gate Arrays (FPGAs) offer a promising substrate for low-latency inference, conventional FPGA accelerators remain arithmetic-centric, using LUTs primarily as building blocks for numerical operators and peripheral logic. In contrast, recent LUT-native neural networks treat LUTs as learnable neurons, revealing promising theoretical potential to exploit their intrinsic logic expressivity. However, existing methods are largely confined to algorithmic optimizations, failing to translate this theoretical potential into high-performance FPGA accelerators. Specifically, their differentiable formulations do not faithfully match FPGA LUT primitives, their physically-unaware topologies compromise routability and timing closure, and their lack of automated optimization flow hinders systematic design space exploration (DSE) and efficient hardware implementation. In this paper, we propose FPGN, an end-to-end physically-aware framework that closes the gap between LUT-native learning and latency-optimized FPGA implementation. FPGN addresses these challenges through (i) a hardware-aligned differentiable formulation for training FPGA-native LUT neurons, (ii) a structured LUT-native topology with a streaming hardware architecture to improve routing locality and timing closure, and (iii) a latency-driven compiler that leverages high-fidelity analytical Quality of Results models to automate DSE and hardware generation. Experiments show that FPGN achieves up to 205$\times$ latency reduction compared to representative FPGA-based BNN accelerators and up to 30$\times$ higher LUT efficiency than prior differentiable LUT-native networks, while maintaining competitive inference accuracy.

• The paper introduces an end-to-end co-design framework that treats 6-input FPGA LUTs as learnable neurons, reducing each neuron’s parameters from 2^64 to 64 while achieving hardware-consistent training through temperature annealing, progressive binarization, and sparse-gradient fine-tuning.
• FPGN combines structured, locality-aware connectivity with fully streaming CNN and MLP architectures, eliminating BRAM and DSP use while improving routing, timing, and scalability through adaptive pipelining, LUT-native reductions, and circular line buffers.
• The paper reports 82.9% CIFAR-10 accuracy with deterministic 658 ns latency on a Versal Premium VP1902, delivering 205× lower latency, 75.53× higher throughput, 1.51× better FPS/LUT efficiency, and 7.31× improved energy efficiency than the FINN BNN baseline.‌

FPGN is an end-to-end co-design framework that treats FPGA look-up tables (LUTs) as learnable neurons rather than as primitives for synthesizing arithmetic operators, and couples this algorithmic shift with a physically-aware topology, a fully streaming hardware architecture, and an automated latency-driven compiler. The paper's central claim is that the gap between LUT-native learning and practical nanosecond-scale FPGA deployment can only be closed through full-stack co-optimization; the reported results support this with a 205× latency reduction and 75.53× throughput increase over the FINN BNN baseline on CIFAR-10, at 82.9% accuracy with a deterministic 658 ns inference latency on a Versal Premium VP1902 device.

Motivation and problem statement

The paper situates itself against three paradigms of neural computation on FPGAs. Arithmetic-centric accelerators such as FINN use LUTs to implement XNOR-popcount datapaths for binary neural networks (BNNs), confining each $k$-LUT to a negligible fraction of its $2^{2^k}$ Boolean functional space. The LUT-as-operator paradigm (LUTNet) replaces fixed XNOR gates with differentiable LUTs but inherits the BNN datapath, forcing learned functions to remain proximal to XNOR because a physical LUT produces only one bit while popcount aggregation requires multi-bit integers. The LUT-as-neuron paradigm (LTN, DLN, DWN) unifies weights and operators into learnable LUTs, but prior instantiations either rely on 2-input gates that underutilize modern 6-LUT fabric or scale poorly: enumerating all Boolean functions of a 6-LUT requires $2^{64} \approx 1.8\times10^{19}$ function-selection weights per neuron.

Three challenges structure the work. First, existing differentiable LUT formulations either lack gradient consistency (DWN's extended finite-difference heuristic introduces forward-backward mismatch) or are computationally intractable. Second, unstructured connectivity in prior LUT-native networks is physically incompatible with FPGA routing: profiling a DWN-style two-layer MLP shows that structured connectivity reduces half-perimeter wirelength by 3.5× and raises achievable clock frequency from 298 MHz to 526 MHz, with unstructured topologies degrading severely as LUT count grows. Third, no systematic design space exploration (DSE) methodology exists for DSP/memory-free LUT-native designs, where conventional analytical QoR models do not apply.

Hardware-aligned training

FPGN's training methodology addresses the first challenge with a relaxation that explicitly models the LUT address decoder. The discrete equality indicator $\mathbb{I}(\mathbf{x};\mathbf{u})$ — a product of $x_i$ or $(1-x_i)$ per bit position — is relaxed to real-valued inputs, yielding a continuous output that is a linear combination of all $2^k$ configuration bits weighted by these indicators. At binary inputs the formulation recovers exact hardware LUT behavior, and the gradient with respect to the active weight is exactly 1, resolving DWN's forward-backward inconsistency. Crucially, parameter complexity scales as $\mathcal{O}(2^k)$ rather than double-exponentially, reducing trainable parameters per 6-LUT neuron from $2^{64}$ to 64 entries.

Direct binary training remains unstable due to extreme gradient sparsity (only one of $2^k$ weights receives an update per sample). The paper therefore adopts a three-stage strategy: smooth continuous optimization via temperature-annealed sigmoid soft binarization, progressive binarization toward discrete boundaries, and final binary fine-tuning using the exact sparse gradients. A bimodal Gaussian initialization centered at $2^{2^k}$0 mitigates gradient attenuation from both multiplicative decay through the indicator products and sigmoid saturation; ablations show $2^{2^k}$1 causes instability during binarization while $2^{2^k}$2 hinders early convergence. Ablations confirm the necessity of the staged approach: vanilla binarization from scratch is unstable, and abrupt fine-tuning transition at epoch 40 triggers an irrecoverable loss spike, whereas the full strategy achieves the lowest final loss under identical topology.

Physically-aware topology and streaming architecture

To reconcile unconstrained connectivity with rigid routing fabric, FPGN enforces structural regularity at three levels. Micro-topologies comprise LUT-vectors (parallel $2^{2^k}$3-LUTs forming FC-style layers), LUT-trees (logarithmic-depth pyramids for binary reduction), and popcount units for integer reduction. In-order flattening makes algorithmic data ordering identical to physical interconnection, and locality-aware padding uses a two-stage mechanism — local repetition within partially-filled LUTs, then ordered replication across remaining LUTs — that simultaneously avoids idle logic and distributes fan-out pressure to prevent timing bottlenecks.

Macroscopically, the network follows a three-stage CNN-like hierarchy: a learned pixel-wise aggregation layer replacing quantization, residual blocks of LUT-native convolutions with popcount units (where the inherent $2^{2^k}$4-to-1 LUT mapping shrinks adder-tree depth from $2^{2^k}$5 to $2^{2^k}$6), and FC output stages. Batch normalization is fused into binarization as an integer comparison, eliminating floating-point routing and DSP usage entirely. The streaming architecture adds configurable spatial unrolling, adaptive pipelining based on logic depth and Carry-Chain stage counts, and a stationary-window circular line buffer that replaces high-fan-in MUX selection with hard-wired shifting patterns, keeping per-register fan-in constant.

Latency-driven compiler

The compiler builds high-fidelity analytical QoR models exploiting the deterministic mapping between hardware templates and FPGA primitives: timing models track cascaded-LUT depth and Carry-Chain stages to place pipeline registers; resource models account for kernels, arithmetic, buffer MUXes, and registers; and a layer-wise latency model decomposes end-to-end latency into start times and processing times. Because the resulting optimization is non-linear and non-convex, DSE is decomposed into an outer steepest-descent search over row-wise unroll factors and an inner MILP over column-wise factors formulated as candidate selection. Resource estimates correlate with Vivado post-PnR results at Pearson coefficients of 0.996 (LUT) and 0.999 (registers); the paper concedes that absolute post-PnR counts are unpredictable due to EDA optimizations, and handles this by calibrating with regression slopes — a defensible assumption given that relative ordering suffices for DSE. Solver efficiency is strong: optimizing FPGN-6-G takes 16 seconds versus days for brute-force search.

Evaluation

End-to-end results on CIFAR-10 and SVHN compare FPGN against re-implemented FINN baselines on the same VP1902 platform:

Method	Platform	$2^{2^k}$7	Acc.	Latency	FPS	FPS/LUT	kFPS/W
FINN	KU115	228 MHz	80.1%	194 µs	29.6k	0.59	8.97
FINN	VP1902	327 MHz	80.1%	135 µs	42.5k	0.86	4.89
LUTNet	KU115	200 MHz	84.8%	–	10.2k	0.10	1.89
FPGN	VP1902	205 MHz	82.9%	658 ns	3.21M	1.30	35.75

Against FINN, FPGN achieves 205× lower latency, 75.53× higher throughput, 1.51× higher LUT efficiency, and 7.31× better energy efficiency, using 2.46M LUTs, zero BRAM, and zero DSPs. Against LUTNet, it delivers 222× throughput speedup with 9× higher LUT efficiency on CIFAR-10 and 30× on SVHN, plus 34.8× energy-efficiency improvement. The paper argues the normalized FPS/LUT metric shows an architectural rather than merely parallelism-based advantage, implying FINN could not match FPGN even with equal LUT budgets. LUTNet's slightly higher accuracy (84.8%) relies on orthogonal heuristics excluded from the comparison; applying 4-bit residual binarization brings FPGN-6-L to 84.42% with a budget comparable to FPGN-6-G.

On MLP-style micro-scale tasks, FPGN matches or exceeds DWN accuracy with fewer LUTs and lower latency (e.g., 58.1% vs. 57.4% on CIFAR-10 MLP at 5.7 ns vs. 14.5 ns; 72.5% vs. 71.5% on KWS at 2.5 ns vs. 3.5 ns). Compared to PolyLUT, NeuraLUT, and AmigoLUT on JSC-CERNBox, FPGN uses fewer LUTs at comparable accuracy. NeuraLUT-Assemble retains superior resource efficiency on small networks (2.1 ns, 1780 LUTs on JSC-OpenML), but its latency grows to 5.7 ns on the larger CERNBox workload while FPGN holds 6.0→5.5 ns scaling behavior — evidence, per the authors, that structured connectivity preserves latency scalability as models grow. Topology ablations show depth scaling dominates width scaling for LUT-native architectures: FPGN-6-L beats FPGN-3-G by 0.57% accuracy with 30% fewer LUTs.

Limitations and open questions

The paper is explicit about several constraints. Accuracy remains bounded by the fully binary operating regime: 82.9% on CIFAR-10 versus ~86–88% for hybrid BNNs and >90% for INT8/INT16 implementations, which trade milliseconds of latency for precision; intermediate-precision configurations within the LUT-native paradigm are left unexplored. The evaluation covers CNN- and MLP-style architectures only; extension to Transformer linear operations (e.g., QKV projections) is proposed but not demonstrated. The compiler's fidelity depends on calibration slopes derived from target-specific synthesis, so portability across fabrics assumes stable correlation. Power figures mix vendor-reported values with Vivado estimates, introducing some comparability uncertainty. Finally, the claim that NeuraLUT-Assemble scales poorly rests on two data points across workload sizes.

Conclusion

FPGN demonstrates that treating the FPGA fabric itself as a trainable LUT-native neural substrate, when combined with physically-aware topology, streaming dataflow, and automated compilation, yields deterministic sub-microsecond inference with competitive accuracy and superior normalized efficiency relative to arithmetic-centric and operator-level paradigms. Its principal open problems are extending the framework beyond fully binary networks and beyond convolutional/MLP topologies, and validating the analytical-model approach across heterogeneous FPGA families.