ESS-Flow: FPGA Integration Framework
- ESS-Flow is a standardized FPGA development framework that separates board-specific elements from common infrastructure to streamline multi-board integration.
- It employs AXI4 interconnects and an automated, script-driven workflow to ensure reproducibility in design, testing, build, and documentation.
- The framework has been successfully used in ESS LLRF deployments on MTCA systems, demonstrating efficient, reliable integration for diverse applications.
ESS-Flow—Editor’s term—denotes the standardized FPGA development and integration flow built around the ESS FPGA Framework and its automated project workflow, especially as used to integrate the European Spallation Source (ESS) Low-Level Radio Frequency (LLRF) application onto FPGA boards in a Micro Telecommunications Computing Architecture (MTCA) crate. In the cited work, the framework exists because ESS FPGA applications, although algorithmically different, repeatedly require the same non-algorithmic infrastructure: communication with a crate CPU over PCIe, access to on-board memory, acquisition and storage paths, and configuration of peripherals such as ADCs, DACs, clocks, and related devices. ESS-Flow therefore functions as both a reusable FPGA integration architecture and a script-driven, reproducible build/test/documentation flow, with AXI4 chosen as the principal on-chip interconnect to support extensibility by IP blocks and seamless integration with Xilinx design tools (Amstutz et al., 2018).
1. Definition and design scope
ESS-Flow is a common platform for FPGA development at ESS, particularly for MTCA systems with a CPU in the crate. Its immediate purpose is to facilitate integration of different algorithms on different FPGA boards without reimplementing the same board-support logic, control/status register infrastructure, and peripheral handling in every project (Amstutz et al., 2018).
A central design goal is portability and reuse. The framework separates board-specific parts from common parts and defines clear interfaces both between those layers and between the framework and the user application. Project-specific changes are centralized in configuration files. This suggests that retargeting across boards is intended to occur by changing board support elements rather than rewriting the surrounding infrastructure. The framework is also positioned as a compromise between two undesirable extremes: board-vendor-only solutions on the one hand, and a full custom implementation of all components on the other. The stated approach is to rely on open standards “as far as possible,” especially AXI4, so that Xilinx-native and third-party IP can be reused inside the same integration shell.
A common misunderstanding is to treat ESS-Flow as the LLRF algorithm itself. In the cited implementation, the control algorithm is only the “custom logic.” The framework proper is the stable shell that supplies peripheral access, upstream communication, configuration services, and a common software model for the crate CPU and the ESS control system. The LLRF example demonstrates successful insertion of a domain-specific accelerator algorithm into that shell rather than redefining the shell around a single control loop.
2. Architectural decomposition
Architecturally, the framework is divided into three parts: the core framework, the data path, and the custom logic (Amstutz et al., 2018). The core framework is the central infrastructure layer controlling communication among FPGA blocks, external board components, and the upstream CPU. The data path handles high-speed, high-throughput sample movement. The custom logic is the application algorithm integrated into a wrapper interface.
The three framework functions identified in the paper define the operational boundary of ESS-Flow. First, the framework provides communication interfaces to peripherals, including access paths to devices such as ADCs and on-board memory, and in the data path also support for sinks such as DACs. Second, it provides upstream communication with the control system over PCIe, which is the path between FPGA board and crate CPU. Third, it provides configuration of on-board peripherals through an embedded controller.
The communication model is centered on AXI4 for two stated reasons: it is an open standard, and it integrates seamlessly with the Xilinx Vivado ecosystem, including Xilinx and third-party IP. Different AXI4 variants are assigned different roles. Full AXI4 is used where memory-mapped, higher-throughput transfers are needed, particularly for memory access. AXI4-Lite is used for register-bank access, control parameters, and status supervision. AXI4-Stream is identified as the intended future standardization for streaming, but the current data path instead uses a simpler protocol composed of data signals and a data-valid signal to maximize throughput and real-time behavior.
This decomposition implies a strict separation of concerns. Sample traffic, register/control traffic, board configuration traffic, and application logic are not conflated. ESS-Flow therefore resembles a reusable backplane architecture more than a monolithic application image: algorithm developers insert domain logic into a framework-defined slot while preserving board services and upstream software conventions.
3. Integration of custom logic and control/data planes
The main insertion point for an application in ESS-Flow is the wrapper around the “custom logic” (Amstutz et al., 2018). The wrapper exposes, at minimum, interfaces for memory read/write access and for data-stream inputs and outputs. Upstream control and register access are provided by the core framework, while high-rate sample data arrive from the data path and are sent onward to sinks such as DACs. In addition to streaming and memory interfaces, the custom logic instantiates its own register bank. That bank has its own register map, but it is attached to the same AXI4-Lite bus as the framework register bank or banks. Address partitioning is therefore transparent to the driver, and the design tools handle the mapping of multiple banks into the overall address map.
Data, control, and bulk transfer follow distinct paths. Sample data originate at the ADCs, pass into a modular pre-processing block, may optionally be written into on-board memory through a memory interface in that block, then are forwarded to the custom logic, and finally to a sink such as the DAC. Bulk data movement between host and board memory is provided through an AXI4 interface plus a DMA engine connected directly to the framework memory controller. An AXI4 interconnect combines and arbitrates memory accesses from different modules before they reach the memory controller. Low-bandwidth control and status communication use a separate AXI4-Lite path that gives CPU access to the framework and custom-logic register banks over PCIe.
Board configuration is handled separately from the crate CPU driver by an embedded MicroBlaze-based controller inside the FPGA image. That controller drives SPI and I2C and controls functions exposed via GPIO. The paper presents this as a software/hardware co-design decision: hardware-specific configuration behavior, including protocol variation among serial peripherals, is implemented in embedded software close to the hardware, while important board functions remain controllable from the CPU through the common register-bank mechanism. The result is that ESS-Flow connects an application not only to ADC/DAC data paths, but also to memory, PCIe control, and peripheral setup through predefined services.
4. Automated workflow and reproducible build process
A major part of ESS-Flow is the automated design workflow intended so that one developer can reproduce exactly the same result obtained by another developer earlier (Amstutz et al., 2018). GNU make is the top-level orchestrator. A developer invokes a target, and make dispatches project scripts written in bash, Tcl, and Python. Those scripts are version-controlled as part of the project and drive the underlying tools.
The first phase is automated testing using self-checking test benches at module, subsystem, and system level. The paper mentions the Vivado simulator, Mentor Graphics Questa, and Simulink as supported tools. Only if tests pass does the flow continue to implementation. Xilinx Vivado is then run in batch mode. Vivado regenerates required IP blocks and block designs from their configurations and executes synthesis, place-and-route, and bit-file generation. Report files are produced during this process, and a Vivado GUI project is also generated to support manual inspection. Vivado XSDK compiles the software for the MicroBlaze configuration controller. Finally, the FPGA bit-file and the MicroBlaze software are combined into a single firmware file loadable to the FPGA.
Documentation generation is built into the same flow. Doxygen extracts module documentation from HDL and software sources and generates HTML documentation. The workflow also distinguishes source-controlled inputs from generated artifacts: only regenerable Xilinx IP configuration files such as .xci should be kept under version control, while generated files such as Vivado reports, XSDK workspace content, and Doxygen output belong in dedicated generated-artifact folders. Planned Docker support is intended to freeze tool versions and environment variables as part of the reproducible development environment, and the heavy use of scripts and version-controlled automation is presented as central to compatibility with ESS continuous integration through Jenkins.
The significance of this workflow is organizational as much as technical. ESS-Flow standardizes not only how FPGA images are architected, but also how they are tested, built, packaged, and documented. That is especially relevant in a multi-board environment where reproducibility and maintainability are as important as timing closure.
5. LLRF deployment and implementation evidence
The framework was applied to the ESS LLRF system implemented on the FPGA of a Struck SIS8300-KU digitizer AMC within an MTCA crate (Amstutz et al., 2018). In the broader crate context, the cavity field controller resides on the SIS8300-KU FPGA, while the piezo controller for cavity tuning is implemented on a PEG-developed RTM controlled by a separate PEG FPGA AMC carrier. The framework therefore operates in a real multi-board MTCA environment rather than on an isolated development board.
Within the SIS8300-KU implementation, the current data path uses the ADCs as the source, sends samples through a non-board-specific pre-processing block, into the custom LLRF logic, and finally to a DAC sink. The pre-processing block is modular and may include filters, decimators, and near-IQ sampling blocks, with a compatible set of such blocks being part of the framework. This is the clearest statement of how ESS-Flow integrates signal-processing applications: common sample conditioning and optional memory capture are provided by the framework, while the control algorithm itself resides in the inserted custom logic.
The paper reports that a first version of the ESS FPGA Framework was developed with support for the Struck SIS8300-KU digitizer and that the framework was tested at 125 MHz and 200 MHz for the central AXI4 interconnect. For a Xilinx Kintex Ultrascale xcku040-ffa1156-c, with AXI4 data width: 512 bit and Xilinx Vivado 2017.1, the reported resource usage is as follows:
| Resource | Utilization | Fraction |
|---|---|---|
| LUT | 51463 / 242400 | 21.2% |
| Flip-Flop | 69973 / 484800 | 14.4% |
| BRAM | 173 / 600 | 28.8% |
| DSP blocks | 86 / 1920 | 4.5% |
| MMCM | 1 / 10 | 10.0% |
| PLL | 3 / 20 | 15.0% |
The authors characterize this as reasonable resource usage, with BRAM being the largest fraction. They further note that BRAM usage can be reduced significantly by removing on-board debugging capability for the configuration-controller software and optimizing the MicroBlaze-side code. The paper also states that “The LLRF control algorithms have been successfully integrated into the framework,” and that beam diagnostic applications at ESS were being ported to the framework as follow-on work. At the same time, the article does not provide a detailed control-loop block diagram, transfer function, cavity model, timing network description, latency budget, or explicit cavity field feedback equations. The LLRF example therefore demonstrates successful integration and deployment rather than algorithmic characterization of cavity control performance.
6. Software model, limitations, and broader significance
ESS-Flow has an explicit upstream software consequence: a common FPGA-side interface simplifies software development on the crate CPU and affects both the kernel driver and control-system access via EPICS at ESS (Amstutz et al., 2018). The base software can be shared across projects, with board- and project-specific parts added as modules. Although the paper does not name a specific driver or middleware package, it clearly describes a layered strategy in which common FPGA abstractions enable common host-side infrastructure.
Several limitations are equally explicit. The current internal streaming protocol is not yet AXI4-Stream, even though migration toward AXI4-Stream is identified as future work. The paper does not provide a concrete address map table, register-bank address range listing, or explicit DMA descriptor format; it only states that address mapping among multiple register banks is handled by the design tools and made transparent to the driver. It does not quantify closed-loop latency or LLRF algorithm performance. Docker support is presented as planned rather than implemented. These boundaries are important because they distinguish what ESS-Flow standardizes already from what remained prospective in the cited framework.
The broader significance of ESS-Flow lies in its standardization role. It turns repeated ESS FPGA engineering tasks—board access, peripheral configuration, PCIe communication, memory movement, software packaging, testing, and documentation—into reusable infrastructure. A plausible implication is that this reduces integration friction across accelerator applications that are algorithmically different but operationally similar. The framework is therefore not LLRF-only; the paper explicitly presents it as an ESS-wide FPGA platform, exemplified by the LLRF system but intended to support multiple applications across MTCA boards.