EMiX: Scalable Multi-FPGA Emulation

• EMiX is a scalable multi-FPGA emulation framework that partitions tiled many-core RISC-V designs along NoC boundaries to overcome single-FPGA resource limits.
• It employs a dual interconnect strategy using low-latency Aurora links and high-throughput 100 Gb Ethernet to balance local and global communication.
• Prototyping on AMD Alveo U55c boards demonstrated full Linux boot and OS-level access, validating its effectiveness in comprehensive system emulation.

Searching arXiv for the EMiX paper and closely related work on multi-FPGA emulation and OpenPiton. EMiX is a scalable multi-FPGA emulation framework for distributed emulation of multi-core RISC-V architectures beyond single-FPGA resource limits. It addresses a pre-silicon validation problem in which full-system emulation of modern many-core SoCs increasingly exceeds the LUT, FF, BRAM, URAM, DSP, I/O, and clocking capacity of a single FPGA. The framework targets tiled many-core RISC-V systems, specifically OpenPiton-style designs, and partitions a monolithic RTL implementation along natural boundaries at NoC edges between tiles, deploying the resulting components across multiple interconnected AMD Alveo U55c FPGAs without requiring fundamental RTL redesign (Kropotov et al., 29 Apr 2026). In the reported prototype, EMiX realizes a 64-core architecture across eight interconnected FPGAs, demonstrates full-system execution including Linux boot, and preserves OS-level access through chipset-hosted UART, memory, and Ethernet interfaces (Kropotov et al., 29 Apr 2026).

1. Problem setting and design target

FPGA-level emulation is a key stage in pre-silicon chip design validation because it enables OS boot and execution of real workloads before tape-out. The central difficulty addressed by EMiX is that many-core systems with large cache hierarchies and coherent interconnects often no longer fit on a single FPGA. This limitation is especially acute for tiled architectures in which each tile integrates a core, L1/L2 caches, and routers attached to multiple on-chip networks (Kropotov et al., 29 Apr 2026).

The target design class is an OpenPiton-style many-core RISC-V system, where tiles are interconnected by three wormhole-routed, 64-bit unidirectional NoCs arranged as a 2D mesh. A chip bridge connects this mesh to peripherals such as UART, memory, and Ethernet. Because the tile array can scale to hundreds of tiles, the monolithic RTL typically exceeds single-device capacity, making full-system emulation infeasible without partitioning (Kropotov et al., 29 Apr 2026). OpenPiton is the architectural lineage explicitly referenced by the EMiX work (Kropotov et al., 29 Apr 2026).

EMiX is motivated in part by limitations of prior academic multi-FPGA efforts, which are described as valuable but often constrained by inter-FPGA connectivity such as PCIe or cloud-specific fabrics. Industrial emulation platforms are described as sophisticated but closed and expensive. Within this context, EMiX is positioned as an open, accessible, commodity-hardware solution intended to preserve full-system software functionality (Kropotov et al., 29 Apr 2026).

2. Architectural principles and novelty

The defining architectural idea in EMiX is to partition a monolithic many-core tiled design along NoC edges between tiles. This preserves tile internals, including cores, caches, routers, and NoC packet semantics, and thereby avoids what the paper characterizes as fundamental RTL redesign. Coherence, memory ordering, and OS-visible devices therefore remain intact because inter-tile communication continues to follow the original NoC semantics, now carried across FPGA boundaries by bridges (Kropotov et al., 29 Apr 2026).

A central design feature is a dual interconnect strategy. Each Alveo U55c provides two QSFP interfaces, and EMiX assigns distinct roles to them. QSFP-1 is used for low-latency point-to-point Aurora links between adjacent FPGAs, while QSFP-0 is used for scalable 100 Gb Ethernet cross-connect via CMAC. This combination provides both direct low-latency neighbor communication and full-cluster reachability (Kropotov et al., 29 Apr 2026). The architecture thus separates latency-sensitive local traffic from globally routable traffic, a choice that the paper states reduces Ethernet traffic at runtime and simplifies CMAC timing at compile time (Kropotov et al., 29 Apr 2026).

Internally, both Aurora and Ethernet paths are exposed through a common AXI-Stream transport abstraction. AXI-Stream is used for multiplexing, channel mapping, clock-domain crossing, and demultiplexing of NoC channels that cross FPGA boundaries. This unification is significant because it isolates communication-specific complexity at the inter-FPGA boundary rather than distributing it throughout the tile RTL (Kropotov et al., 29 Apr 2026).

A further principle is full-system usability. The FPGA that hosts the chipset interfaces retains OS-level access through UART, memory, and Ethernet, and the prototype demonstrates Linux boot as well as normal networking, including ping and scp (Kropotov et al., 29 Apr 2026). This places EMiX in the category of system-capable emulation frameworks rather than isolated communication-only prototypes.

3. Partitioning methodology and formalization

EMiX partitions designs at tile boundaries along NoC edges. Each FPGA hosts one or more tiles, and the partition orientation is configurable, including vertical or horizontal cuts across the 2D mesh. The practical criteria identified for partitioning are minimizing the number and volume of NoC links crossing FPGA boundaries, balancing per-FPGA utilization against LUT, FF, BRAM, URAM, and DSP constraints, and simplifying CDC placement by isolating clock-domain crossings inside AXI-Stream bridges (Kropotov et al., 29 Apr 2026).

The paper does not provide an explicit ILP, but it presents a conceptual formalization of EMiX’s objectives. Let the mesh be $G = (V, E)$ with tiles $V$ and NoC edges $E$. Each tile $v$ has a resource vector

$$r(v) = [r_{\mathrm{LUT}}(v), r_{\mathrm{FF}}(v), r_{\mathrm{BRAM}}(v), r_{\mathrm{URAM}}(v), r_{\mathrm{DSP}}(v)],$$

and each FPGA $k$ has capacity

$$c(k) = [c_{\mathrm{LUT}}(k), c_{\mathrm{FF}}(k), c_{\mathrm{BRAM}}(k), c_{\mathrm{URAM}}(k), c_{\mathrm{DSP}}(k)].$$

If $x(v,k) \in \{0,1\}$ indicates assignment of tile $v$ to FPGA $k$, and $V$0 estimates traffic on NoC edge $V$1, the objective is cut minimization subject to resource constraints (Kropotov et al., 29 Apr 2026).

The stated objective is

$V$2

subject to the standard per-FPGA resource bounds, such as

$V$3

and analogous constraints for FF, BRAM, URAM, and DSP, together with

$V$4

This suggests a graph-partitioning view in which EMiX uses architectural regularity of tiled meshes as a practical substitute for invasive RTL decomposition. A plausible implication is that the approach is particularly well matched to NoC-based many-core systems whose module boundaries already align with communication edges.

In implementation practice, adjacent partition edges are mapped to Aurora point-to-point links to minimize latency, whereas more distant communication uses Ethernet via CMAC for scalable cross-connect. The FPGA that hosts the chipset maintains Ethernet connectivity so that OS-level access remains available (Kropotov et al., 29 Apr 2026).

4. Inter-FPGA transport, coherence preservation, and memory behavior

EMiX’s transport substrate is explicitly two-level. On the low-latency path, QSFP-1 carries AMD Aurora 64b/66b over GTY SerDes, and a custom NoC–Aurora bridge packetizes NoC traffic into Aurora frames. On the scalable path, QSFP-0 carries 100 Gb Ethernet through AMD 100 Gb CMAC IP, with a NoC–CMAC bridge transporting NoC packets over Ethernet. Ethernet frames include source and destination MAC addresses, and reliable delivery is achieved via standard Ethernet mechanisms and retransmission when necessary (Kropotov et al., 29 Apr 2026).

The common AXI-Stream layer abstracts these paths into a single transport interface used for multiplexing, channel mapping, CDC, and demultiplexing. The combination of direct neighbor Aurora links and any-to-any Ethernet connectivity is described as reducing load on the Ethernet fabric while simplifying CMAC-based timing and implementation (Kropotov et al., 29 Apr 2026).

The paper presents a conceptual throughput and latency model, although not a numerically instantiated one. Aurora effective throughput per link is expressed as

$V$5

while Ethernet throughput is approximated as

$V$6

End-to-end latency over a path $V$7 is decomposed as

$V$8

where switch latency vanishes for direct Aurora point-to-point traffic (Kropotov et al., 29 Apr 2026).

Coherence is preserved by design rather than reimplemented. OpenPiton’s coherence runs over its NoCs with wormhole routing, and EMiX keeps tile internals unchanged while bridging NoC packet flows across FPGA boundaries. Coherence requests, responses, and data packets therefore traverse the same NoC semantics as in the original RTL, with the bridges translating only between NoC packets and Aurora or CMAC frames without altering protocol behavior (Kropotov et al., 29 Apr 2026). This directly addresses a common misconception that distributed FPGA deployment necessarily requires redesign of coherence or memory-ordering mechanisms; in EMiX, the stated objective is precisely to avoid that redesign.

Memory and peripherals remain centralized in the reported prototype. The first FPGA hosts UART, HBM memory, and Ethernet. Remote tiles access those resources through NoC packets that traverse inter-FPGA links, while address mapping and coherence behavior remain those of the original RTL (Kropotov et al., 29 Apr 2026). This simplification preserves software visibility but also creates a potential convergence point for memory traffic. The paper explicitly notes this as a hotspot risk, partially mitigated by using Aurora for nearby traffic and Ethernet for global reachability (Kropotov et al., 29 Apr 2026).

5. Toolchain, implementation flow, and prototype realization

The implementation uses AMD AXI-Stream infrastructure, AMD Aurora 64b/66b, and AMD 100 Gb CMAC IP. The architecture narrative in the paper describes a four-stage flow: identify NoC partition boundaries across tiles; assign tiles to FPGAs while placing chipset interfaces on one FPGA; instantiate NoC–Aurora and NoC–CMAC bridges within a unified AXI-Stream transport that includes channel mapping and CDC; and then synthesize, place-and-route, and deploy on Alveo U55c hardware in the Makinote cluster (Kropotov et al., 29 Apr 2026).

The reported evaluation platform is a Makinote cluster with 96 Alveo U55c FPGAs, of which eight are used for the EMiX prototype. Those eight are configured as four Aurora-connected pairs interconnected via Ethernet, with eight cores per FPGA (Kropotov et al., 29 Apr 2026). The design is an OpenPiton many-core system with in-house RISC-V cores and three 64-bit unidirectional wormhole-routed NoCs. The software stack includes a bare-metal multicore memory test executed sequentially across cores and full Linux boot (Kropotov et al., 29 Apr 2026).

The first FPGA hosts UART, HBM, and Ethernet. Under Linux, the integrated 100 Gb Ethernet operates correctly, and the paper reports successful ping and scp tests for OS-level access (Kropotov et al., 29 Apr 2026). This makes the prototype not merely a structural partitioning demonstration but a verified full-system emulation environment.

The paper emphasizes OS-level support rather than internal debug tooling. Software visibility is provided through the chipset FPGA’s Ethernet and UART, while AXI-Stream and standardized IP blocks are said to help instrumentation (Kropotov et al., 29 Apr 2026). This suggests a deployment philosophy oriented toward end-to-end system validation rather than deep proprietary probe infrastructures.

6. Evaluation results, trade-offs, and limitations

The main reported quantitative results are concise but informative. The multi-FPGA EMiX prototype runs at 50 MHz. Linux boot time is approximately 15 minutes across 8 FPGAs, compared with about 5 minutes for a single-FPGA scenario at the same 50 MHz. Per-FPGA LUT utilization is approximately 55%, and the additional overhead is approximately 16%, attributed primarily to CMAC, Aurora, NoC–CMAC, and NoC–Aurora IPs (Kropotov et al., 29 Apr 2026).

Metric	Reported result
Prototype scale	64 cores across 8 FPGAs
Clock frequency	50 MHz
Linux boot time	Approximately 15 minutes
Single-FPGA comparison	About 5 minutes at the same 50 MHz
LUT utilization per FPGA	Approximately 55%
Communication overhead	Approximately 16%

These numbers reveal the principal system-level trade-off. EMiX extends emulation beyond the resource ceiling of one FPGA, but distributed communication introduces additional serialization, bridge latency, CDC overhead, and potentially Ethernet switch traversal, which lengthen end-to-end software execution time (Kropotov et al., 29 Apr 2026). The threefold increase in Linux boot time is the clearest empirical manifestation of this cost.

The paper identifies several bottlenecks and sensitivities. More inter-FPGA NoC cuts increase the volume of traffic subject to serialization and CDC. Centralized peripherals and memory simplify software access but can create hotspots. Communication IP overhead consumes fabric resources and may limit tile count per FPGA, although the reported LUT utilization leaves headroom (Kropotov et al., 29 Apr 2026). Partition orientation and the mapping of busy NoC edges to Aurora versus Ethernet materially influence latency and congestion. This suggests that EMiX’s scaling behavior is not solely a function of FPGA count; it also depends on how well partition geometry aligns with the traffic structure of the emulated SoC.

The reported limitations are explicit. The work does not provide maximum demonstrated scale beyond the 64-core, 8-FPGA prototype, nor detailed throughput or latency microbenchmarks. Reliance on Ethernet switches for cross-connect can introduce queuing, switch latency, and possible retransmissions. Excessive cross-FPGA traffic can saturate CMAC or Ethernet channels, induce back-pressure, and complicate timing closure. Mismanaged CDC or AXI-Stream channel mapping can cause data loss or deadlock, with mitigation attributed to standardized IPs and retransmission on Ethernet (Kropotov et al., 29 Apr 2026).

7. Position within the emulation landscape and prospective directions

EMiX is presented as an open alternative to two categories of existing systems. Relative to prior academic frameworks, it uses commodity FPGA boards’ native QSFP interfaces rather than fabrics described as PCIe- or cloud-constrained. Relative to industrial platforms such as HAPS, Veloce, and Palladium, it offers an open approach on accessible Alveo U55c hardware, though without claims of equivalent automation or performance (Kropotov et al., 29 Apr 2026). Its principal distinguishing features are the dual-channel interconnect, the unified AXI-Stream abstraction, and demonstrated OS-capable full-system execution on commodity hardware (Kropotov et al., 29 Apr 2026).

The paper does not claim automated partitioning, and this is an important boundary condition for interpreting the contribution. Partition selection still requires manual reasoning about NoC cuts, traffic, and resource balance (Kropotov et al., 29 Apr 2026). A plausible implication is that EMiX currently occupies a middle ground: more accessible than closed industrial emulators, but less automated than them.

Future work is stated as “comprehensive performance evaluation of scalability and performance overheads,” indicating that deeper measurement and optimization are expected (Kropotov et al., 29 Apr 2026). The paper also states that EMiX “will be released as an open-source platform,” although it does not provide a repository URL, license, or release plan (Kropotov et al., 29 Apr 2026). The intended release is described as presumably including the NoC–Aurora and NoC–CMAC bridges, AXI-Stream plumbing, and deployment scripts.

For adaptation to other tiled designs, the paper’s practical guidance is specific: use natural architectural boundaries such as NoC link edges between tiles; align high-traffic neighboring cuts with Aurora point-to-point links; reserve Ethernet or CMAC links for longer-range communication; account for roughly 16% communication overhead in capacity planning; keep chipset functions on a designated FPGA to preserve OS-level access; and validate the system incrementally from bare-metal memory tests to Linux bring-up and networking verification (Kropotov et al., 29 Apr 2026). This guidance reflects the core methodological claim of EMiX: scaling beyond a single FPGA is feasible when inter-FPGA partitioning is treated as a transport problem at existing NoC boundaries rather than as an invitation to rewrite the RTL.