FlowTracer: AI Training Network Analyzer

Updated 4 July 2026

FlowTracer is a Python-based network analysis tool that reconstructs hop-by-hop paths of real application flows in RoCEv2-enabled AI training clusters.
It diagnoses network imbalance by computing a Flow Imbalance Metric that highlights uneven flow distribution due to ECMP hash collisions.
FlowTracer’s evaluations compare standard ECMP and static routing, demonstrating a 30% reduction in imbalance and enhanced throughput via parallel processing.

Searching arXiv for FlowTracer and closely related network tracing papers to ground the article in current literature. FlowTracer is a Python-based network analysis tool designed to reconstruct the exact hop-by-hop paths of real application flows and quantify how evenly those flows are distributed over the physical links of a data-center network. Its primary use case is the diagnosis of network path usage imbalance in AI training clusters, especially RoCEv2-enabled distributed LLM training environments, where Equal-Cost Multi-Path routing distributes traffic over parallel paths but hash collisions can still concentrate multiple elephant flows onto a subset of links, creating congestion and performance bottlenecks (Jamil et al., 2024).

1. Problem setting and operational motivation

Distributed AI training places sustained stress on folded Clos and leaf-spine fabrics because communication is dominated by long-lived, high-bandwidth flows. In the setting described for FlowTracer, communication between GPUs is typically implemented over RDMA over Converged Ethernet, and the relevant flows are elephant flows that can reach hundreds of Gbps per flow. Because these jobs are synchronization-critical, GPUs must wait at iteration boundaries until all communications complete; the slowest path therefore constrains job progress (Jamil et al., 2024).

FlowTracer addresses a specific failure mode of Equal-Cost Multi-Path routing. ECMP selects among equal-cost next hops by hashing flow header fields, often a 5-tuple or a subset. In practice, multiple large flows can hash to the same leaf–spine or spine–leaf link, while other equal-cost links remain lightly loaded or idle. The problem becomes more acute when VXLAN overlays reduce the entropy visible to the ECMP hash. In the environment studied, even a single collision can halve the effective per-flow throughput on a 100 Gbps link. FlowTracer is intended to expose this mismatch between nominal path diversity and actual path usage by passively tracing real workload traffic rather than injecting synthetic probes (Jamil et al., 2024).

2. Cluster environment and routing context

The case study uses a RoCEv2-enabled cluster with 16 servers and an 8-switch, full-bisection leaf-spine fabric. Each server has 56-core Intel Xeon CPUs, Ubuntu 22.04, 540 GiB RAM, and NVIDIA ConnectX-6 DX NICs. Each server has two dual-port 100 Gbps NICs, yielding an aggregate of 400 Gbps per server. The fabric consists of 4 leaf switches and 4 spine switches; leaf switches are 3.2 Tbps, spine switches are 1.6 Tbps, and all links run at 100 Gbps. The network has a 1:1 subscription ratio and full bisection bandwidth (Jamil et al., 2024).

Component	Configuration
Servers	16 hosts, 400 Gbps per server
NICs	NVIDIA ConnectX-6 DX, two dual-port 100 Gbps NICs
Switches	4 leaf and 4 spine
Link rate	100 Gbps
Topology	2-rack leaf-spine, full bisection bandwidth

This environment is representative of AI training networks in which multiple equal-cost paths exist between endpoints. Inter-rack traffic typically traverses about three hops, while intra-rack traffic remains within a rack. The paper evaluates FlowTracer on a bipartite cross-rack communication pattern chosen to stress the spine layer, which is precisely where ECMP collisions are most costly for RoCE workloads (Jamil et al., 2024).

3. Architecture and path reconstruction workflow

FlowTracer takes as input a workload description consisting of source-destination server pairs and the number of flows between each pair. A Workload Analyzer parses this description and passes it to a Process Controller, which partitions the set of source-destination pairs across processes. Within each process, the set of 5-tuples for a given pair, denoted $\tau_{s,d}$ , is further partitioned across threads as

$\{\tau_{s,d}^{(1)}, \tau_{s,d}^{(2)}, \dots, \tau_{s,d}^{(T)}\}, \quad \bigcup_{i=1}^{T} \tau_{s,d}^{(i)} = \tau_{s,d}.$

Each thread then traces the end-to-end paths of its assigned flows (Jamil et al., 2024).

Path discovery begins at the source host. For TCP and UDP traffic, FlowTracer uses Linux kernel utilities such as ss to extract 5-tuples and consults the kernel routing table to determine the egress interface for the first hop. For RoCEv2 traffic, kernel routing information is insufficient because RDMA bypasses the kernel’s TCP/IP stack, so FlowTracer instead interacts with the NIC driver to obtain RDMA flow information, and can also use sFlow on the leaf switches as a fallback. After determining the first hop, FlowTracer performs hop-by-hop reconstruction: at each switch it invokes the Arista ECMP hash visibility CLI with the flow’s 5-tuple and ingress interface, obtains the selected egress interface, maps that interface to the next physical link by consulting an operator-provided topology file, and repeats the procedure until the destination is reached. The output is an ordered per-flow path over host, leaf, and spine interfaces (Jamil et al., 2024).

4. Flow Imbalance Metric

The central quantitative output of FlowTracer is the Flow Imbalance Metric, a Mean Absolute Percentage Error over per-link flow counts. If $\text{actual\_flows}_i$ is the number of flows traversing link $i$ and there are $n$ links in the segment being analyzed, then the ideal uniform load is

$\text{ideal\_flows}_i = \frac{\sum_{j=1}^{n} \text{actual\_flows}_j}{n}.$

The metric is then

$\text{FIM} = \frac{1}{n} \sum_{i=1}^{n} \left| \frac{\text{actual\_flows}_i - \text{ideal\_flows}_i}{\text{ideal\_flows}_i} \right| \times 100.$

A value of $0$ indicates perfect balance; larger values indicate stronger path usage skew (Jamil et al., 2024).

FlowTracer computes this metric from reconstructed per-flow paths by counting how many flows traverse each link in a given layer, such as leaf-to-spine or spine-to-leaf links. In the RoCE experiments, there are 256 flows in total, and the ideal distribution corresponds to 4 flows per link, which is visualized in the paper as a red horizontal reference line. The metric is intentionally flow-count-based rather than byte-rate-based. This is well aligned with the evaluated workload because the RoCE flows are long-lived elephant flows of similar bandwidth. The paper also notes that if flow rates vary strongly, flow counts are only an approximation of actual load (Jamil et al., 2024).

5. Experimental results and operational use

FlowTracer is used to compare standard ECMP routing with a preprogrammed static routing configuration. Under standard ECMP, throughput across 16 server pairs is highly variable, and FlowTracer reports a FIM of 36.5%. Under static routing, throughput is near line rate across server pairs and the FIM drops to 6.2%. The example is presented as a 30% reduction in imbalance, and the corresponding bar charts show that ECMP produces severe overuse of some links and underuse of others, whereas static routing keeps per-link flow counts close to the ideal value of 4 (Jamil et al., 2024).

Routing configuration	FIM	Throughput characterization
Standard ECMP	36.5%	Highly variable across pairs
Static routing	6.2%	Near line rate across pairs

The tool is also evaluated for runtime scalability. Completion time grows with the number of traced flows, but decreases with thread-level parallelism; for 128 flows, 8 threads are about 2.6 times faster than 2 threads. Three SSH strategies are compared: baseline, persistent, and parallel+persistent. The baseline method, which repeatedly creates and closes SSH sessions, is the slowest; persistent SSH reduces overhead substantially, and the parallel+persistent method is the fastest. The study reports that FlowTracer can run in tens of seconds for realistic workloads, which places it in the category of offline or near-realtime debugging rather than continuous dataplane telemetry. Operationally, the intended workflow is iterative: identify a problematic AI workload, reconstruct real flow paths, quantify imbalance, change routing, and rerun FlowTracer to validate the new configuration (Jamil et al., 2024).

FlowTracer belongs to a broader family of flow-oriented measurement and tracing systems, but its function is narrower than generic flow monitoring and different from packet traceback. SENSOR, for example, is a distributed flow monitoring platform for collecting unsampled IPFIX from multiple strategic points in a university network using open-source components such as yaf, nfacctd, nfdump, SiLK, GoFlow2, Kafka, and Flowpipeline (Paradzik et al., 21 Jul 2025). Tracemax, by contrast, is a single-packet IP traceback strategy that reconstructs the exact path of a packet by appending compact router port identifiers to the IPv4 Option Field, targeting DDoS defense and packet-path forensics (Hillmann et al., 2020). Flowcorder moves in yet another direction by instrumenting end-host transport implementations with eBPF probes and exporting per-connection KPIs over IPFIX (Tilmans et al., 2019).

The name itself is not unique to data-center networking. In a separate line of work, “FlowTracer” denotes a reinforcement-learning framework for LLMs that traces answer-targeted reasoning flow on an attention-induced directed acyclic graph and uses token throughput to shape token-level rewards (Dong et al., 9 Jun 2026). This suggests that the term has become a generic label for systems that reconstruct and analyze flow through structured state spaces. In the networking literature covered here, however, FlowTracer most specifically denotes the path-reconstruction and imbalance-analysis tool for AI training clusters introduced in 2024, with emphasis on RoCEv2 traffic, ECMP behavior, and the Flow Imbalance Metric (Jamil et al., 2024).