TD-Link: A Daisy-Chain Optical Architecture for Integrated Data Readout and Deterministic Timing Distribution in Large-Scale Detector Systems

Abstract: TD-Link is a custom optical communication architecture that combines high-throughput data readout and sub-nanosecond timing synchronization over a single optical fiber for large-scale detector systems. The protocol adopts a multidrop daisy-chain ring topology connecting a Data Concentrator to up to sixteen FERS front-end boards per link, with up to eight independent links per concentrator. Operating at 3.125~Gb/s, TD-Link carries data, synchronization, and control traffic within the same serial stream through a token-based streaming protocol that minimizes per-hop latency and supports on-the-fly payload fragmentation. Transmitter lane alignment on the concentrator is achieved by exploiting the half-full condition of the multi-gigabit transceiver elastic buffer as a one-bit phase detector: a firmware finite-state machine iteratively adjusts the transmit phase interpolator until the FIFO write-to-read pointer difference reaches half-depth, locking each lane to a deterministic phase condition. A Digital Dual Mixer Time Difference (DDMTD) circuit is employed for inter-concentrator synchronization, measuring and compensating the phase offset between the recovered transceiver clock and the FPGA fabric reference clock. On the FERS boards, the recovered clock is cleaned by an external zero-delay PLL and retransmitted downstream, preserving phase coherence along the daisy chain. Experimental validation with CERN PicoTDC-equipped FERS boards demonstrates a board-to-board synchronization sigma of 7~ps for boards sharing a coaxial reference clock and below 28~ps for boards on independent concentrators. The results are stable across power cycles, confirming the robustness of the alignment strategy.

• The paper presents an optical daisy-chain architecture that integrates bidirectional data transport with sub-30 ps timing synchronization for large-scale detector systems.
• It details an FPGA-based deterministic phase alignment using elastic buffer flags and DDMTD circuits to achieve high-precision, scalable clock recovery across distributed boards.
• Empirical results show robust synchronization performance with time differences of 24–28 ps, enabling cost-effective, dense detector arrays with reduced cabling and infrastructure complexity.

Deterministic Timing and High-Throughput Data Acquisition with TD-Link in Large-Scale Detector Systems

System Overview and Motivation

Modern large-scale detector systems, especially those employing distributed front-end boards for photon, charge, or timing detection, must meet two stringent demands: high aggregate data bandwidth and sub-nanosecond timing synchronization among hundreds or thousands of distributed boards. Traditionally, satisfying both requirements involves deploying extensive cabling for parallel data and star networks for distributed clocking, leading to high complexity and cost. Solutions like White Rabbit achieve high-precision timing based on switched Ethernet but are suboptimal for compact, embedded detector arrays with dense board hierarchies.

The TD-Link optical architecture addresses these challenges with a custom protocol designed for the FERS-5200 detector readout ecosystem. Leveraging a multidrop daisy-chain ring topology at 3.125 Gb/s, TD-Link integrates bidirectional detector data, configuration, control commands, and timing synchronization over a single fiber per link. The combined use of FPGA-based deterministic lane alignment and all-digital phase measurement enables scalability to thousands of channels with sub-30 ps synchronization performance, while avoiding excessive infrastructure overhead.

Figure 1: Pictorial overview of a TD-Link system—eight optical ports per concentrator, each ring hosting up to sixteen FERS-5200 front-end boards for integrated data/readout and time distribution.

Master Clocking Architecture and Synchronization Mechanisms

The DT5215 Data Concentrator, based on a Xilinx Zynq UltraScale+ MPSoC, operates as the master node, distributing both clock and data across up to eight optical rings. Each ring, realized over standard SFP optics and multimode fiber, handles a daisy-chain of up to sixteen FERS boards. On the master, a low-jitter, multi-output PLL—with selectable references (internal, external, GPS)—generates two phase-coherent 156.25 MHz outputs that independently drive two QPLL groups, partitioning the eight transceiver lanes into two quads.

Phase determinism is hindered by the reset-dependent, random output phases of independent QPLLs, which impact inter-lane and inter-concentrator synchronization. TD-Link tackles this in two tiers:

1. Intra-Concentrator Alignment: For rings within the same Data Concentrator, a custom finite-state machine (FSM) iteratively aligns each transmit lane by exploiting the elastic buffer's half-full flag as a phase detector. The phase interpolator (PI) is swept until the write-read pointer latency corresponds to half the FIFO depth, fixing the phase deterministically per ring.
2. Inter-Concentrator Alignment: For systems with multiple concentrators, a Digital Dual Mixer Time Difference (DDMTD) circuit compares the phase of the slave's recovered transceiver clock with the master's reference. Fine phase trimming is applied via a programmable PLL delay, closing the synchronization loop with picosecond-level residuals.

   Figure 2: TD-Link master clock distribution with two QPLLs—each driving four transceivers—which must be phase-aligned post-reset via PI-based FSMs; DDMTD circuits supply inter-concentrator phase comparison.

   

   Figure 3: Multi-concentrator topology: masters distribute reference clock to slaves; each group of four rings is independently phase-aligned using DDMTD feedback.

Integrated Data Transport and Deterministic Command Execution

TD-Link's token-based train protocol enables fully streaming, on-the-fly data aggregation across a daisy-chain, with each board appending its payload and CRC to passing data frames, avoiding cumulative per-hop processing and store-and-forward delays. Control commands and global synchronization events (T0) are multiplexed with data in the same physical stream. Deterministic start/stop of acquisition is enforced by distributing a global timestamp with per-board corrections precomputed by the master, ensuring acquisition phase alignment throughout the chain independent of board location.

FERS Slave Datapath and Clock Recovery

Each FERS-5200 board recovers the embedded clock from the upstream optical stream using the FPGA’s CDR block; this clock is filtered by an external low-jitter PLL (zero-delay) before both driving local logic (time-to-digital conversion, ADC) and retransmitting downstream. This clock-cleaning approach forestalls jitter accumulation that would otherwise occur in a naïve forwarding scheme and stabilizes timing reference propagation along the daisy chain.

Figure 4: Block diagram of the FERS slave datapath—showing clock recovery, jitter cleaning, and phase-coherent retransmission along the daisy chain.

Figure 5: Detail of FERS slave clock recovery—CDR delivers a jittery 156.25 MHz clock, which is then cleaned for onward distribution to all subsystems.

DDMTD Phase Measurement and Deterministic Transmit Buffer Alignment

TD-Link implements a fully digital DDMTD phase detector, inheriting methodology from White Rabbit, to achieve sub-picosecond measurement of phase offsets between arbitrary clock domains. DDMTD magnifies small phase differences by mixing clocks with a narrow frequency offset, allowing high-resolution digital time-interval measurements inside the FPGA fabric.

Figure 6: Principle of the DDMTD phase detector—magnifies phase offset between clocks using digital mixing and counter-based edge separation measurement.

Within each concentrator, deterministic inter-lane phase alignment cannot be resolved with global PLL delay adjustment due to independent QPLL phases per quad. TD-Link's innovation is to retain the transmit FIFO (rather than bypassing for fixed latency) and use its internal half-full flag as a robust, one-bit phase detector. The FSM ultimately fixes each lane to the same logical phase by advancing the transmit PI until the flag signals the desired alignment.

Figure 7: Architecture of the deterministic transmit buffer alignment: PI shifts read clock, FIFO fill-level feedback supplies phase information to FSM.

Figure 8: FSM logic for transmit buffer alignment—Phase 1: sweep PI to bracket transition; Phase 2: lock to 0→1 TX buffer status transition, ensuring deterministic pointer relationship.

Empirical Characterization and Performance

TD-Link's synchronization is empirically validated using FERS-5200 boards equipped with PicoTDC ASICs (resolution <$10$ ps) in a variety of configurations. The standard deviation ($\sigma$) of the time difference between correlated pulses across two boards is the primary performance metric. Key results:

• Coaxial Reference (Intrinsic Limit): $\sigma \simeq 7$ ps (intrinsic floor from TDC quantization noise).
• Same-Link Daisy Chain: $\sigma \simeq 24$ ps (includes two CDR stages and PLL cleaning).
• Different Rings (Same Concentrator, Different Quads): $\sigma \simeq 27$ ps—demonstrates FSM-alignment loop does not measurably degrade performance.
• Different Concentrators (Full Synchronization Chain): $\sigma \simeq 28$ ps—shows marginal increase from single-concentrator case, indicating that DDMTD-based inter-concentrator synchronization is highly effective.

  Figure 9: Baseline (coaxial reference) jitter: $\sigma \simeq 7$ ps, setting the intrinsic system floor.

  

  Figure 10: Same-ring optical daisy chain: $\sigma \simeq 24$ ps, dominated by per-board CDR and PLL.

  

  Figure 11: Two rings (different QPLL quads), same concentrator: $\sigma \simeq 27$ ps.

  

  Figure 12: Two FERS boards belonging to different concentrators: $\sigma \simeq 28$ ps (full chain).

Repeated measurements across power cycles confirm sub-1 ps reproducibility, and temperature drift is the main systematic for very-long-term phase stability (tens of ps), but is automatically corrected at each reset. Quadrature subtraction of the TDC contribution isolates the optical chain’s residual jitter budget.

Implications, Comparative Context, and Future Directions

TD-Link achieves timing precision (25–30 ps RMS, end-to-end) comparable to or exceeding that of state-of-the-art switched-fabric protocols such as White Rabbit, while optimizing for dense, linear topologies common in large detector arrays. A bold architectural claim is that the combination of master clock discipline, cleaned per-board retransmission, and FPGA-only FSM phase alignment yields stable, deterministic synchronization independent of system scale, provided the chain’s length and the number of concentrators stay within architectural guidelines.

In practical terms, this enables large-scale SiPM or gaseous detector arrays to be built with minimal optical infrastructure and no need for separate timing networks or dedicated switches. The approach also reduces cost and complexity for R&D and moderate-scale detector installations, while preserving pathways toward integration in larger facilities via hierarchical chaining.

Theoretically, the all-digital timing alignment techniques pioneered in TD-Link—specifically, the use of elastic buffer fill-level as a phase detector and the integration of DDMTD logic in standard FPGA fabric—open opportunities for deterministic latency and clock-domain crossing in other high-performance DAQ and timing-sensitive distributed systems, decoupled from the limitations of FPGA PLL architectures. Future innovations may focus on further scaling, in-situ correction for thermal drift, and porting alignment schemes into mid-range FPGAs for broader DAQ applications.

Conclusion

TD-Link presents an optical communication and synchronization protocol capable of delivering sub-30 ps RMS timing alignment alongside high-throughput data acquisition in a daisy-chain topology. By integrating deterministic FPGA-alignment, digital phase measurement, and ring-based data transport, the system achieves robust, scalable, and infrastructure-efficient readout suitable for next-generation large-scale detector front-ends. Empirical results confirm the robustness, power-cycle stability, and practical synchronization parity with entrenched switched-timing protocols, establishing TD-Link as a compelling architecture for precision distributed detector systems.