Replay-Streams Protocol

• Replay-streams protocol is a method for synchronized data streaming with precise action–reaction ordering, enabling reproducible experiments in interactive systems.
• It employs detailed capture and timestamping of commands (JSON) and frames (RTP with QR metadata) to ensure deterministic replay and accurate simulation of original events.
• The protocol extends to self-purified continual learning by integrating noise filtering and buffer management, and supports advanced QoE/QoS metrics like delay and jitter.

The replay-streams protocol encompasses a class of synchronized data streaming methods that enable reproducible evaluation and learning in interactive and continual systems. It has been instantiated in contexts as varied as cloud gaming Quality of Experience/Service (QoE/QoS) benchmarking and continual learning under noisy data streams. Despite differences in application, a unifying aspect of these protocols is the explicit recording, formatting, and deterministic playback of temporally ordered event or data streams—most notably, bidirectional “action–reaction” exchanges—under tightly controlled conditions. Central to their operation are algorithmic designs ensuring precise temporal coupling, message integrity, reliability under network impairments, and, in learning settings, mechanisms for noise purification and buffer management (Shirmarz et al., 17 May 2025, Kim et al., 2021).

1. Architecture and Action–Reaction Synchronization

Replay-streams-based architectures capture full end-to-end transactional histories, typically structuring communication as a lock-step progression through a unified, timestamped sequence. For example, in CGReplay for cloud gaming traffic analysis, this is realized by two agents—a player and a server—exchanging precisely ordered commands (“actions”) and video frames (“reactions”). Each atomic event (command or frame) is timestamped at capture, and a sync-order list $S$ is compiled:

$S = \{F_1, F_2, F_3, C_1, C_2, F_4, ...\}$

where $F_k$ denotes frame $k$ and $C_j$ denotes command $j$. During replay, both the sender (server) and receiver (player) step through $S$ in lock-step, enforcing the original inter-event intervals and causal order: the server transmits up to the next required input, then blocks for the corresponding command, while the player waits for required frames before advancing with further input (Shirmarz et al., 17 May 2025).

2. Capture Phase: Data and Synchronization Artifacts

In practice, the capture phase entails two major data streams. First, user inputs are sampled (e.g., via USB-sniffing at 30 Hz), recorded as JSON objects encoding command identifiers, timestamps, joystick axes, button states, and placeholders for acknowledgments. Second, output frames are acquired (e.g., by screen grabbing at 30 fps), stamped as PNG files with embedded QR codes encoding frame identifiers and capture timestamps. These data are then collated and sorted by timestamp to form the master sync-order sequence.

Stream Type	Format	Content Example
Uplink Cmd	JSON	C_id, timestamp, axes
Downlink	PNG + QR metadata	F_id, timestamp

This collated ordering is persisted as a plain-text file, ensuring reproducibility for later replay and protocol analysis (Shirmarz et al., 17 May 2025).

3. Replay Scheduling, Network Transport, and State Machines

Replay-streams protocols implement deterministic scheduling algorithms based on original capture inter-event gaps:

$$\Delta_i = t_i^s - t_{i-1}^s,\quad T_{send,i} = T_{send,i-1} + \Delta_i$$

with event emissions scheduled accordingly. Downlink (frame) data are typically transported over RTP (H.264, with optional SCReAM congestion control for realism), while uplink (command) data are carried as raw UDP JSON packets. State machines govern both agents: the server alternates between streaming frames, awaiting commands, resending lost frames, or rolling back upon NACKs; the player cycles through receiving frames, sending commands, and periodic acknowledgment handling.

Agent	States	Key Transitions
Server	STREAM_FRAMES, WAIT_FOR_CMD, ...	STREAM → WAIT on input req., WAIT → STREAM on ACK
Player	RECV_FRAMES, SEND_CMD, COLLECT_ACK	RECV → SEND on input req., RECV ↔ COLLECT_ACK

Reliability is integrated via timeouts and negative acknowledgments: on command timeouts, retransmissions are triggered; on NACK (frame loss), streaming rewinds window-wise (Shirmarz et al., 17 May 2025).

4. Protocol Formats and Synchronization Integrity

Replay-streams protocols specify low-level message formats to support synchronization and verification:

• Uplink Command (JSON over UDP):
  • Fields: C_id (uint32), timestamp (uint64, µs), axes ([int16]), button (uint8), ack (bool)
• Downlink Frame (RTP + H.264 NAL):
  • RTP: header (pt=96), sequence, timestamp (90 kHz), SSRC
  • Payload: H.264 unit, with QR metadata for F_id and ts embedded in I-frames

During replay, receivers validate the QR code-labelled frame IDs to guarantee correspondence with the intended event sequence (Shirmarz et al., 17 May 2025).

5. Timing, Synchronization, and QoE/QoS Metrics

Accurate replay of original time intervals is fundamental, yielding controlled experiments ideal for QoE/QoS assessment. The end-to-end one-way delay for a frame is defined as $D_k = T_{recv,k} - t_k^s$, and packet timing jitter is computed as per RFC 3550:

$$J = \frac{1}{N-1} \sum_n |\Delta_n - \mu|, \quad \mu = \frac{1}{N-1}\sum_n \Delta_n$$

This temporal fidelity enables the offline computation of PSNR or SSIM between exact-captured and delivered frames, and in situ logging of server/client FPS ratios, command delivery, frame loss, retransmission, and per-action reaction latencies. For example, preliminary results with a 50 ms RTT yielded action-reaction latencies around 42 ms with stall rates below 0.5% (Shirmarz et al., 17 May 2025).

6. Extension: Replay-Streams for Self-Purified Continual Learning

Replay-streams protocols were adapted for continual learning under noise in the Self-Purified Replay (SPR) framework, where buffer "streams" are filtered and replayed for robust representation learning (Kim et al., 2021). Here, two buffers are maintained: a delayed buffer $D$ of recent potentially noisy examples and a purified buffer $P$ of trusted samples. Periodically, $D$ is processed:

• Self-supervised NT-Xent contrastive learning is performed jointly on $D \cup P$: every example is augmented into two positive views, mapped through normalized projection, and NT-Xent loss is computed.
• Self-Centered filtering applies a stochastic centrality-based algorithm: graph adjacency matrices (per class) are constructed from feature cosine similarities, centrality scores extracted, and a Beta mixture model splits clean from noisy examples.
• Only samples with high posterior "clean" probability are retained in $P$.

This ensures the replayed (and thus learned-from) stream maintains resilience to catastrophic forgetting and noisy label contamination. Computationally, the dominant costs are all-pair similarities ($O(B^2 d)$) and eigenvector/BMM computation within $D$, with practical buffer sizes ranging from hundreds to one thousand (Kim et al., 2021).

7. Practical Considerations and Applications

Replay-streams protocols deliver major advances in experimental reproducibility, enabling controlled, artifact-complete replays of complex, real-world interaction traces or streaming data. Their applicability spans:

• Cloud gaming research: reproducible QoE/QoS assessment, network impairment emulation, algorithm benchmarking (Shirmarz et al., 17 May 2025).
• Stream-based continual learning: robust, self-purified buffer management under noisy and nonstationary input (Kim et al., 2021).

Practical deployment is facilitated by public code, GPU-accelerated graph routines, class-balance heuristics, and flexible support for both action-reaction loops and unlabelled "task-free" data streams. These protocols underpin principled evaluations by binding synchronization, reliability, and replay fidelity to deterministic, analysable artifacts.

A plausible implication is that the replay-streams paradigm will continue to propagate into domains requiring verifiable experimental reproducibility and accurate simulation of streaming interactions, especially where logging restrictions or noise render direct live measurement infeasible.