SANA-Streaming: UAV ABR & Video Editing

• SANA-Streaming is a dual-framework approach integrating sensor-augmented neural adaptive bitrate streaming for UAVs and a hybrid Diffusion Transformer for real-time video-to-video editing.
• The UAV streaming component employs reinforcement learning with LSTM-based temporal modeling and sensor quantization to improve QoE by over 21% under volatile wireless conditions.
• The video editing module uses cycle-reverse regularization and optimized GPU kernels achieving up to 24 FPS with a 30% reduction in inter-frame flicker for temporally consistent outputs.

SANA-Streaming encompasses two distinct, widely cited frameworks for real-time streaming applications: (1) sensor-augmented neural adaptive bitrate (ABR) streaming over UAV links via reinforcement learning and (2) high-resolution, latency-bounded video-to-video editing using a hybrid diffusion transformer pipeline. Both variants implement real-time constraints and system/model co-design, but target different modalities and technical challenges: network adaptation (ABR) and temporally consistent generative editing. The following sections delineate the technical composition, methodologies, and empirical results reported in (Xiao et al., 2019) and (Zhao et al., 28 May 2026).

1. Sensor-Augmented Neural Adaptive Bitrate (SA-ABR) Streaming over UAVs

SA-ABR, also referenced as SANA-Streaming in the UAV/ABR literature, is a client-side reinforcement learning solution for air-to-ground adaptive video streaming. It incorporates real-time UAV sensor data into the bitrate selection process to better adapt to volatile wireless channels.

System Architecture

Key modules include:

• Sensor Data Collector integrates GPS, IMU (accelerometer, gyroscope), barometer, and compass, sampling once per video chunk to yield quantized flight-state vectors (distance, velocity, acceleration).
• Network Monitor computes application-layer throughput using TCP packet counts, averaging over every chunk (2 s).
• Buffer Manager simulates playback state and rebuffering without an actual decoder, updating buffer occupancy $b_t$ and rebuffer time $T_t$ as:
  • If $b_t \geq f_t$:

  $$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$ - Else:

  $$b_{t+1} = t_\text{chunk},\qquad T_t = \lceil (f_t-b_t)/0.5\rceil\cdot 0.5$$

• DRL Agent (Actor–Critic) ingests quantized sensors and 8-step throughput history, passes them through a 2-layer LSTM (64 hidden units/layer) and two dense layers (30/10 units), outputting a distribution over possible next-bitrate actions.

• Video Encoder exposes the fixed bitrate set $A = \{300, 750, 1850, 2850\}$ Kbps (covering 240p–1080p).

Dataflow

For each chunk, sensor/network features and an 8-step throughput vector are input to the agent, which samples the next bitrate. Upon download completion, the buffer state and reward are updated, and a transition tuple $(s_t, a_t, r_t, s_{t+1})$ is fed back for training.

2. MDP and Deep Reinforcement Learning Formulation

The ABR control loop is formulated as a Markov Decision Process $(\mathcal S, \mathcal A, \mathcal P, \mathcal R)$:

• State $s_t = (u_t, v_t, x_t)$ with:

  • $u_t = (d_t, v_t, a_t)$, quantized UAV flight state: $T_t$0 (distance $T_t$1 m or $T_t$2 m), $T_t$3, and $T_t$4.
  • $T_t$5: current buffer occupancy $T_t$6 s and immediately previous bitrate $T_t$7.
  • $T_t$8: recent throughput history in Mbps.
• Action space: $T_t$9.
• Transition: determined by real throughput trace and buffer simulation dynamics.
• Reward: Per-chunk QoE,

$b_t \geq f_t$0

where $b_t \geq f_t$1, $b_t \geq f_t$2 Kbps, $b_t \geq f_t$3.

Actor–Critic training uses 10 parallel actors running in varied UAV environments, n-step TD updates (n = 41), and no replay buffer.

3. LSTM Temporal Modeling and Sensor Quantization

The throughput LSTM, with sequence length 8, encodes both short- and medium-range temporal dependencies arising from rapidly varying wireless channels attributable to UAV motion. Ablation shows that performance (QoE) significantly degrades with shorter history (e.g., $b_t \geq f_t$415% QoE drop for length 2 vs 8). Quantization of sensor readings (distance, velocity, acceleration) reduces the variance and improves mean QoE versus unquantized/raw side-information.

4. Experimental Evaluation and Results: UAV ABR

The system was implemented on a DJI Matrice 100 UAV platform (25 m altitude, speeds up to 19.5 m/s), streaming to a ground laptop over IEEE 802.11n (2.4 GHz). Over 1,000 real 100 s throughput traces, SANA-Streaming yields:

• QoE improvement: 21.4% over Pensieve (next-best ABR baseline)
• Component gains: utility +10.8%, smoothness penalty –35.3%, rebuffer penalty unchanged
• Ablations: removing sensors leads to QoE reductions of 7–30%; raw sensor input (continuous) increases variance and reduces mean QoE relative to quantized
• Network temporal modeling: LSTM outperforms CNN baseline, delivering +17.5% QoE and much lower rebuffer/smoothness penalties

The agent selects bitrate actions that stabilize the buffer and track channel changes efficiently, maintaining playback stability under highly non-stationary UAV mobility (Xiao et al., 2019).

5. SANA-Streaming for Real-Time Video-to-Video Editing

The more recent SANA-Streaming system (2026) addresses the video-to-video editing domain, integrating a hybrid Diffusion Transformer (DiT) model, a novel cycle-reverse regularization scheme, and highly optimized system-level GPU kernels and quantization strategies.

Hybrid Diffusion Transformer Architecture

• Backbone: 20 blocks—15 Gated DeltaNet (GDN, linear attention) and 5 softmax-attention blocks (evenly interleaved).
• GDN blocks provide global recurrent memory via compressed states, maintaining constant time/memory.
• Softmax blocks inject local, high-fidelity modeling via windowed cross-chunk attention and a persistent "sink" token.
• Attention complexity & kernel design: Linear blocks incur $b_t \geq f_t$5 cost and require only $b_t \geq f_t$6 cached state; softmax windows reduce cost to $b_t \geq f_t$7.

Cycle-Reverse Regularization

• Forward edit: Applies the desired prompt ($b_t \geq f_t$8), generating a chunk $b_t \geq f_t$9.
• Reverse branch: Attempts to reconstruct the source $$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$0 from $$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$1 using an inverse prompt ($$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$2), penalized via a flow-matching L2 loss.
• Key property: Enables enforcement of semantic/temporal consistency over unpaired training data, as only raw videos and synthetic "reverse" prompts are required.

The total per-chunk training loss is

$$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$3

where $$b_{t+1} = b_t + t_\text{chunk} - f_t,\qquad T_t = 0$$4 is the distillation loss from a short-clip teacher.

6. System-Level Optimizations: Fused Kernels and Mixed-Precision Quantization

SANA-Streaming leverages:

• Fused GDN kernels: Three-phase, Triton-based GPU kernels optimize memory and computation, yielding 1.5×–2.2× per-block and 1.59× end-to-end speed-up over naïve PyTorch on RTX 5090.
• Mixed-Precision Quantization:
  • Policies: BF16 (reference), FP8 TensorCore, NVFP4 Grouped 4-bit.
  • Final policy: FP4 on 62% parameters (67% FLOPs), FP8 on robustness-critical layers, BF16 on highly sensitive modules.
  • Performance: DiT latency for one-minute video reduced from 26.8 s (BF16) to 16.8 s, with LPIPS within 0.17 of reference.

Empirically, this system achieves 24 FPS at 1280×704 resolution (end-to-end), 58 FPS within the DiT module, and matches or exceeds existing SOTA in OpenVE benchmarks (Zhao et al., 28 May 2026).

Benchmarking Table

Method	E2E FPS (1280×704)	Avg. OpenVE Score
Lucy-Edit	~6.7	2.50
SANA-Streaming	24	2.62
SANA (distilled)	762 (DiT core)	2.42

7. Temporal Coherence and Editing Quality

Cycle-Reverse regularization yields a ≈30% reduction in inter-frame flicker over long sequences, measured via non-edited region stability. The mixture of global memory (GDN) and local softmax attention enables both long-range consistency and high-fidelity edits without paired edited video.

SANA-Streaming's design, combining modeling, training, and hardware-aware optimization, establishes new throughput and quality benchmarks for real-time streaming video editing and temporally robust UAV video streaming. Empirical results validate its effectiveness in both highly dynamic network environments and low-latency GPU-accelerated video generation contexts (Xiao et al., 2019, Zhao et al., 28 May 2026).