Analyzing Symbolic Properties for DRL Agents in Systems and Networking

Abstract: Deep reinforcement learning (DRL) has shown remarkable performance on complex control problems in systems and networking, including adaptive video streaming, wireless resource management, and congestion control. For safe deployment, however, it is critical to reason about how agents behave across the range of system states they encounter in practice. Existing verification-based methods in this domain primarily focus on point properties, defined around fixed input states, which offer limited coverage and require substantial manual effort to identify relevant input-output pairs for analysis. In this paper, we study symbolic properties, that specify expected behavior over ranges of input states, for DRL agents in systems and networking. We present a generic formulation for symbolic properties, with monotonicity and robustness as concrete examples, and show how they can be analyzed using existing DNN verification engines. Our approach encodes symbolic properties as comparisons between related executions of the same policy and decomposes them into practically tractable sub-properties. These techniques serve as practical enablers for applying existing verification tools to symbolic analysis. Using our framework, diffRL, we conduct an extensive empirical study across three DRL-based control systems, adaptive video streaming, wireless resource management, and congestion control. Through these case studies, we analyze symbolic properties over broad input ranges, examine how property satisfaction evolves during training, study the impact of model size on verifiability, and compare multiple verification backends. Our results show that symbolic properties provide substantially broader coverage than point properties and can uncover non-obvious, operationally meaningful counterexamples, while also revealing practical solver trade-offs and limitations.

• The paper presents a systematic method for formal symbolic property verification of DRL agents by comparing paired policy executions over input regions.
• It introduces the diffRL framework, a solver-agnostic approach that decomposes symbolic properties into finite verification queries for both discrete and continuous control.
• Empirical evaluations in adaptive video streaming, wireless resource allocation, and congestion control highlight practical improvements in DRL safety and robustness.

Symbolic Property Analysis for Deep Reinforcement Learning Agents in Systems and Networking

Introduction and Motivation

The deployment of deep reinforcement learning (DRL) agents in control loops for systems and networking (e.g., adaptive video streaming, wireless resource management, congestion control) necessitates assurance that agent behavior remains within operationally acceptable bounds across a broad range of inputs. Classical verification approaches have predominantly focused on pointwise properties—verifying that a property holds for specific, manually selected input-output pairs—yielding limited input coverage and often missing unsafe or anomalous behaviors in the unexamined input regions.

This paper presents a systematic methodology for the formal analysis of symbolic properties for DRL agents operating in systems and networking. Symbolic properties collectively specify expected relations covering ranges of input states, enabling the assessment of critical behavioral patterns such as robustness and monotonicity with respect to state perturbations.

Figure 1: A visual comparison of traditional pointwise verification against symbolic property analysis, which provides increased coverage through whole-region input reasoning.

Symbolic Properties: Formalization and Taxonomy

The central technical contribution is a unified formulation for symbolic properties parameterized by a comparison function $f$ and thresholds over pairs of DRL policy executions:

$$\forall x \in X, l_{s_i} \leq s_i \leq u_{s_i}: f(\pi(x), \pi(x+s)) \leq d$$

where $\pi$ represents the (deterministic) DNN-based policy, $x$ is a state sampled from an operational domain $X$, and $s$ is a bounded input perturbation.

This abstraction encompasses:

• Robustness properties: Ensuring the policy is insensitive to bounded input perturbations (i.e., small noise should not cause disproportionately large output changes).
• Monotonicity properties: Ensuring that increasing (or decreasing) a particular feature produces directionally consistent output behavior, as mandated by domain semantics.

Both robustness and monotonicity properties are critical for operational deployment: the former guarantees stability against input noise and adversarial manipulation; the latter encodes expected safety invariants relevant to resource allocation and congestion management.

Figure 2: Symbolic property verification operates over pairs of policy executions differentiated by controlled input perturbations.

diffRL: Comparative Encoding and Solver-Agnostic Decomposition

To make symbolic analysis tractable with existing verification engines, the paper introduces diffRL, a framework that systematically transforms symbolic property checking into a set of solver-compatible queries.

The key innovation is a comparative encoding: two coupled policy executions (with base and perturbed inputs, respectively) are evaluated, and their output actions are compared. Since discrete control often relies on an $\arg\max$ selection over DNN outputs, the symbolic property is decomposed into a finite set of sub-properties, each corresponding to a concrete pair of output neurons whose joint selection could indicate a property violation.

Figure 3: diffRL framework overview: Trained agent and property specification are encoded into solver-compatible comparative queries, dispatched to multiple verification engines, and results are synthesized for property assessment.

For DRL agents with discrete action spaces, each invalid action pair generates an independent verification query. For continuous-action agents, property checking directly involves the comparison of DNN output means, leveraging the policy's continuous nature.

This decomposition is exact, introducing no conservatism or relaxation, and maintains compatibility with diverse DNN verification paradigms—namely MIP (Gurobi), SMT (Marabou), and BaB (Alpha-Beta-CROWN). This design ensures maximal practical coverage, given the varied strengths and bottlenecks inherent to modern verifiers.

Empirical Evaluation: DRL in Networking Applications

The effectiveness and tractability of symbolic property analysis are rigorously evaluated through three case studies representative of DRL use in networking:

Adaptive Video Streaming (Pensieve)

Analysis focus: Monotonicity (Capacity Utilization, Rebuffering Avoidance) and Robustness with respect to throughput and buffer state perturbations.

The verification traces the evolution of property satisfaction throughout policy training, evidencing that monotonicity violations decrease as policies attain higher reward, but do not vanish entirely—some violations persist or even re-emerge later in training, highlighting the non-monotonic effect of DRL parameter updates on safety. Model size reduction (e.g., halving hidden units) significantly improves verifiability without reward degradation.

Figure 4: Capacity Utilization property satisfaction for Pensieve across training checkpoints and input coverage—safe (green), unsafe (red), and timed out/unknown (gray) queries, resolved by MIP ($\star$) or CROWN ($\diamond$).

Aggregating results across verification engines (MIP, BaB) reduces unknowns and leverages solver complementarity, especially for larger networks.

Wireless Resource Allocation (CMARS)

Analysis focus: Monotonicity (Contention-Aware Allocation, Channel Compensation) and Robustness against noisy demand and channel measurements.

Verification reveals operationally meaningful counterexamples, including robustness violations where minimal input perturbations yield shifts covering over 80% of the allocation range, reflecting high susceptibility to noise and indicating underrepresented operational regimes in training data. Increased network depth does not guarantee improved property satisfaction and, counterintuitively, deeper models are sometimes less safe and harder to verify.

Figure 5: Solver-wise comparison of outcomes for symbolic property queries in CMARS across properties, policy network sizes, and action space dimensionality.

Multi-engine verification remains essential. MIP solvers excel for smaller networks, while Marabou and BaB engines help resolve complex queries in larger configurations.

Figure 6: Query execution time breakdown per backend on the channel compensation property, illustrating solver scalability trends by model complexity and action space size.

Congestion Control (Aurora)

Analysis focus: Continuous action space—monotonicity (capacity utilization with respect to improved network signals) and robustness to transient noise.

Symbolic analysis is seamlessly adapted to continuous control, verifying safe operation for a broad (70%) input domain, but exposing counterexamples at maximum domain coverage. Here too, MIP is critical to tractability for the largest input sets, demonstrating again that solver-agnostic strategies allow pushing symbolic verification to practical boundaries in DRL control.

Theoretical and Practical Implications

This work demonstrates that symbolic property analysis can lift the coverage of DRL agent verification from brittle pointwise checks to operationally meaningful input regions—catching pathological behaviors that are both rare and impactful. This approach exposes subtle policy flaws, such as robustness violations and monotonicity failures, that often align with real-world failure modes previously undetectable with pointwise methods.

One of the paper's strong technical claims is that property satisfaction is not monotone as a function of training progress or model complexity. Model size, training stage, and input coverage jointly affect verifiability and actual compliance. Deeper models or further training can both increase decision boundary complexity—potentially making solver search harder—and fail to improve safety if not explicitly incentivized during optimization.

Practical implications include:

• Multi-engine verification becomes necessary. Aggregating results across MIP, SMT, and BaB engines offers meaningful gains in reducing unknowns and uncovering corner-case behaviors.
• Symbolic analysis enables counterexample-driven refinement. Violations detected through these methods can guide targeted retraining or constraint-based regularization, facilitating safer DRL deployment.
• Adaptable to architectural and domain specifics. The approach generalizes to discrete or continuous action spaces, various DNN architectures, and is applicable to any domain where expected agent behaviors can be formulated as symbolic predicates over regions of the input space.

Speculation on Future Developments

Several research avenues are highlighted:

• Automated property mining: Leveraging empirical patterns or gradient analysis to suggest domains and features where monotonicity and robustness should be formally checked, reducing reliance on human expertise.
• Softmax and stochastic policies: Extending symbolic analysis to encompass stochastic action selection (beyond deterministic $\arg\max$) via advanced property encodings and relaxation techniques [wei2023convex].
• Counterexample-driven robustification: Incorporating detected violations into training loops or via constraints, building on counterexample-guided repair paradigms.
• Scalability: Exploiting domain- and architecture-specific inductive biases for more scalable verification and tighter symbolic encodings.

Conclusion

This work constitutes a rigorous, generic, and practically tractable methodology for symbolic property analysis of DRL agents in systems and networking, demonstrating that leveraging comparative encoding and decomposition strategies over existing formal engines meaningfully advances the state of DRL agent assurance. The diffRL framework and empirical analysis show that symbolic properties enable broader, operationally relevant guarantees, support robust system integration, and yield actionable diagnostic signals for agent improvement and safe deployment.

---

Reference: "Analyzing Symbolic Properties for DRL Agents in Systems and Networking" (2604.04914).