A Comparative Study of Dynamic Programming and Reinforcement Learning in Finite Horizon Dynamic Pricing

Abstract: This paper provides a systematic comparison between Fitted Dynamic Programming (DP), where demand is estimated from data, and Reinforcement Learning (RL) methods in finite-horizon dynamic pricing problems. We analyze their performance across environments of increasing structural complexity, ranging from a single typology benchmark to multi-typology settings with heterogeneous demand and inter-temporal revenue constraints. Unlike simplified comparisons that restrict DP to low-dimensional settings, we apply dynamic programming in richer, multi-dimensional environments with multiple product types and constraints. We evaluate revenue performance, stability, constraint satisfaction behavior, and computational scaling, highlighting the trade-offs between explicit expectation-based optimization and trajectory-based learning.

• The paper demonstrates that Fitted DP consistently achieves the highest mean revenue and stability across various dynamic pricing environments.
• It reveals that RL methods like PPO and A2C converge to near-optimal performance with increased training, effectively adapting to complex constraints.
• The study highlights key trade-offs between expectation-based optimization and trajectory-driven learning, emphasizing scalability in real-world pricing applications.

Comparative Analysis of Dynamic Programming and Reinforcement Learning in Finite-Horizon Dynamic Pricing

Introduction and Motivation

This paper systematically compares Fitted Dynamic Programming (DP) and model-free Reinforcement Learning (RL) methods in finite-horizon dynamic pricing. Pricing decisions are crucial in domains such as real estate, airline revenue management, live event ticketing, and e-commerce, where inventory is limited and demand is stochastic. Traditional DP relies on explicit demand models and Bellman recursion, achieving optimality under well-specified assumptions, but faces scalability issues as product typologies or operational constraints increase. RL methods, conversely, learn directly from interaction trajectories and are increasingly adopted for their flexibility in modeling market dynamics, especially where explicit probability models are unavailable or computationally infeasible.

The study extends prior work by presenting a unified empirical benchmark across environments of escalating complexity—from single typology, unconstrained cases to multi-typology settings with coupled constraints and heterogeneous demand, thus providing novel insights into scalability, constraint adaptation, and trade-offs between explicit expectation optimization and trajectory-based learning.

Problem Formulation: Typologies, Constraints, and Demand Models

The environments considered are parameterized by typologies—each representing an independent inventory-demand pair with possible coupling via shared constraints. There are $K$ typologies per environment, each with price $p_{i, t}$, stochastic Poisson demand $Q_{i, t}$, and inventory $N_{i, t}$. Sales are inventory constrained with $S_{i, t} = \min(Q_{i, t}, N_{i, t-1})$ and cumulative revenue updated each period.

Penalties are imposed at fixed steps for failing to reach aggregate revenue targets, introducing non-separable rewards and inter-temporal coupling across typologies:

$$\text{penalty}_\tau = \mu \max(0, R^{\text{target}} - R_\tau),$$

where $\mu$ is the penalty severity and $R_\tau$ is cumulative revenue.

Demand intensity is specified as:

$$d_i(p, t) = \max(0, \alpha_i - \beta_i p)\left(1 + \frac{t}{T}\right),$$

with independent demand per typology and deterministic urgency increase over time.

Solution Approaches

Dynamic Programming with Demand Estimation

DP utilizes linear regression to fit demand from episodic data:

$$d(p, t) = \beta_0 + \beta_1 p + \beta_2 \frac{t}{T} + \beta_3 p \cdot \frac{t}{T},$$

and achieves optimal control via recursive Bellman updates on discretized state and action spaces.

Complexity scales quadratically with inventory and multiplicatively with number of typologies and revenue bins:

$p_{i, t}$0

making DP computationally prohibitive in multi-dimensional or constrained settings.

Reinforcement Learning

RL agents (DQN, PPO, A2C) interact with vectorized environments, observing normalized state variables and optimizing via sampled trajectories and stochastic gradient updates. RL does not enumerate expectations or demand realizations, and computational cost scales linearly with episode count and horizon:

$p_{i, t}$1

where $p_{i, t}$2 is the per-episode network cost.

Experimental Environments

The benchmark comprises five environments, each with distinct dimensionality and constraints:

• Environment 1: Single typology, no constraint—baseline.

  Figure 1: Single typology environment serving as the baseline for model comparison.

• Environment 2: Two identical typologies, aggregate revenue constraint at step $p_{i, t}$3.

  Figure 2: Two identical typologies with a joint revenue constraint, increasing complexity and coupling.

• Environment 3: Two heterogeneous typologies, differing demand functions and price ranges, with constraint.

  Figure 3: Two heterogeneous typologies illustrating increased structural complexity and coupling.

• Environment 4: Single typology with intermediate revenue constraint.

  Figure 4: Single typology under a constraint regime, isolating constraint effects.

• Environment 5 (Appendix): Single typology with non-linear demand dependency.

  Figure 5: Single typology with non-linear demand, validating robustness of comparative findings.

Numerical Results and Analysis

Revenue Performance and Stability

Across all environments, Fitted DP consistently achieves highest mean revenue and minimal variance bands, regardless of training budget, due to deterministic optimization over learned model parameters. RL methods, particularly PPO and A2C, require larger training budgets (≥400 episodes) to converge toward DP performance, with PPO providing best convergence and stability in low-data regimes and A2C showing superior results as data increases. DQN is characterized by persistently higher variance and lower mean revenue.

In environment 1 (single typology), RL performance lags DP for under 100 episodes but aligns closely as episode count increases; PPO shows stable trajectories while DQN's variance remains elevated.

Impact of Constraints and Dimensionality

In environments 2 and 3 (multi-typology, constrained), DP retains clear advantages in stability and constraint satisfaction, attaining highest probability mass above revenue targets prior to the penalty step.

Figure 6: Distribution of cumulative revenue before penalty (t=7) for all methods in environment 2; DP achieves highest constraint satisfaction.

RL methods struggle more with constraint adaptation, as evidenced by larger dispersion and probability mass below targets, especially for DQN. PPO and A2C manage constraint-aware behavior but incur penalties more frequently.

Figure 7: Revenue distribution before penalty in heterogeneous typology environment; DP remains strongest, RL methods exhibit increased variability.

Policy-gradient methods (PPO, A2C) scale better with increased dimensionality, demonstrating resilience and convergence given sufficient training, while value-based DQN fails to reliably adapt.

Robustness to Non-linear Demand

In environment 5, introducing non-linear demand did not alter qualitative results: DP maintained highest and most stable revenue; PPO excelled in low-data scenarios, and A2C improved steadily with training. DQN again underperformed.

Training Time and Practical Implications

DP is efficient in low-dimensional environments (training time on order of seconds), but training time increases dramatically for multi-typology settings (on the order of ~1200 seconds versus ≤22 seconds for RL methods at 2000 episodes). RL methods, particularly PPO, scale favorably in larger action/state spaces and remain practical for real-world applications with growing typology complexity.

Implications and Future Directions

Theoretical implications indicate that DP, given a well-specified demand model, is optimal and robust but is fundamentally limited by state-space scaling and non-separable reward structures induced by constraints. RL methods are versatile and scalable, capable of matching DP performance with sufficient episodic data, and adapt to high-dimensional, constraint-laden environments albeit with increased exploration variance.

Practically, RL is preferable in settings where demand models are ambiguous or operational complexity precludes exhaustive enumeration, especially as product assortments and constraints proliferate.

Future directions include extension to alternative demand specifications (e.g., log-linear, censored, competitive effects), integration of additional operational constraints (e.g., fairness, strategic withholding), and hybrid architectures leveraging both model-based and trajectory-driven learning paradigms.

Conclusion

This comparative study establishes Fitted DP as a robust benchmark for finite-horizon dynamic pricing under known demand, but demonstrates that policy-gradient RL methods (particularly PPO and A2C) attain near-optimal performance in both unconstrained and constrained environments as training data increases and problem dimensionality grows. Value-based RL (DQN) is less stable and less competitive, particularly in constraint-heavy and high-dimensional settings.

The research highlights trade-offs between expectation-based optimization and sampling-driven policy learning, underscoring scalability and flexibility advantages of RL in operational contexts where explicit demand modeling is infeasible or computationally prohibitive. The comparative conclusions remain valid across linear and non-linear demand settings, with strong practical relevance for data-driven revenue management and pricing system design.