Value-Based Deep RL Scales Predictably

Abstract: Scaling data and compute is critical to the success of machine learning. However, scaling demands predictability: we want methods to not only perform well with more compute or data, but also have their performance be predictable from small-scale runs, without running the large-scale experiment. In this paper, we show that value-based off-policy RL methods are predictable despite community lore regarding their pathological behavior. First, we show that data and compute requirements to attain a given performance level lie on a Pareto frontier, controlled by the updates-to-data (UTD) ratio. By estimating this frontier, we can predict this data requirement when given more compute, and this compute requirement when given more data. Second, we determine the optimal allocation of a total resource budget across data and compute for a given performance and use it to determine hyperparameters that maximize performance for a given budget. Third, this scaling behavior is enabled by first estimating predictable relationships between hyperparameters, which is used to manage effects of overfitting and plasticity loss unique to RL. We validate our approach using three algorithms: SAC, BRO, and PQL on DeepMind Control, OpenAI gym, and IsaacGym, when extrapolating to higher levels of data, compute, budget, or performance.

• The paper presents a framework showing that performance in value-based deep RL scales predictably with data and compute, governed by the UTD ratio.
• It employs power law models to derive optimal hyperparameters, such as batch size and learning rate, across different resource budgets.
• Empirical evaluations across diverse environments validate that scaling laws accurately predict resource needs and guide optimal configuration.

The paper "Value-Based Deep RL Scales Predictably" (2502.04327) investigates the scaling properties of value-based deep reinforcement learning algorithms, specifically addressing the common perception of their instability and unpredictability. It presents empirical evidence demonstrating that, contrary to this belief, methods like Soft Actor-Critic (SAC), Bootstrap Re-parameterized Outputs (BRO), and Projected Q-Learning (PQL) exhibit predictable scaling behavior concerning data and computational resources. The core contribution lies in establishing a framework for predicting resource requirements and optimal hyperparameter configurations for large-scale experiments based on insights derived from smaller-scale runs.

Data-Compute Pareto Frontier and the UTD Ratio

A key finding is the existence of a Pareto frontier between the amount of environment interaction data ($D$) and the computational resources ($C$, measured in gradient steps) required to achieve a specific performance level $J$. This implies a fundamental trade-off: achieving the same performance level is possible with less data by using more compute, or with less compute by using more data.

The position on this Pareto frontier is primarily controlled by the Updates-to-Data (UTD) ratio, denoted by $\sigma = N/D$, where $N$ is the total number of gradient updates performed. The paper demonstrates that the relationship between the minimum data required ($D_J$) to reach performance $J$ and the UTD ratio $\sigma$ can be accurately modeled by a power law:

$$D_J(\sigma) \approx D_{min} + \left(\frac{\beta_J}{\sigma}\right)^{\alpha_J}$$

Here, $D_{min}$ represents a theoretical minimum data requirement, while $\alpha_J$ and $\beta_J$ are constants specific to the performance level $J$ and the task. This equation shows that increasing $\sigma$ (performing more updates per data point) reduces the data needed, approaching $D_{min}$ asymptotically.

Conversely, the compute required, $C_J$, also depends predictably on $\sigma$. Since $C = B \cdot N = B \cdot \sigma \cdot D$, where $B$ is the batch size, the compute cost $C_J$ to reach performance $J$ can be expressed as a function of $\sigma$ by substituting the expressions for $D_J(\sigma)$ and the optimal batch size $B^*(\sigma)$ (discussed below). The derived functional form (detailed in Appendix A of the paper) approximates $C_J(\sigma)$ as a sum of two power laws, indicating that increasing $\sigma$ generally increases the required computation, although the relationship is mediated by the optimal batch size. Plotting $D_J(\sigma)$ against $C_J(\sigma)$ for a fixed $J$ explicitly reveals the Pareto frontier.

Predictable Hyperparameter Relationships

The predictability extends to key hyperparameters, namely the batch size ($B$) and the learning rate ($\eta$). The study finds that the optimal values of these hyperparameters, $B^*$ and $\eta^*$, are strongly correlated with the UTD ratio $\sigma$, but not significantly correlated with each other. This contrasts with some heuristics adapted from supervised learning where batch size and learning rate scaling are often linked.

Specifically, the optimal batch size $B^*$ is found to decrease with increasing $\sigma$, following a power law relationship:

$$B^*(\sigma) \approx \left(\frac{\beta_B}{\sigma}\right)^{\alpha_B}$$

This inverse relationship suggests that smaller batch sizes are preferable when performing many updates on the same data (high $\sigma$). This behavior is hypothesized to mitigate overfitting, a common concern when reusing data extensively in off-policy RL.

Similarly, the optimal learning rate $\eta^*$ also tends to decrease with increasing $\sigma$, again modeled by a power law:

$$\eta^*(\sigma) \approx \left(\frac{\beta_\eta}{\sigma}\right)^{\alpha_\eta}$$

Lowering the learning rate at high UTD ratios is proposed as a mechanism to counteract plasticity loss. With numerous gradient steps, a high learning rate can lead to catastrophic forgetting or instability, whereas a smaller learning rate promotes more stable convergence.

These predictable relationships between ($B^*$, $\eta^*$) and $\sigma$ are crucial. They imply that simply scaling up data or compute while keeping hyperparameters fixed, or tuning them independently, is suboptimal. Instead, $B$ and $\eta$ should be co-adapted based on the chosen UTD ratio $\sigma$.

Optimal Resource Allocation Strategy

The framework enables the determination of an optimal resource allocation strategy given a total budget $F$. The budget can be defined as a weighted sum of compute and data costs, $F = C + \delta D$, where $\delta$ represents the relative cost of acquiring one data point versus performing one gradient step.

By analyzing the performance achieved for different combinations of $D$ and $C$ (parameterized by $\sigma$) under a fixed budget constraint, the paper shows that the optimal UTD ratio, $\sigma^*(F_0)$, which maximizes performance for a given budget $F_0$, can also be predicted. This relationship is again modeled using a power law:

$$\sigma^*(F_0) \approx \left(\frac{\beta_\sigma}{F_0}\right)^{\alpha_\sigma}$$

Estimating this relationship allows practitioners to predict the best UTD ratio $\sigma^*$ for a target budget. Once $\sigma^*$ is determined, the corresponding optimal $B^*(\sigma^*)$ and $\eta^*(\sigma^*)$ can be inferred using their respective power laws. This provides a complete set of hyperparameters predicted to yield the best performance for the given budget $F_0$, effectively determining the optimal point on the data-compute Pareto frontier for that budget.

Methodology and Validation

The findings are supported by extensive empirical evaluations using SAC, BRO, and PQL algorithms across diverse environments, including DeepMind Control Suite, OpenAI Gym, and IsaacGym benchmarks. The methodology involved:

1. Systematic Sweeps: Performing runs across wide ranges of $\sigma$, $B$, and $\eta$.
2. Optimal Curve Estimation: Employing techniques like bootstrapping and isotonic regression to robustly identify the optimal performance achievable for each $\sigma$ and the corresponding optimal $B^*$ and $\eta^*$.
3. Power Law Fitting: Fitting the proposed power law models to the empirical data relating $D_J$, $C_J$, $B^*$, $\eta^*$, and $\sigma^*$.
4. Extrapolation: Validating the fitted models by demonstrating their ability to accurately predict resource requirements ($D_J$, $C_J$) or optimal hyperparameters ($\sigma^*, B^*, \eta^*$) for significantly larger scales (higher data/compute budgets, higher target performance levels) than those used for model fitting. This extrapolation capability is crucial for practical utility.

The validation results (e.g., Figure 1 in the paper) show strong predictive accuracy when extrapolating resource requirements and optimal settings, confirming the predictability of the scaling behavior.

Practical Implementation Considerations

These findings offer a practical methodology for scaling value-based deep RL experiments more efficiently:

1. Small-Scale Probing: Conduct initial experiments at smaller scales, systematically varying the UTD ratio $\sigma$ and performing nested sweeps over batch size $B$ and learning rate $\eta$ for each $\sigma$.
2. Scaling Law Estimation: Use the data from these small-scale runs to fit the power law models described above, establishing the relationships $D_J(\sigma)$, $C_J(\sigma)$, $B^*(\sigma)$, $\eta^*(\sigma)$, and potentially $\sigma^*(F)$.
3. Prediction for Target Scale: Based on the target (e.g., a desired performance level $J$, a total budget $F$, or available data $D$ / compute $C$), use the fitted laws to:
   • Predict the required $D$ and $C$ along the Pareto frontier for target performance $J$.
   • Predict the optimal $\sigma^*$ for a given budget $F$.
   • Predict the optimal $B^*$ and $\eta^*$ corresponding to the chosen or predicted $\sigma$.
4. Resource-Aware Configuration: Select the operating point ($\sigma$) on the Pareto frontier based on practical constraints. If data collection is expensive, choose a higher $\sigma$ (trading data for compute). If computation is the bottleneck, choose a lower $\sigma$ (trading compute for data).
5. Run Large-Scale Experiment: Execute the large-scale experiment using the predicted optimal hyperparameters ($\sigma^*, B^*, \eta^*$).

This approach replaces brute-force hyperparameter search at the target scale with a more principled prediction based on identified scaling laws, potentially saving significant resources. It highlights the UTD ratio $\sigma$ as a central hyperparameter governing the data-compute trade-off and mandates co-adaptation of $B$ and $\eta$ according to $\sigma$.

Conclusion

In summary, the research presented in "Value-Based Deep RL Scales Predictably" (2502.04327) provides compelling evidence that the scaling of value-based deep RL algorithms is not inherently pathological but follows predictable laws. By characterizing the data-compute Pareto frontier controlled by the UTD ratio and establishing predictable relationships for optimal hyperparameters ($B^*, \eta^*$), the work offers a practical framework for predicting resource requirements and optimizing configurations for large-scale applications. This enables more efficient resource allocation and hyperparameter tuning, moving towards a more systematic approach to scaling deep reinforcement learning.