Reward Function-Based Optimization

• Reward function-based optimization is a framework where decision functions are tuned to maximize a quantitative reward signal using gradient-based techniques.
• It exploits the structure of candidate functions, such as loop-free programs, to efficiently propagate gradient signals and reduce sample complexity.
• This approach offers theoretical guarantees and empirical benefits, outperforming traditional reinforcement learning and black-box optimization methods.

Reward function-based optimization denotes the class of methodologies where system behaviors or policy parameters are directly tuned to maximize an objective encoded as a reward function. Rather than explicitly programming the optimal policy, designer intervention is focused on providing (or learning) a quantitative measure of performance, correctness, or utility, which may be accessible as a black-box or structured program. The practical challenge is then to synthesize functions, heuristics, or control policies that maximize expected reward over input distributions or system runs. This paradigm generalizes beyond standard reinforcement learning settings by allowing the synthesizer to leverage the structure or properties of the reward function, often through continuous or gradient-based optimization techniques. Recent advances, especially as formalized in "Programming by Rewards" (Natarajan et al., 2020), position reward function-based optimization as a scalable alternative to both manual code optimization and reinforcement learning approaches with monolithic or inefficient exploration.

1. Formal Frameworks and Problem Definition

In reward function-based optimization, the central entity is the reward function $r$, a black-box or executable function that quantifies the quality of a system output or the result of a decision function. The optimization objective is to synthesize a function $f$ (e.g., a program, a policy, a heuristic) such that

$$f^* \in \arg\max_{f \in \mathcal{F}} \mathbb{E}[r(f(x))],$$

where $x$ denotes inputs or system features, $f$ is chosen from a class $\mathcal{F}$ (often constrained for interpretability or efficiency), and the expectation averages over relevant distributions (possibly stochastic evaluations of $r$).

The "Programming by Rewards" (PBR) framework (Natarajan et al., 2020) instantiates this by:

• Defining $\mathcal{F}$ as a domain-specific language (DSL) of loop-free if–then–else programs ("Imp") that allow branching on linear input features and linear computations in leaves.
• Treating $r$ as a black-box function, only accessible via queries.
• Seeking to maximize $\mathbb{E}[r(f(x))]$ by efficiently searching the parameterized program space.

This formalization is not limited to reinforcement learning or program synthesis: it includes any setting where a structured, parametrized function must be found to optimize a reward accessible only through evaluations.

2. Structural Exploitation and Optimization Techniques

A key technical advance in reward function-based optimization is the use of problem and function class structure to mitigate sample inefficiency. Rather than treating $f$ as a black-box, optimization leverages knowledge of its parameterization:

• For Imp programs, parameters may appear in both conditional (branching thresholds) and linear computations at leaves.
• Continuous optimization (e.g., gradient descent updates) can be used, despite $r$ being a black box, by estimating gradients via randomized perturbation techniques (bandit/zeroth-order optimization).
• When $f$ has structure (e.g., is a shallow decision tree or linear function), the gradient signal can be efficiently "pushed" through the tree, leading to update/sample complexity that depends only on the number of function outputs ("decisions" $m$), not the total parameter count $d$.

Concretely, the method operates as follows:

• For each parameter update, estimate the effect of small perturbations in $f$'s parameters on the expected reward by random sampling.
• Apply a chain rule through the Imp program tree, exploiting linearities and known branching structure to efficiently propagate the estimated gradients.
• Use batch/offline "two-point" gradient estimators to accelerate convergence when possible.

This sharply contrasts with flat black-box optimization (where all parameters must be estimated separately), and with reinforcement learning approaches that ignore or are agnostic to the internal compositional structure of the decision function.

3. Theoretical Guarantees and Complexity

When the reward function $r$ is concave and Lipschitz continuous, strong sample complexity results can be established:

• For the case where $f$ is a constant decision, known bandit convex optimization results (e.g., Flaxman et al., 2005) guarantee regret after $T$ queries is $R_T \leq C \cdot m / T^{1/4}$; reaching $\epsilon$-optimality requires $O(m^4 / \epsilon^4)$ queries.
• If $f$ utilizes the available structure (e.g., as a linear function or tree), the convergence rate depends only on the output dimension $m$, not the parameter dimension $d$.
• Two-point estimators in offline settings further accelerate convergence.

These results imply that structure-exploiting approaches may be applicable to high-dimensional parameter spaces if the effective "decision complexity" is low.

4. Empirical Performance and Practical Integration

Empirical validation in (Natarajan et al., 2020) includes both real-world case studies and synthetic benchmarks, with integration into the PROSE program synthesis framework. Notable findings:

• In tasks such as optimizing ranking heuristics in RegexPair and FormatDateTimeRange, PBR achieves competitive performance to hand-tuned code after approximately 250 reward calls, each of which evaluates on hundreds of tasks, indicating strong sample efficiency.
• Against discrete-action reinforcement learning baselines (e.g., UCB on discretized parameter grids), PBR vastly outperforms in both solution quality and learning rate, as the baseline suffers exponentially in the number of discretized parameters.
• Structure-exploiting optimization (e.g., tree learning for piecewise threshold functions) is shown to require an order of magnitude fewer reward queries than black-box parameter tuning.

The system's interpretability is highlighted: synthesized decision functions in Imp are routine, human-readable, and amenable to validation/debugging, in contrast to neural network-based black-box policies.

5. Applications, Scope, and Implications

The PBR framework—and reward function-based optimization more generally—has robust applications:

• System heuristics tuning (search, ranking, threshold selection) in platforms like PROSE, where expert-written code is the traditional baseline.
• Adaptively synthesizing decision rules that respond to changing workloads or benchmarks.
• Any domain where nonfunctional properties (e.g., latency, efficiency, accuracy) are encoded as quantitative reward signals, but the exact mapping from context to action is not analytically tractable.

Broader implications:

• PBR bridges the spectrum between white-box synthesis (requiring full system access) and black-box, sample-inefficient RL; it treats the decision function as structured, while leaving the reward as opaque.
• Generated code is both high-performing and interpretable—critical for maintainability and transparency in performance-sensitive systems.
• These methods are particularly valuable when exhaustive access to the reward or underlying environment is infeasible, and in cases where the decision class matches the simple programmatic forms usually written and tuned by domain experts.

6. Limitations and Design Choices

Reward function-based optimization assumes that:

• The class of candidate functions (e.g., loop-free Imp programs) is expressive enough to contain near-optimal solutions for the task.
• The reward function can be efficiently queried, even if expensive per call, as the method requires repeated evaluations.
• The reward function is well-behaved (e.g., concave, Lipschitz) to guarantee sample efficiency and convergence.

Though powerful, it does not substitute for full RL or global optimization in cases with highly non-smooth, discontinuous rewards or when the function structure cannot be well exploited. The approach's effectiveness hinges on the DSL being a "sweet spot"—expressive yet amenable to continuous optimization.

7. Comparison to Baselines and Theoretical Positioning

Reward function-based optimization as realized in PBR significantly outperforms:

• Black-box bandit optimization approaches, which fail to scale in parameter dimensions.
• Reinforcement learning with discretized action spaces, which rapidly becomes intractable due to exponential combinatorial explosion.
• White-box program synthesis/sketching methods, which require full reward function visibility, often lack scalability, and incur higher computational overhead.

By using a constrained DSL and leveraging gradient estimation tailored to program structure, PBR achieves a practical balance between expressiveness, efficiency, and interpretability, substantiated both by theoretical analysis and empirical evaluation (Natarajan et al., 2020).