Telescoping Reward in Tool Composition

• Telescoping reward is a cumulative reward function that decomposes supervision into format, parsing, execution, and answer components, ensuring dense feedback in multi-step trajectories.
• It aggregates verifiable checks across API calls to mitigate sparse rewards and path-dependence, exemplified by the TIER framework.
• Empirical results demonstrate that telescoping rewards enable stable deep-task learning with high accuracy, outperforming traditional sparse and trajectory-supervised methods.

Telescoping reward refers to a finely decomposed and cumulatively aggregated reward function that provides structured, step-wise supervision signals for multi-step tool-use trajectories in reinforcement learning (RL). In the context of LLMs operating over compositional sequences of API/tool invocations, telescoping reward frameworks are designed to mitigate the sparse feedback problem inherent in traditional outcome-based rewards and the path-dependence of trajectory-matching supervision. The TIER (Trajectory-Invariant Execution Rewards) framework constitutes a concrete and well-documented instantiation of telescoping rewards, offering dense, interpretable sequence-level feedback derived from verification of individual decision points throughout the trajectory (Kulkarni et al., 16 May 2026).

1. Mathematical Structure of Telescoping Rewards in Multi-Step Tool Composition

Let a multi-step tool-use trajectory be a sequence of $N$ API calls, represented as an abstract syntax tree (AST) $y$. A telescoping reward framework, as exemplified by TIER, assigns a normalized scalar reward $R_\text{total}(y) \in [0, 1]$, decomposed into distinct components:

$$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$

This sum lies in an unnormalized range of $[0, 10]$ ($[0,1]$ for format, $[0,3]$ for parse, $[0,1]$ for execution, $[0,5]$ for answer), and is rescaled to $[0,1]$ prior to policy optimization. The reward does not assign a scalar $y$0 per API step but each component itself aggregates verifiable step-level feedback, telescoping granular sub-rewards into a single scalar at the trajectory level.

2. Formal Specification of Reward Components

Each TIER reward component is defined as follows, enforcing step-wise aggregation by design:

• Format Validity: $y$1, where $y$2 iff the trajectory AST parses as well-formed JSON/XML—failure in format zeroes all subsequent components.
• Schema Adherence (Parsing):

$y$3

• $y$4 iff all API call names match the schema.
• $y$5
• $y$6
• $y$7.
  • Execution Success: $y$8; $y$9 if all calls return success at runtime; $R_\text{total}(y) \in [0, 1]$0 otherwise (zero-tolerance to failed calls).
  • Answer Correctness: $R_\text{total}(y) \in [0, 1]$1; $R_\text{total}(y) \in [0, 1]$2 if the final output matches ground truth, $R_\text{total}(y) \in [0, 1]$3 otherwise.

All components are computed on the full AST $R_\text{total}(y) \in [0, 1]$4, with each component telescoping stepwise statistics or checks as relevant.

3. Stepwise Aggregation and Telescoping Effect

Telescoping reward refers to the cumulative aggregation of interpretable checks (e.g., format correctness, schema compliance, runtime success) across all API calls in the trajectory. For example, the $R_\text{total}(y) \in [0, 1]$5 component aggregates schema validation over all $R_\text{total}(y) \in [0, 1]$6 steps:

$R_\text{total}(y) \in [0, 1]$7

where $R_\text{total}(y) \in [0, 1]$8 is the indicator for API name validity at step $R_\text{total}(y) \in [0, 1]$9, and $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$0 and $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$1 are clipped scores for parameter and type correctness, respectively. Execution and answer components introduce all-or-nothing or sharply weighted signals that penalize or reward the entire trajectory contingent on success at any point.

This telescoping architecture yields nonzero, shaped feedback throughout the trajectory. An isolated parameter name error at step $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$2 only reduces $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$3 by $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$4, while a runtime error at any step zeros $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$5. Final-answer correctness is weighted most heavily to prevent reward hacking and focus optimization on true end-task performance.

4. Theoretical Motivation and Departures from Conventional Reward Schemes

Classical RL return, $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$6, admits sparse rewards near $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$7 for outcome-based RL, or pseudo-rewards $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$8 via reward shaping for improved credit assignment. Trajectory-supervised approaches rely on matching the entire action sequence to a reference, penalizing viable alternative solutions and collapsing in the presence of combinatorial path diversity.

Telescoping reward, as formulated in TIER, instead constructs all intermediate signals from verifiable, environment-grounded checks—not from reference trajectories—yielding the following properties:

• Trajectory invariance: All valid solution paths (including different API-call orderings or equivalent compositions) receive maximum reward.
• Graded penalty over error modes: Different types and localizations of errors are reflected distinctly in $$R_\text{total}(y) = R_\text{format}(y) + R_\text{parse}(y) + R_\text{exec}(y) + R_\text{answer}(y)$$9.

Such a scheme adheres to potential-based reward shaping, preserving optimal solutions, while dramatically improving optimization signal density and credit assignment for deep compositional problems (Kulkarni et al., 16 May 2026).

5. Illustrative Example and Quantitative Reward Breakdown

For a 3-step DepthBench task (“Get the weather at my flight’s destination”), a valid trajectory with correct API names, parameters, and execution yields:

• $[0, 10]$0
• $[0, 10]$1, $[0, 10]$2, $[0, 10]$3 so $[0, 10]$4
• $[0, 10]$5 (all calls succeed)
• $[0, 10]$6 (final output correct)
• $[0, 10]$7, $[0, 10]$8

A single error, e.g. a parameter mismatch on step 1 ($[0, 10]$9), results in $[0,1]$0, $[0,1]$1, $[0,1]$2, $[0,1]$3. Thus, telescoping reward provides graded penalties, immediately reflecting the localization and severity of errors in the aggregate scalar.

6. Empirical Performance and Ablation Analyses

On DepthBench (0–6 step compositions), telescoping reward via TIER achieves $[0,1]$490% accuracy for $[0,1]$5–$[0,1]$6 step tasks, outperforming sparse outcome-based RL and trajectory-supervised reward, both of which collapse at greater depths. Detailed results:

Method	Overall	0-step	1-step	2-step	3-step	4-step	5-step	6-step
Simple	66.2	98.0	94.4	1.3	3.3	0.0	0.0	0.0
ToolRL	67.5	92.2	90.0	14.7	17.3	24.4	0.0	0.0
TIER	98.6	98.0	99.4	97.9	100.0	93.3	90.0	90.0

Reward component ablations show that removing parsing, execution, or answer signals causes marked performance collapse at 5–6 steps, indicating that all components are required for stable deep-task learning:

Reward used	Overall	6-step
format+answer only	66.2	0.0
+execution only	63.6	0.0
+parsing only	71.3	0.0
Full TIER	98.6	90.0

Transfer experiments on BFCL v3 and NestFUL corroborate TIER’s superiority over alternatives, with notably higher exact-match scores and robust performance across model scales.

7. Significance and Implications

Telescoping reward, as structurally implemented in TIER, establishes a paradigm for dense, step-aware supervision in hierarchical task construction without reliance on gold-standard trajectories. By directly aggregating environment-verifiable properties (syntactic well-formedness, schema conformance, runtime execution, and semantic correctness), it enables stable and scalable policy improvement in settings where both outcome-only and path-anchored reward signals are insufficient. This framing is particularly salient for real-world tool-use agents, multi-step program synthesis, and modular LLM-in-the-loop control tasks, facilitating credit assignment and convergence in the face of extensive solution diversity and compositional depth (Kulkarni et al., 16 May 2026).