---
title: 'Timely-RL: Real-Time Adaptive Reinforcement Learning'
url: https://www.emergentmind.com/topics/timely-rl
type: topic
---

# Timely-RL: Real-Time Adaptive Reinforcement Learning

A Timely-RL system is a reinforcement learning (RL) methodology designed to enable agentic systems—such as large language models (LLMs) with tool-use capabilities—to operate optimally under hard real-time constraints. Unlike standard RL protocols for reasoning agents, Timely-RL is constructed to directly train agents for wall-clock time-awareness, strategic adaptation to latency, and deadline-sensitive action planning. The paradigm shifts the definition of “test-time” from generation length or FLOPs to true elapsed time, demarcating a class of RL agents that maximize task performance subject to dynamic, unpredictable external delays.

## 1. Wall-Clock Test-Time Formalization

Timely-RL begins with a formal redefinition of test-time in agentic environments where frequent tool calls create variable, exogenous latency. The cumulative elapsed time for $N$ reasoning steps is modeled as:
\[
t_\text{all} = \sum_{i=1}^N t_\text{gen}^{(i)} + \sum_{i=1}^N t_\text{tool}^{(i)}
\]
where $t_\text{gen}^{(i)}$ denotes in-model generation time and $t_\text{tool}^{(i)}$ is tool or environment latency at step $i$ [2601.16486].

Agents are evaluated under a fixed budget $T_b$ such that generating the answer after $t_\text{all} > T_b$ constitutes a failure, regardless of token count. Adaptive policy optimization must thus ingest real-time latency feedback to decide not only “what” to generate, but “when” and “how long” to engage external modules.

## 2. Timely-Eval Benchmark and Regime Sensitivity

Timely-RL is empirically grounded in Timely-Eval, a benchmark suite that probes time-budget adaptation across heterogeneous task classes:
- **Interactive Games:** Jericho text adventures, synthetic latencies injected into tool calls (0, 2, 10, 50s).
- **ML Tasks:** MLEBench-Lite problems; code execution with variable wall-clock delays.
- **General Reasoning:** MATH, AIME, GPQA-Diamond under strict real-time cutoffs [2601.16486].

Three latency regimes are characterized by the ratio $m_i = t_\text{tool}^{(i)} / t_\text{gen}^{(i)}$:
- **Tool-dominated ($m_i \gg 1$):** Planning prioritizes fewer, higher-quality tool interactions.
- **Generation-dominated ($m_i \ll 1$):** Strategy shifts toward brevity and more frequent calls.
- **Balanced ($m_i \approx 1$):** Requires dynamic depth-breadth trade-off.

## 3. Timely-RL Training Procedure

Timely-RL is a two-stage process:

### 3.1 Supervised Cold-Start Fine-Tuning (SFT)
An initial policy is distilled from time-budget-aware teacher traces (e.g., Qwen3-235B-Instruct), using explicit wall-clock calls (get_duration) and step-wise reasoning sequences (~1M examples across diverse tasks). This injects primitive awareness of time constraints and tool typologies into the base model.

### 3.2 RL with Wall-Clock Reward Shaping
Subsequent RL optimization utilizes a wall-clock reward signal:
\[
R(t, r) = 
\begin{cases}
0 & t > T_{\max} \\
r_f & t \le T_{\max},\ r = 0 \\
r_f + r + \lambda U(t) & t \le T_{\max},\ r > 0
\end{cases}
\]
Where $r$ represents task performance (e.g., normalized score, accuracy), $r_f$ is an on-time completion bonus, and $U(t) = \sin((\pi/2)\cdot \min(t/T_{\max}, 1))$ encourages full, but not overshot, utilization of the budget. A $\lambda$ coefficient mediates efficiency–performance trade-offs. This reward is maximized via policy gradient updates derived from the VeRL framework, a PPO-style algorithm with clipped importance weights [2601.16486].

## 4. Policy Optimization and Inference Flow

The agent’s state at each step incorporates:
- Current prompt and cumulative history (including prior actions and tool outputs)
- Per-turn latency feedback from external calls
- Internal LLM hidden state

Actions comprise reasoning token generation and/or tool invocation. At every turn, the agent receives feedback on elapsed time and must dynamically plan the depth and tempo of its reasoning steps. Inference proceeds until $t_\text{all} \ge T_b$ or a terminal answer token is output.

## 5. Empirical Properties and Advantages

Timely-RL systems exhibit several notable empirical behaviors:
- **Latency-Sensitivity:** Smaller, faster LLMs outperform larger ones when tool latency is low; at high latency, the largest model dominates due to superior answer quality per tool interaction.
- **Adaptive Reasoning Lengths:** Reasoning trace length increases monotonically with time budget $T_b$ for Timely-RL agents; prior models show little adaptation.
- **On-Time Completion and Accuracy:** Timely-RL-trained models consistently exceed SFT-only baselines on tight deadlines (e.g., on-time success rates rise 5–8% absolute on constrained MATH/GPQA/AIME benchmarks).
- **Task-Specific Scaling:** Interactive environments benefit linearly from additional time, whereas ML tasks saturate quickly.

## 6. Broader Implications, Open Limitations, and Future Directions

Timely-RL recasts test-time as the primary optimization resource for agentic LLM deployments where unpredictability of API or tool latency is the norm. Moving away from token-length metrics to elapsed seconds enables robust, deadline-focused reasoning and practical integration into real-world tool-chaining workflows. Key limitations include exclusive focus on text tasks, lack of multi-agent negotiation, and open questions about richer risk-reward balancing for high-stakes domains.

Potential future directions include:
- Extension to multimodal Timely-RL (vision, audio).
- Hierarchical subtask planning with nested time constraints.
- Adversarial latency robustness and uncertainty-aware dynamic budgeting [2601.16486].

Timely-RL thus defines a class of wall-clock-adaptive RL algorithms for deadline-constrained, agentic reasoning environments, underlining the centrality of time as the operative resource in next-generation interactive AI systems.

Source: https://www.emergentmind.com/topics/timely-rl