Complexity-Boosting Reinforcement Learning (CBRL)

• CBRL is a reinforcement learning design pattern that embeds structured complexity through adaptive curricula, weak-to-strong policy aggregation, and contextual scaffolding.
• It leverages techniques like gradient boosting over Bellman residuals and auxiliary tasks to improve sample efficiency and tackle sparse-reward, long-horizon problems.
• Empirical studies, such as in geometry proving tasks and grid-world simulations, demonstrate significant performance gains with CBRL methods compared to conventional RL approaches.

Searching arXiv for the papers and terminology to ground the article in the cited literature. Complexity-Boosting Reinforcement Learning (CBRL) denotes a family of reinforcement-learning ideas in which learning is made more effective on difficult problems by introducing additional structure into the optimization process, the policy class, the exploration mechanism, or the training distribution. In the narrowest and most literal usage, the phrase is the explicit name of the curriculum method used to train InternGeometry on synthesized geometry tasks of progressively adjusted DDAR proof length (Zhao et al., 11 Dec 2025). In a broader interpretive sense, closely related work treats “boosting” as additive growth of value-function complexity, weak-to-strong policy aggregation, complexity-aware exploration, symbolic or contextual scaffolding, and auxiliary simplifications of otherwise hard RL problems (Abel et al., 2016, Brukhim et al., 2021). The acronym itself is not uniform across the literature: “CBRL” also denotes “Context Bootstrapped Reinforcement Learning” and “Chaos-based reinforcement learning” in distinct papers (Agashe et al., 19 Mar 2026, Matsuki et al., 2024).

1. Terminological scope and taxonomy

A common misconception is that CBRL names a single, settled algorithmic paradigm. The cited literature suggests instead that the label is heterogeneous, and that “boosting” refers to different objects in different subfields.

Usage	Representative paper	What is boosted
Complexity-Boosting Reinforcement Learning	“Achieving Olympia-Level Geometry LLM Agent via Complexity Boosting Reinforcement Learning” (Zhao et al., 11 Dec 2025)	Training-task complexity via adaptive curriculum
Context Bootstrapped Reinforcement Learning	“Context Bootstrapped Reinforcement Learning” (Agashe et al., 19 Mar 2026)	Early exploration in RLVR via annealed few-shot context
Chaos-based reinforcement learning	“Chaos-based reinforcement learning with TD3” (Matsuki et al., 2024)	Exploration through internal chaotic dynamics
Broad precursor / interpretive sense	“Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains” (Abel et al., 2016)	Representational complexity through additive residual learners

The same ambiguity appears in papers that are highly relevant to a broad CBRL reading without using the phrase literally. Some works boost representational capacity by adding weak learners episode by episode; some boost policy optimization by aggregating weak policy learners; some reduce effective complexity by transfer learning, auxiliary short-delay tasks, or predictive shaping; and some raise effective capability by injecting symbolic or contextual priors (Brukhim et al., 2021, Mu et al., 2021, Wu et al., 2024). A neutral encyclopedia treatment therefore has to distinguish the exact name from the broader methodological pattern.

2. Boosting as function-space growth and weak-to-strong policy aggregation

One influential precursor is GEQL, introduced in “Exploratory Gradient Boosting for Reinforcement Learning in Complex Domains” (Abel et al., 2016). GEQL works in the standard discounted model-free setting with objective

$\sum_{t=1}^{\infty} \gamma^{t-1} r_t,$

and Bellman optimality equation

$$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$

Its central mechanism is an additive $Q$-approximation

$\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$

initialized at $\hat Q_0(s,a)=0$, where each $h_j$ is a weak regressor. After each episode, the method fits a regressor to the Bellman residual

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$

and updates

$\hat Q \leftarrow \hat Q+\alpha_t h.$

This is boosting in the literal gradient-boosting sense: representational complexity grows stage by stage as new weak learners are appended. The paper pairs this with IAUU exploration, using a state-collapsing function $\phi:\mathcal S\to\{1,\dots,m\}$, cluster-action counts $M(c,a)$, and a Gibbs exploration rule

$$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$0

A plausible CBRL interpretation is that GEQL boosts complexity primarily in the representational sense: the value function starts simple and becomes more expressive as training proceeds.

A different line of work boosts policies rather than value approximators. Brukhim, Hazan, and Singh reduce reinforcement learning to a sequence of weak learning problems in “A Boosting Approach to Reinforcement Learning” (Brukhim et al., 2021). Their method uses an outer non-convex Frank–Wolfe or conservative-policy-iteration-style loop,

$$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$1

and an inner boosting loop that converts weak supervised learners into an approximate linear optimizer over policy space. The final policy is improper relative to the base class $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$2, and is represented as a policy tree or two-layer neural network over base policies. The main guarantees are stated for binary-action discounted MDPs and take the form

$$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$3

in the episodic model, with an analogous $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$4-reset bound involving $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$5 and $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$6. In this literature, “boosting” refers to weak-to-strong policy aggregation and to progressive growth of effective policy-class expressivity.

Subsequent agnostic-boosting work sharpens the same reduction. “Sample-Efficient Agnostic Boosting” improves the supervised boosting primitive and, when plugged into the Brukhim–Hazan–Singh reduction, improves RL sample complexity for binary-action discounted MDPs from $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$7 to $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$8 in the episodic model and from $$Q^\star(s,a)=\mathbb{E}\!\left[r+\gamma \max_{a'}Q^\star(s',a')\right].$$9 to $Q$0 with $Q$1-reset access (Ghai et al., 2024). “Sample-Optimal Agnostic Boosting with Unlabeled Data” then shows that, in the corresponding RL reduction with reward-free trajectories, the expensive channel is the reward-labeled episode budget rather than total interaction alone (Ghai et al., 6 Mar 2025). In this branch of the literature, complexity is boosted less by enlarging a network directly than by aggregating weak learners into a stronger policy improver with better sample-oracle tradeoffs.

3. Curriculum-defined CBRL in geometry

The paper that explicitly names the method “Complexity-Boosting Reinforcement Learning” is “Achieving Olympia-Level Geometry LLM Agent via Complexity Boosting Reinforcement Learning” (Zhao et al., 11 Dec 2025). Its setting is not generic control but long-horizon geometry proving by an LLM agent, InternGeometry, built on InternThinker-32B and coupled to a symbolic geometry engine, InternGeometry-DDAR. The agent alternates between natural-language “Think” steps and formal DSL “Action” steps, receives symbolic feedback, and maintains a compressed dynamic memory that supports more than two hundred interactions with the symbolic engine per problem.

Here CBRL is a multi-stage curriculum RL pipeline over synthesized geometry tasks. Task complexity is denoted $Q$2 and is defined by DDAR proof step count or DDAR proof length. The policy is trained on problems drawn from $Q$3, while the curriculum chooses the target complexity by maximizing expected absolute advantage: $Q$4 The appendix gives the binary-reward analysis

$Q$5

which is maximized at $Q$6, yielding the paper’s operational rule that training should stay near the moderate-difficulty regime. In the actual curriculum loop, if the batch-average reward exceeds $Q$7, complexity is increased by $Q$8; otherwise it is decreased by $Q$9.

The reward is deliberately simple: $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$0 The outcome reward $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$1 is $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$2 only when the proof is complete. The step effectiveness reward $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$3 is $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$4 only for propositions successfully proven by the engine or for constructions that are both accepted and used in the final proof. The result is a trajectory-level credit rule that rewards effective steps in successful trajectories while penalizing ineffective steps and failed trajectories.

Empirically, this exact CBRL instantiation is tied to the strongest claims in the corpus. InternGeometry solves $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$5 of $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$6 IMO geometry problems from 2000–2024, exceeding the average gold medalist score of $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$7, using only $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$8K training examples, reported as $\hat Q_t(s,a)=\sum_{j=1}^t \alpha_j h_j(s,a),$9 of the data used by AlphaGeometry 2 (Zhao et al., 11 Dec 2025). The ablation study isolates the curriculum itself: full CBRL achieves $\hat Q_0(s,a)=0$0, whereas “SFT Cold Start” gives $\hat Q_0(s,a)=0$1, “Easy Data Only” $\hat Q_0(s,a)=0$2, “Challenging Data Only” $\hat Q_0(s,a)=0$3, and “Same Data without Schedule” $\hat Q_0(s,a)=0$4. In this narrow sense, CBRL is an adaptive complexity curriculum whose purpose is to keep sparse-reward training in the regime that best matches current policy capability.

4. Scaffolding hard RL with auxiliary tasks, symbolic priors, and contextual support

Several papers instantiate the same broad idea without using the exact phrase “Complexity-Boosting Reinforcement Learning.” “Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delays” introduces AD-RL, which treats long observation delay as the main source of hardness and constructs an auxiliary task with shorter delay $\hat Q_0(s,a)=0$5 (Wu et al., 2024). The original delayed state is

$\hat Q_0(s,a)=0$6

while the auxiliary task uses

$\hat Q_0(s,a)=0$7

AD-RL learns $\hat Q_0(s,a)=0$8 on the easier short-delay problem and uses it either to bootstrap long-delay $\hat Q_0(s,a)=0$9-updates or to improve the long-delay policy. The paper argues that the sample-complexity gain scales as $h_j$0, while also showing that making $h_j$1 too small can introduce approximation bias in stochastic environments. This is a clear scaffolded-complexity design: an easier sibling task accelerates learning on a harder one.

“Boosting deep Reinforcement Learning using pretraining with Logical Options” introduces H$h_j$2RL, a two-stage hybrid hierarchical framework in which logical options and a differentiable symbolic logic manager shape exploration and policy learning during pretraining, after which the symbolic machinery is discarded and the final policy is refined as a standard neural policy (Ye et al., 6 Mar 2026). The hybrid policy is

$h_j$3

where $h_j$4 is induced by symbolic option selection and $h_j$5 is the neural policy. The paper’s broader lesson is that complexity can be added as semantic abstraction, temporal abstraction, and staged learning, then compiled into neural parameters.

In RLVR, “Context Bootstrapped Reinforcement Learning” uses temporary in-context scaffolding rather than symbolic structure (Agashe et al., 19 Mar 2026). A few-shot bank $h_j$6 is prepended to some training prompts with probability $h_j$7, where the injection probability follows a linear curriculum that starts high and anneals to zero: $h_j$8 The key claim is that demonstrations increase the probability of successful rollouts early in training, then disappear so that reasoning patterns must be internalized into the weights. This is “boosting” in the sense of temporary contextual support for sparse-verifier optimization rather than additive model growth.

Other scaffolding methods target reward structure or computation budgets. “Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards” first learns CPC representations offline, then shapes reward either by latent clustering or by the dense term

$h_j$9

arguing that latent geometry better reflects environment dynamics than raw observations (Lu et al., 2019). “Boosting the Convergence of Reinforcement Learning-based Auto-pruning Using Historical Data” uses transfer learning, augmented transfer learning, and assistant learning to reuse historical pruning traces and accelerate RL-based auto-pruning by $$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$0 on ResNet20 and $$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$1 on ResNet56, ResNet18, and MobileNet v1 (Mu et al., 2021). These methods do not all increase task difficulty; some instead reduce search complexity, improve transferability, or densify otherwise sparse objectives.

5. Exploration, uncertainty, and dynamic complexity

A different cluster of papers treats complexity as a property of exploration itself. “Chaos-based reinforcement learning with TD3” defines CBRL as a framework in which exploration comes from internally generated chaotic dynamics rather than external random noise (Matsuki et al., 2024). The actor is an Echo State Network with reservoir dynamics

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$2

and action readout

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$3

The TD3-CBRL variant removes both action exploration noise and target smoothing noise, setting $$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$4 and $$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$5. Empirically, the agent can suppress exploratory behavior as learning progresses and resume exploration when the environment changes, but excessively large chaoticity harms flexible switching between exploration and exploitation.

In continuous-time model-based RL, COMBRL uses epistemic uncertainty as the complexity signal (Iten et al., 28 Oct 2025). The unknown system evolves according to

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$6

and COMBRL plans under an uncertainty-aware model by maximizing

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$7

The same framework covers reward-driven and unsupervised settings, with Gaussian processes, Bayesian neural networks, or ensembles providing uncertainty estimates. The paper proves sublinear regret in the reward-driven setting and a sample-complexity bound in the unsupervised setting. Under a broad CBRL interpretation, what is being boosted is exploration pressure toward dynamically complex, poorly modeled regions.

Papadimitriou and Peng provide the main worst-case complexity result in “The complexity of non-stationary reinforcement learning” (Papadimitriou et al., 2023). In their finite-horizon model, modifying the reward or transition probabilities of a single existing state-action pair can force amortized update time

$$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$8

under SETH, even just to maintain an $$\delta_i=r_i+\gamma \max_{a'}\hat Q(s_{i+1},a')-\hat Q(s_i,a_i),$$9-approximation of the optimal start-state value. By contrast, adding a new action without modifying old ones admits an approximate maintenance algorithm with amortized runtime

$\hat Q \leftarrow \hat Q+\alpha_t h.$0

This yields an important theoretical distinction: some tiny local edits are genuine complexity boosters, whereas insertion-only changes preserve monotonic structure and are much easier to handle.

6. Empirical regimes, misconceptions, and limitations

Across the corpus, CBRL-style methods are most persuasive in domains where flat RL suffers from sparse reward, partial observability, long horizons, or weak heuristics. GEQL is only mildly differentiated on Blackjack and $\hat Q \leftarrow \hat Q+\alpha_t h.$1-Chain, but on Minecraft’s Visual Grid World the booster with IAUU reaches average reward $\hat Q \leftarrow \hat Q+\alpha_t h.$2, close to the optimal $\hat Q \leftarrow \hat Q+\alpha_t h.$3, whereas the same booster with uniform exploration gets $\hat Q \leftarrow \hat Q+\alpha_t h.$4 and the best batchboost baseline with uniform exploration is around $\hat Q \leftarrow \hat Q+\alpha_t h.$5; in Visual Hill Climbing, only the gradient booster shows non-negligible learning and IAUU helps further (Abel et al., 2016). H$\hat Q \leftarrow \hat Q+\alpha_t h.$6RL produces especially large long-horizon gains on Kangaroo and DonkeyKong and shows that symbolic options, logic rules, and staged pretraining matter more than simply appending symbolic inputs to PPO (Ye et al., 6 Mar 2026). Context Bootstrapped RL improves RLVR success across two model families and five Reasoning Gym tasks, and on Q programming raises average test-pass rate from $\hat Q \leftarrow \hat Q+\alpha_t h.$7 to $\hat Q \leftarrow \hat Q+\alpha_t h.$8 and success rate from $\hat Q \leftarrow \hat Q+\alpha_t h.$9 to $\phi:\mathcal S\to\{1,\dots,m\}$0 (Agashe et al., 19 Mar 2026).

The same evidence also marks clear limitations. Many methods are strongly domain-specific: auto-pruning assumes sequential layerwise decisions and reusable pruning trajectories; H$\phi:\mathcal S\to\{1,\dots,m\}$1RL depends on symbolic state extraction, handcrafted option libraries, and logic rules; InternGeometry depends on the expressivity of InternGeometry-DDAR and on DDAR proof length as a proxy for task difficulty (Mu et al., 2021, Ye et al., 6 Mar 2026, Zhao et al., 11 Dec 2025). Early boosting-style RL methods often assume small discrete action spaces, engineered visual features, or online episodic batch-style regression rather than fully incremental stochastic TD updates (Abel et al., 2016). Chaos-based exploration has so far been demonstrated mainly on a simple $\phi:\mathcal S\to\{1,\dots,m\}$2 goal-reaching task with low-dimensional handcrafted observations, not on Atari or MuJoCo-scale benchmarks (Matsuki et al., 2024).

A second misconception is that “boosting” always means the same thing. The literature shows otherwise. In some papers it means literal gradient boosting over Bellman residuals; in some it means aggregating weak policy learners; in some it means annealed demonstration support; in some it means symbolic or delay-reduced scaffolding; and in one paper it means an explicit curriculum over synthesized task complexity (Abel et al., 2016, Brukhim et al., 2021, Wu et al., 2024, Zhao et al., 11 Dec 2025). A third misconception is that complexity boosting always makes tasks harder. Several methods instead reduce effective search complexity, reuse old trajectories, or replace flat sparse-reward optimization with shaped or structured surrogates (Lu et al., 2019, Mu et al., 2021).

Taken together, the literature suggests that CBRL is best treated not as a single algorithm but as a design pattern. Its central move is to add just enough structure—representational, curricular, hierarchical, contextual, auxiliary, or epistemic—to make otherwise brittle RL optimization learnable. The exact object being “boosted” differs across papers, but the recurring technical theme is the same: hard RL problems often become tractable when complexity is introduced in a controlled, staged, or uncertainty-aware form rather than left to emerge from flat end-to-end search alone.