---
title: 'BoreaRL: Climate-Adaptive Boreal Forest RL'
url: https://www.emergentmind.com/topics/borearl
type: topic
---

# BoreaRL: Climate-Adaptive Boreal Forest RL

Searching arXiv for the specified paper and closely related work on multi-objective RL and climate/forest management.
arxiv_search(query="BoreaRL multi-objective reinforcement learning climate-adaptive boreal forest management", max_results=5, sort_by="submittedDate")
BoreaRL is a multi-objective reinforcement learning environment for climate-adaptive boreal forest management, introduced in "BoreaRL: A Multi-Objective Reinforcement Learning Environment for Climate-Adaptive Boreal Forest Management" [2509.19846]. It is presented as the first multi-objective reinforcement learning environment for this domain and is designed around a physically grounded simulator of coupled energy, carbon, and water fluxes. The motivating problem is that boreal forests store 30–40% of terrestrial carbon, much of it in climate-vulnerable permafrost soils, so management decisions involve nontrivial trade-offs between carbon sequestration and permafrost preservation that current tools cannot adequately address [2509.19846].

## 1. Scientific scope and problem setting

BoreaRL is framed around climate-adaptive management in boreal ecosystems where management interventions affect both aboveground carbon accumulation and belowground thermal dynamics. The central premise is that optimizing for carbon sequestration and optimizing for thaw avoidance are not interchangeable goals. In the formulation reported for BoreaRL, these objectives are explicitly separated and studied as a multi-objective decision problem rather than being collapsed into a single proxy target from the outset [2509.19846].

The environment is intended to support two complementary research settings. In **site-specific mode**, a single partially observable Markov decision process is fixed, enabling controlled study of repeatable, location-targeted strategies. In **generalist mode**, environmental conditions are sampled episode by episode over realistic site distributions, so the learning problem becomes one of robustness under environmental stochasticity. This division makes BoreaRL suitable both for algorithmic diagnosis and for studying policy transfer across heterogeneous boreal conditions [2509.19846].

A plausible implication is that BoreaRL occupies an intermediate position between purely stylized RL benchmarks and operational forest simulators: it preserves explicit process-based biophysics while exposing an RL interface with annual management actions and long-horizon credit assignment.

## 2. Simulator architecture and physical formulation

At the core of the environment is **BoreaRL-Sim**, described as a process-based, sub-annual simulator that resolves coupled energy, water, and carbon fluxes on $n$-minute time-steps over a full 365-day year for each annual RL decision [2509.19846]. The simulator spans canopy, trunk, snowpack, surface soil, and deep soil layers and solves a five-node energy balance,

$$
C_i\,\frac{dT_i}{dt} = \sum F_{in,i}-\sum F_{out,i}\,.
$$

For the canopy, the reported balance includes net radiation, sensible heat, latent heat, a photosynthesis-linked energy term, melt, and canopy–trunk coupling. The component expressions include

$$
R_{net,can}=Q_{\rm solar}(1-\alpha_{can})+\epsilon_{can}(L_{down,atm}+L_{up,ground})-2\epsilon_{can}\sigma T_{can}^4,
$$

$$
H_{can}=h_{can}(T_{can}-T_{air}),
$$

$$
LE_{can}=\alpha_{PT}\tfrac{\Delta}{\Delta+\gamma}R_{net,can}\,f_{VPD}\,f_{SWC},
$$

and

$$
G_{photo}=GPP\cdot J_{\rm per\,gC}.
$$

The last term couples energy to the carbon cycle via light-use efficiency. Similar balances govern trunk, snow, and soil nodes. Conductive flux across the deep soil boundary is then used as the permafrost thaw proxy [2509.19846].

This design is significant because the thaw objective is not represented as a direct static label or heuristic penalty. Instead, it arises from the simulator’s sub-annual thermodynamic evolution. That modeling choice helps explain why the thaw objective is reported to be substantially harder for current multi-objective RL methods: the signal is noisy, delayed, and mediated through coupled physical processes rather than immediate stand-level inventory changes.

## 3. Markov decision process and reward design

BoreaRL-Env wraps BoreaRL-Sim as a partially observable Markov decision process,

$$
(\mathcal S,\mathcal A,\mathcal O,P,R,\gamma).
$$

The observation space depends on the training paradigm. In site-specific mode, the observation is $o_t\in\mathcal O\subset\mathbb R^{43}$. In generalist mode, it is $\mathbb R^{105}$. The observation includes year, stem density, conifer fraction, carbon stocks, age-class histograms, and disturbance history; in generalist mode it additionally includes sampled site parameters $(\phi)$ and a scalar preference weight $w_C\in[0,1]$ [2509.19846].

The action space is discrete:

$$
a_t\in\{0,\dots,24\}.
$$

Each action encodes one of five density changes $\{-100,-50,0,50,100\}$ stems ha$^{-1}$ and one of five conifer-fraction targets $\{0,0.25,0.5,0.75,1\}$, yielding $5\times5=25$ maneuvers. The transition operator $P_\phi(s_{t+1}\mid s_t,a_t)$ is implemented implicitly by updating stand density and species mix and then running the sub-annual physics loop together with stochastic disturbance checks for fire, insects, recruitment, and natural mortality [2509.19846].

The reward is two-dimensional:

$$
R_t=[R_{carbon,t},R_{thaw,t}].
$$

The carbon component is

$$
r_{carbon}=c_n + s_b + h_b - (p_b + p_s) - p_d - p_i,
$$

where $c_n=\mathrm{clip}\bigl(\Delta C/2.0,-1,1\bigr)$ normalizes net ecosystem carbon change, including harvested wood products; $s_b$ and $h_b$ are currently zero bonuses on total stock and HWP sales; $p_b=0.5\,e_b/15.0$ and $p_s=0.5\,e_s/20.0$ penalize exceedance of biomass and soil carbon limits; $p_d=1$ if density $\ge 2000$, else $0$; and $p_i$ penalizes ineffective thinning or planting [2509.19846].

The thaw component is

$$
r_{thaw}=\mathrm{clip}\Bigl(\frac{f_n-5\,f_p}{40.0},\,-1,1\Bigr),
$$

where $f_p$ and $f_n$ are annual positive and negative heat fluxes through the deep soil boundary, with a 5:1 penalty ratio for warming versus cooling [2509.19846].

Users express preferences through

$$
\lambda=(w_C,1-w_C)\in\Delta^1,
$$

and train either fixed-$\lambda$ agents or preference-conditioned policies using linear scalarization,

$$
r^\lambda_t = w_C\,R_{carbon,t}+(1-w_C)\,R_{thaw,t}.
$$

This formulation makes the carbon–thaw trade-off explicit and measurable. It also clarifies a potential misconception: maximizing carbon gain is not treated as a sufficient proxy for permafrost preservation. The reward design separates the two objectives formally, and the reported results indicate that the resulting optimization landscape is strongly asymmetric [2509.19846].

## 4. Training paradigms and algorithmic baselines

The two principal training paradigms differ in whether site parameters are fixed or sampled. In site-specific mode, $\phi=\phi_\star$ is fixed, with deterministic weather seed, zero temperature noise, and fixed soil and disturbance priors; agents learn over $T=50$ annual steps. In generalist mode, each episode samples $\phi\sim\mathcal D_{site}$ over realistic ranges including latitude, mean annual temperature, seasonal amplitude, and soil conductivity, and the objective becomes

$$
J(\pi) =\mathbb{E}_{\phi\sim\mathcal D_{site}}
\Bigl[\mathbb{E}_{\tau\sim P_\pi^\phi}\sum_{t=0}^{T-1}\gamma^t\,\lambda^\top R_\phi(s_t,a_t)\Bigr].
$$

The reported observation dimensionalities are 43 and 105, respectively [2509.19846].

| Paradigm | Site specification | Observation dimensionality |
|---|---|---|
| Site-specific mode | $\phi=\phi_\star$ fixed | 43 |
| Generalist mode | $\phi\sim\mathcal D_{site}$ sampled each episode | 105 |

Four representative multi-objective RL approaches were evaluated [2509.19846].

**Fixed-$\lambda$ EUPG** trains a single policy $\pi_\theta(a\mid o)$ on a constant scalarization $r^\lambda_t$ using Expected Utility Policy Gradient. The policy input is augmented with cumulative returns per objective for ESR-consistency, and the objective is

$$
J_{\rm EUPG}(\theta;\lambda)
=\mathbb{E}_{\phi,\tau}\Bigl[\sum_t\gamma^t\,r^\lambda_t\Bigr].
$$

**Variable-$\lambda$ EUPG** is preference-conditioned. At the start of each episode, $\lambda\sim\mathcal D_\Lambda$ is sampled and $w_C$ is fed into the observation, producing the training objective

$$
J_{\rm VarEUPG}(\theta)
=\mathbb{E}_{\lambda\sim\mathcal D_\Lambda}
\,\mathbb{E}_{\phi,\tau}\Bigl[\sum_t\gamma^t\,r^\lambda_t\Bigr].
$$

**PPO Gated** uses Proximal Policy Optimization with separate gated network heads for planting versus thinning actions, intended to prevent illegal moves, and is trained on preference input with standard PPO objective, clip 0.2 and GAE 0.95.

**Curriculum PPO** uses a two-network scheme in which a curriculum selector $f_\phi(o_{site})\in[0,1]$ learns which episodes to train on; episodes are accepted when $f_\phi(o_{site})>\tau$ with threshold 0.5. Its objective is

$$
J_{\rm Curr}(\theta,\phi)
=\mathbb{E}_{\lambda,\phi}\Bigl[\mathbb{E}_{\text{select}\sim f_\phi}
\mathbb{E}_{\tau\sim P_{\pi_\theta}^\phi}\sum_t\gamma^t\,r^\lambda_t
\mid \text{select}=1\Bigr].
$$

Generalist agents used $3\times10^5$ timesteps, while site-specific agents used $1\times10^5$ [2509.19846].

## 5. Empirical asymmetry between carbon and thaw objectives

A central result reported for BoreaRL is a pronounced asymmetry in learning difficulty. Across both training paradigms, carbon-focused agents with $w_C=1$ learn rapidly, whereas thaw-focused agents with $w_C=0$ remain near baseline [2509.19846].

In site-specific mode, carbon policies increase scalarized reward by 80–100% above baseline within $5\times10^4$ steps, while thaw policies show less than 5% improvement even after $1\times10^5$ steps. In generalist mode, the same gap persists, which the paper attributes to the noisy, delayed character of the thaw signal [2509.19846].

The learned stand structures also diverge. Carbon policies ramp density to approximately $1280$ stems ha$^{-1}$, whereas thaw policies hold near approximately $1000$–$1020$ stems ha$^{-1}$. Species strategies differ as well: purely carbon agents maintain current conifer fraction, thaw agents shift toward deciduous, and mixed policies raise conifer fraction only under intermediate preference weights [2509.19846].

These findings directly bear on benchmark interpretation. A superficial reading might treat the environment as a standard forest carbon maximization task with an added ecological regularizer. The reported outcomes contradict that simplification. The thaw objective behaves as a distinct and substantially harder control target, and this suggests that performance on carbon-centered forestry tasks does not transfer straightforwardly to permafrost-sensitive management.

## 6. Curriculum learning, emergent strategies, and benchmark role

In the generalist, preference-conditioned benchmark, both Variable-$\lambda$ EUPG and PPO Gated are reported to collapse to near-zero performance for most weights. Curriculum PPO, by contrast, converges quickly to stable performance across the weight spectrum. Its evaluation Pareto front dominates the others, with 50% fewer $\lambda$-monotonicity violations—50% versus 75–100% for fixed/EUPG/PPO Gated—and superior front coverage [2509.19846].

Analysis of the final policies identifies three distinct management philosophies. **PPO Gated** pursues aggressive carbon farming, with high early planting, densities greater than $1400$ stems ha$^{-1}$, and conifer fractions $0.8$–$0.9$, producing maximal carbon but little permafrost benefit. **Variable-$\lambda$ EUPG** remains conservative, with linear density growth and conifer fraction around $0.5$, achieving mediocre outcomes on both objectives. **Curriculum PPO** adopts moderate densities around $1200$ stems ha$^{-1}$ and conifer fractions around $0.6$, maximizing thaw protection $r_{thaw}$ without sacrificing much carbon [2509.19846].

Across evaluation episodes, longer growing seasons under Curriculum PPO correlate strongly with higher thaw rewards, which the paper interprets as effective use of shading, evapotranspiration, and altered snowpack dynamics for permafrost insulation [2509.19846]. This is scientifically important because it indicates that successful policies exploit coupled biophysical pathways rather than merely adjusting stand density in isolation.

The environment is open-sourced under a BSD-3 license at the reported repository URL, with code for BoreaRL-Sim, environment wrappers, training scripts, and experimental seeds. Installation requires Python 3.8+, PyTorch $(\ge 1.10)$, gymnasium, morl-baselines, NumPy, and optionally Numba for JIT acceleration. The commands `python train.py --config=generalist` and `site-specific` are reported to reproduce the published results after `pip install -r requirements.txt` [2509.19846].

Taken together, these characteristics establish BoreaRL as a benchmark for multi-objective RL in climate applications. The benchmark’s defining contribution is not only that it couples a rigorously parameterized forest simulator to an RL interface, but also that it exposes a specific failure mode of current MORL methods: carbon objectives are tractable under standard approaches, whereas thaw preservation remains difficult in both specialist and generalist settings. This suggests that future progress in the area will depend not only on better policy optimization, but also on methods that can cope with delayed, noisy, physically mediated objectives of the kind BoreaRL makes explicit.

Source: https://www.emergentmind.com/topics/borearl