---
title: 'Ground-Compose-Reinforce: Compositional RL Schema'
url: https://www.emergentmind.com/topics/ground-compose-reinforce
type: topic
---

# Ground-Compose-Reinforce: Compositional RL Schema

Searching arXiv for the named framework and closely related papers to ground the article in current literature.
Tool call: arxiv_search({"query":"all:\"Ground-Compose-Reinforce\" OR ti:\"Ground-Compose-Reinforce\" OR ti:\"Compositional Learning of Visually-Grounded Concepts Using Reinforcement\" OR ti:\"Human-like compositional learning of visually-grounded concepts using synthetic environments\" OR ti:\"Compositional Instruction Following with Language Models and Reinforcement Learning\"","max_results":10,"sort_by":"relevance"})
Ground-Compose-Reinforce denotes a recurrent three-stage schema in recent reinforcement-learning and vision-language research: an agent first **grounds** atomic concepts or symbols in perceptual inputs, then **composes** those grounded units into more complex task specifications, and finally **reinforces** behavior through an RL objective defined over the resulting task structure. The phrase appears explicitly in work on visually grounded navigation, compositional instruction following, and formal-language tasking, but the concrete implementation varies substantially across papers: grounding may mean invariant concept acquisition in a synthetic 3D room, semantic parsing into Boolean task expressions, learned labelling of atomic propositions for Reward Machines, or explicit evidence localization for intermediate reasoning steps [2309.04504] [2504.06618] [2501.12539] [2507.10741] [2606.29915].

## 1. Conceptual schema and scope

Across the literature, the three verbs refer to distinct operations. In the 3D navigation work on visually grounded concepts, **Ground** denotes pretraining on single-attribute RL tasks so that the agent learns color- or shape-invariance; **Compose** denotes fine-tuning on conjunctions such as color-shape targets; and **Reinforce** denotes the fact that the RL objective alone, together with the curriculum from simpler to more complex environments, is used to induce this behavior [2309.04504]. In the later synthetic-environment extension, the same pattern is applied to richer linguistic concept classes, including determiners and prepositions, again with curriculum learning as the main mechanism for improving efficiency [2504.06618].

A different instantiation appears in CERLLA, where **Ground** is semantic parsing of a natural-language instruction into a Boolean task expression, **Compose** is closed-form assembly of a task policy from primitive world-value functions, and **Reinforce** is used both to learn the primitives and to refine the parser context through rollout-based success feedback [2501.12539]. In the formal-language framework that explicitly adopts the name Ground-Compose-Reinforce, **Ground** is a learned labelling function from states to atomic propositions, **Compose** is value composition under Reward Machine semantics, and **Reinforce** is PPO on an augmented MDP with potential-based reward shaping [2507.10741].

This distribution of meanings suggests that Ground-Compose-Reinforce is best understood not as a single algorithm but as a family of compositional design principles. The commonality lies in the separation of atomic grounding from downstream task composition, followed by RL over the composed task representation.

| Paper | Ground / Compose / Reinforce instantiation | Representative claim |
|---|---|---|
| [2309.04504] | Single-attribute concept pretraining / color-shape recomposition / A2C navigation | Concept pretraining reduced episodes for zero-shot compositional learning by about 20 times |
| [2504.06618] | Grounding determiners and prepositions / held-out instruction recombination / actor-critic with curriculum | Curriculum reduced required training episodes by 15% in determiner environments and enabled prepositional learning |
| [2501.12539] | LLM semantic parsing / min-max-Boolean value composition / DQN plus RL-driven in-context refinement | Reached the oracle upper-bound success rate of 92% on 162 tasks |
| [2507.10741] | Learned proposition labelling / Reward Machine composition / PPO with reward shaping | End-to-end baselines failed to generalize to unseen compositions |

## 2. Visually grounded compositional learning in synthetic 3D navigation

The 2023 navigation study provides one of the clearest early formulations of the pattern. The environment is a first-person synthetic 3D room in Unity in which an agent must navigate to a single target object among four spawned at fixed positions. Static landmarks include a door, window, shelf, and human figure; the objects are drawn from five shapes—capsule, cube, cylinder, prism, sphere—and five colors—red, green, blue, yellow, black. Each object is identified by a $(\text{Color}, \text{Shape})$ pair, and instructions may specify a conjunction such as “red cube,” a three-word combination such as “green sphere prism,” or a single attribute such as “blue” [2309.04504].

The agent architecture is a standard multimodal recurrent actor-critic. A $128\times128$ RGB frame is encoded by three convolutional layers into 64 feature maps of size $7\times7$, flattened into a 3136-dimensional vector. Language is represented either by two one-hot vectors for color and shape, each linearly mapped to 128 dimensions and concatenated, or by a frozen pretrained text encoder such as CLIP or BERT followed by a 128-dimensional projection. The 3136-dimensional visual embedding and 128-dimensional language embedding are concatenated, passed through a 256-dimensional mixing layer, and then through an LSTM whose hidden state feeds an actor head over $\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$ and a critic head $V(s_t)$ [2309.04504].

Training uses synchronous A2C with RMSProp at learning rate $2.5\times10^{-4}$. The reward function gives $+10$ for reaching the correct target, $-3$ for colliding with non-target objects, $-1$ for hitting walls, and $-10$ for exceeding 500 steps. The performance criterion is an average episodic return of at least 9 over 100 consecutive episodes. Concept learning is not enforced by auxiliary contrastive or classification losses; rather, it is operationalized by RL pretraining in the single-attribute $C\vee S$ environment under
$$
L_{\text{concept}}(\theta) = - \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^T r_t^{(C\vee S)}\right],
$$
followed by compositional training in $C\wedge S$ under
$$
L_{\text{comp}}(\theta) = - \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^T r_t^{(C\wedge S)}\right].
$$
The policy objective is
$$
J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^T \gamma^t r_t\right],\qquad
\nabla_\theta J(\theta)\approx \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^T \nabla_\theta \log \pi_\theta(a_t\mid s_t)\cdot A_t\right].
$$

The key empirical result is that agents trained naively on 20 of the 25 possible color-shape pairs nevertheless learned to decompose and recompose those combinations, achieving zero-shot success on the 5 held-out pairs. The one-hot agent required about 67K episodes to meet the training criterion and about 95K to succeed zero-shot on held-out combinations. When pretrained on single-attribute concepts for 168K episodes and then fine-tuned on $C\wedge S$, the same system required only 0.6K episodes to learn the 20 training pairs and 5.5K episodes to reach zero-shot performance on held-out combinations. Only agents trained on both concept and compositional learning solved a more complex out-of-distribution environment in zero-shot fashion, and only text encoders pretrained on image-text datasets, such as CLIP, reduced the number of episodes needed for compositional learning while also generalizing to five unseen colors—orange, cyan, pink, purple, white—with mean return about 6.9 [2309.04504].

## 3. Determiners, prepositions, and curriculum as a compositional scaffold

The 2025 extension generalizes the synthetic-environment program from color-shape conjunctions to richer linguistic concept classes, specifically determiners and prepositions. Navigation-from-instruction is formalized as an MDP $(S,A,T,R,\gamma)$ in which the state $s_t=(o_t,I)$ contains the raw RGB image $o_t\in\mathbb{R}^{3\times128\times128}$ and a discrete instruction $I$ of length $L$ words, with $L=3$ in determiner environments, $L=5$ in preposition environments, and $L=7$ in combined determiner-plus-preposition environments. The action space remains $\{\text{forward}, \text{backward}, \text{turn\_left}, \text{turn\_right}\}$, transitions are deterministic physics-based rollouts in Unity, and the reward is $+10$ for reaching the correct target, $-3$ for reaching a wrong target, $-1$ for colliding with walls, and $-10$ if no target is reached by $T_{\max}=500$ steps, with $\gamma$ typically set to $0.99$ [2504.06618].

The instruction vocabulary is partitioned into determiners, prepositions, colors, and shapes. Determiners are
$\{\text{“A”},\text{“Few”},\text{“Some”},\text{“Many”},\text{“This”},\text{“That”},\text{“These”},\text{“Those”}\}$,
and prepositions are
$\{\text{“Above”},\text{“Below”},\text{“In front of”},\text{“Behind”},\text{“Beside”},\text{“On”},\text{“Between”},\text{“Among”}\}$.
Each word slot is one-hot encoded, mapped through a shared linear embedding to 128 dimensions, and concatenated into a 128-dimensional language representation. Visual processing again uses three convolutional layers producing a 3136-dimensional vector, followed by linear fusion to 256 dimensions and an LSTM. Actor and critic heads are defined by
$$
\pi_\theta(a_t\mid s_t)=\text{Softmax}(W_a s_t+b_a),\qquad
V_\theta(s_t)=W_v s_t+b_v,
$$
with hidden size $H=256$ [2504.06618].

The central intervention is curriculum learning. For determiners, tasks are ordered as 2 simple determiners, then 4 mid-level determiners, then all 8. For prepositions, the sequence is $2P\rightarrow4P\rightarrow8P$. Progression requires at least 80% success—described as average $+10$ reward—over 1000 consecutive episodes. This scaffold materially changes sample efficiency. Training all eight determiners from scratch required 0.87M episodes; the $4D\rightarrow8D$ curriculum required $0.66\text{M}+0.11\text{M}=0.77\text{M}$, and the $2D\rightarrow4D\rightarrow8D$ curriculum required $0.38+0.33+0.11=0.82\text{M}$. Prepositions were substantially harder: naive training on 8P failed to converge by 3.5M episodes, whereas $4P\rightarrow8P$ required $1.57+1.30=2.87\text{M}$ and $2P\rightarrow4P\rightarrow8P$ required $0.90+0.30+0.98=2.18\text{M}$, reported as about 38% fewer episodes versus naive extrapolation [2504.06618].

The generalization results are correspondingly stratified. Held-out determiner instructions yielded about 77–79% success on test, close to training performance. Held-out preposition instructions reached about 76% success, compared with 24% for non-converged agents. In the combined $D+P$ setting with 160,000 instructions, pretraining on prepositions with $4P\rightarrow8P$ and then fine-tuning for only 0.1M episodes produced 53% success in a zero-shot $D+P$ test set, whereas removing curriculum or pretraining abolished convergence in preposition environments and prevented generalization to $D+P$ [2504.06618]. A common misconception is that compositional generalization in RL is uniform across linguistic operators; these results indicate that relational concepts such as prepositions are substantially more difficult than determiner concepts under otherwise similar architectures and objectives.

## 4. Semantic parsing and value-function composition

CERLLA demonstrates a more explicitly symbolic variant of Ground-Compose-Reinforce. Instructions are parsed by a frozen LLM, such as GPT-4, into Boolean expressions over primitive symbols using operators $\wedge$, $\vee$, and $\neg$. At each episode, the parser is given a short system instruction, up to $K=10$ retrieved in-context examples from a set $C$ via BM25, and the new instruction $u$, then returns $B=10$ candidate parses. There is no gradient-based parser loss in the implemented system; parser improvement occurs by augmenting $C$ with successful examples identified through rollout-based evaluation [2501.12539].

Composition is performed by a bank of primitive world-value functions $\{Q_i(s,a)\}$, one per primitive attribute, pretrained by DQN under an augmented reward that strongly penalizes reaching the wrong goal. Closed-form composition then yields a task $Q$-function:
$$
Q_{i\wedge j}(s,a)=\min\{Q_i(s,a),Q_j(s,a)\},\qquad
Q_{i\vee j}(s,a)=\max\{Q_i(s,a),Q_j(s,a)\},
$$
$$
Q_{\neg i}(s,a)=(Q_{\max}(s,a)+Q_{\min}(s,a))-Q_i(s,a).
$$
Given a Boolean formula $\ell$, the same operator tree is applied recursively to obtain $Q_{\text{task}}(s,a;\ell)$, from which the policy is induced by softmax with temperature $\tau$ or by greedy action selection [2501.12539].

Reinforcement enters twice. First, each primitive $Q_i$ is learned with the temporal-difference objective
$$
\mathcal{L}_i=
\mathbb{E}_{(s,a,r,s')\sim\mathcal D}
\left[r+\gamma\max_{a'}Q_i(s',a')-Q_i(s,a)\right]^2.
$$
Second, candidate parses are filtered by policy performance: each candidate $\ell^j$ induces a policy $\pi_{\ell^j}$ that is evaluated over $N=100$ rollouts, producing empirical success rate $\hat s_j$. If $\hat s_j\ge\tau=92\%$, the pair $(u,\ell^j)$ is added to or replaces an entry in the in-context example set $C$ [2501.12539].

The reported sample-complexity profile is highly asymmetric between primitive learning and downstream adaptation. CERLLA learns $m=9$ primitives in about 19M environment steps, then accumulates about 162 in-context examples in another 0.6M steps, for about 19.6M total steps to reach 92% average success on all 162 tasks. The best non-compositional baseline, a single Q-network over language and vision, is trained for 21M steps but reaches only about 80% success. On held-out splits of 81 training and 81 test tasks, CERLLA reaches about 90% on both train and test in about 1M steps, whereas the non-compositional baseline maintains a substantial generalization gap even after 21M steps [2501.12539]. Here, “compose” is not latent recombination inside a recurrent policy but explicit algebra over reusable value functions.

## 5. Formal-language tasking with Reward Machines

The most formal version of Ground-Compose-Reinforce is the neurosymbolic framework based on Reward Machines. It adopts Reward Machines as the task-specification language, with atomic propositions $\mathcal{AP}=\{x_1,\dots,x_n\}$ and propositional conditions generated by
$$
\varphi ::= x \mid \neg x \mid (\varphi\wedge\varphi)\mid(\varphi\vee\varphi).
$$
An RM is
$\mathcal{R}=\langle\mathcal{U},u_0,\mathcal{F},\mathcal{AP},\delta_u,\delta_r\rangle$,
where $\delta_u$ and $\delta_r$ define state transitions and instantaneous rewards over truth assignments to the atomic propositions. Given a learned labelling function $\mathcal{L}:\mathcal{S}\to2^{\mathcal{AP}}$, the induced reward and automaton transition are
$$
r_{t+1}=\delta_r(u_t,\mathcal{L}(s_{t+1})),\qquad
u_{t+1}=\delta_u(u_t,\mathcal{L}(s_{t+1})).
$$
Grounding is thus the problem of learning $\hat{\mathcal{L}}(s)\approx\mathcal{L}(s)$ from labelled state data [2507.10741].

The grounding module is dataset-driven. In GeoGrid, the labeler is a two-layer convolutional network with Conv$(6\rightarrow16)\rightarrow$ReLU$\rightarrow$Conv$(16\rightarrow32)\rightarrow$ReLU$\rightarrow$flatten$\rightarrow$MLP$\rightarrow$5-way logits. In DrawerWorld, it is a two-layer MLP $(39\rightarrow1600\rightarrow\text{ReLU}\rightarrow11\text{-way logits})$. The objective is binary cross-entropy per proposition:
$$
\mathcal{L}_{\mathrm{label}}
= -\sum_{x\in\mathcal{AP}}
\Bigl[
\omega_t(x)\log \sigma(z_x(s_t))
+(1-\omega_t(x))\log(1-\sigma(z_x(s_t)))
\Bigr].
$$
Composition occurs at the value-function level. Primitive value functions for eventual satisfaction of literals, $V^*_{\Diamond x}$ and $V^*_{\Diamond\neg x}$, are learned once; DNF decomposition then uses max over clauses and min over literals, and a high-level value iteration over the RM graph approximates $V^*_{\mathcal{R}}(s,u)$ for arbitrary Reward Machines [2507.10741].

Reinforcement is performed on the augmented MDP with state $(s_t,u_t)$ under PPO and shaped reward
$$
\tilde r_{t+1}
=
\delta_r(u_t,\hat{\mathcal{L}}(s_{t+1}))
+
\lambda\bigl(\gamma V^*_{\mathcal{R}}(s_{t+1},u_{t+1})-V^*_{\mathcal{R}}(s_t,u_t)\bigr).
$$
The data regime is deliberately small relative to downstream task complexity: GeoGrid uses 5,000 random-policy episodes of length 100 labelled automatically, and DrawerWorld uses 350 manually controlled episodes performing generic interactions. Downstream tasks include temporally extended sequences, loops, logical ordering, and safety constraints, some of which require behaviors never seen together in the grounding data. In these experiments, the framework is reported as the only method to consistently solve all tasks in both domains, with end-to-end baselines failing to generalize to unseen compositions; ablations further show that the compositional reward shaping is critical in sparse settings such as DrawerWorld [2507.10741].

## 6. Antecedents, extensions, and interpretive boundaries

A control-theoretic antecedent appears in compositional RL for discrete-time stochastic control systems. There, the same triad can be recognized as **Ground** through implicit finite abstraction of each subsystem, **Compose** through assume-guarantee synthesis over a network of subsystems and conjunctions of scLTL specifications, and **Reinforce** through minimax-Q learning on the product of each abstraction with an automaton-derived reward structure. The framework provides an abstraction error bound
$$
\left|P[\Sigma_i^x{}_{\rho,\xi}\models\phi]-P[\hat\Sigma_i^{\hat x}{}_{\rho,\xi}\models\phi]\right|
\le
\epsilon_i,
\qquad
\epsilon_i:=T\cdot L_i\cdot(\delta_i H_{x_i}+\mu_i H_{w_i}),
$$
and a compositional lower bound on global satisfaction probability under decentralized policies. Sparse automaton rewards are mitigated by potential-based shaping, and multi-level discretization is used as a warm start to accelerate tabular minimax-Q learning [2208.03485]. Although the paper does not use the term Ground-Compose-Reinforce as a formal label, it supplies a closely related structural template.

A later extension broadens the phrase into grounded visual reasoning rather than navigation or formal tasking. H-GRPO uses a pretrained vision encoder and an LLM decoder to generate grounded reasoning triplets $\tau_i=\langle q_i,a_i,b_i\rangle$, where each sub-question and sub-answer is paired with a localized evidence bounding box. The RL objective uses a composite reward
$$
\mathcal{R}(y)=\alpha\cdot\mathcal{R}_{\text{format}}(y)+\beta\cdot\mathcal{R}_{\text{answer}}(y)\cdot\mathcal{R}_{\text{HS}}(y),
$$
with permutation invariance enforced by Hungarian matching over predicted and reference triplets. The training dataset contains 10K automatically synthesized grounded chains from Visual7W, Visual-CoT, A-OKVQA, and ERQA, plus a gold-standard seed set of 40 human-verified examples. Reported gains include improvements from 54.4% to 70.2% on RoboSpatial and from 46.0% to 52.4% on MMStar for Qwen2.5-VL under H-GRPO, along with an interpretability score of 4.73 on a 1–5 scale [2606.29915]. This does not redefine the earlier RL-tasking frameworks, but it shows that the same ground-compose-reinforce motif can be transferred to evidence-grounded reasoning chains.

Two interpretive boundaries recur across the literature. First, Ground-Compose-Reinforce is not synonymous with generic language-conditioned RL. The grounding object may be perceptual invariances, logical forms, atomic propositions, or evidence regions, and the composition operator may be recurrent recombination, curriculum-mediated recomposition, Boolean min/max algebra, Reward Machine semantics, or permutation-invariant matching. Second, zero-shot or few-shot compositionality does not appear as an automatic by-product of large-scale training alone. In these studies, gains are tied to structural scaffolds: single-attribute pretraining, curriculum ordering, primitive value reuse, formal language semantics, or intermediate evidence supervision [2309.04504] [2504.06618] [2501.12539] [2507.10741] [2606.29915].

Taken together, the literature positions Ground-Compose-Reinforce as a technical program for separating atomic grounding from higher-order task composition while preserving RL as the mechanism for behavior acquisition. The most conservative reading is that this separation improves sample efficiency, strengthens out-of-distribution generalization to unseen combinations, and makes the task structure more explicit than monolithic end-to-end baselines under the experimental settings reported in these papers.

Source: https://www.emergentmind.com/topics/ground-compose-reinforce