Ground-Compose-Reinforce: Compositional RL Schema

• Ground-Compose-Reinforce is a compositional RL paradigm that separates atomic grounding, task composition, and reinforcement to improve sample efficiency and generalization.
• The framework applies distinct operations—pretraining on invariant perceptual features, recomposing complex task specifications, and using RL objectives—with applications in 3D navigation, semantic parsing, and formal-language tasks.
• Empirical findings show that modular architectures and curriculum learning reduce training episodes dramatically while enhancing zero-shot performance on unseen task combinations.

Searching arXiv for the named framework and closely related papers to ground the article in current literature. Tool call: arxiv_search({"query":"all:\"Ground-Compose-Reinforce\" OR ti:\"Ground-Compose-Reinforce\" OR ti:\"Compositional Learning of Visually-Grounded Concepts Using Reinforcement\" OR ti:\"Human-like compositional learning of visually-grounded concepts using synthetic environments\" OR ti:\"Compositional Instruction Following with LLMs and Reinforcement Learning\"","max_results":10,"sort_by":"relevance"}) Ground-Compose-Reinforce denotes a recurrent three-stage schema in recent reinforcement-learning and vision-language research: an agent first grounds atomic concepts or symbols in perceptual inputs, then composes those grounded units into more complex task specifications, and finally reinforces behavior through an RL objective defined over the resulting task structure. The phrase appears explicitly in work on visually grounded navigation, compositional instruction following, and formal-language tasking, but the concrete implementation varies substantially across papers: grounding may mean invariant concept acquisition in a synthetic 3D room, semantic parsing into Boolean task expressions, learned labelling of atomic propositions for Reward Machines, or explicit evidence localization for intermediate reasoning steps (Lin et al., 2023, Lin et al., 9 Apr 2025, Cohen et al., 21 Jan 2025, Li et al., 14 Jul 2025, Peh et al., 29 Jun 2026).

1. Conceptual schema and scope

Across the literature, the three verbs refer to distinct operations. In the 3D navigation work on visually grounded concepts, Ground denotes pretraining on single-attribute RL tasks so that the agent learns color- or shape-invariance; Compose denotes fine-tuning on conjunctions such as color-shape targets; and Reinforce denotes the fact that the RL objective alone, together with the curriculum from simpler to more complex environments, is used to induce this behavior (Lin et al., 2023). In the later synthetic-environment extension, the same pattern is applied to richer linguistic concept classes, including determiners and prepositions, again with curriculum learning as the main mechanism for improving efficiency (Lin et al., 9 Apr 2025).

A different instantiation appears in CERLLA, where Ground is semantic parsing of a natural-language instruction into a Boolean task expression, Compose is closed-form assembly of a task policy from primitive world-value functions, and Reinforce is used both to learn the primitives and to refine the parser context through rollout-based success feedback (Cohen et al., 21 Jan 2025). In the formal-language framework that explicitly adopts the name Ground-Compose-Reinforce, Ground is a learned labelling function from states to atomic propositions, Compose is value composition under Reward Machine semantics, and Reinforce is PPO on an augmented MDP with potential-based reward shaping (Li et al., 14 Jul 2025).

This distribution of meanings suggests that Ground-Compose-Reinforce is best understood not as a single algorithm but as a family of compositional design principles. The commonality lies in the separation of atomic grounding from downstream task composition, followed by RL over the composed task representation.

Paper	Ground / Compose / Reinforce instantiation	Representative claim
(Lin et al., 2023)	Single-attribute concept pretraining / color-shape recomposition / A2C navigation	Concept pretraining reduced episodes for zero-shot compositional learning by about 20 times
(Lin et al., 9 Apr 2025)	Grounding determiners and prepositions / held-out instruction recombination / actor-critic with curriculum	Curriculum reduced required training episodes by 15% in determiner environments and enabled prepositional learning
(Cohen et al., 21 Jan 2025)	LLM semantic parsing / min-max-Boolean value composition / DQN plus RL-driven in-context refinement	Reached the oracle upper-bound success rate of 92% on 162 tasks
(Li et al., 14 Jul 2025)	Learned proposition labelling / Reward Machine composition / PPO with reward shaping	End-to-end baselines failed to generalize to unseen compositions

2. Visually grounded compositional learning in synthetic 3D navigation

The 2023 navigation study provides one of the clearest early formulations of the pattern. The environment is a first-person synthetic 3D room in Unity in which an agent must navigate to a single target object among four spawned at fixed positions. Static landmarks include a door, window, shelf, and human figure; the objects are drawn from five shapes—capsule, cube, cylinder, prism, sphere—and five colors—red, green, blue, yellow, black. Each object is identified by a $(\text{Color}, \text{Shape})$ pair, and instructions may specify a conjunction such as “red cube,” a three-word combination such as “green sphere prism,” or a single attribute such as “blue” (Lin et al., 2023).

The agent architecture is a standard multimodal recurrent actor-critic. A $128\times128$ RGB frame is encoded by three convolutional layers into 64 feature maps of size $7\times7$, flattened into a 3136-dimensional vector. Language is represented either by two one-hot vectors for color and shape, each linearly mapped to 128 dimensions and concatenated, or by a frozen pretrained text encoder such as CLIP or BERT followed by a 128-dimensional projection. The 3136-dimensional visual embedding and 128-dimensional language embedding are concatenated, passed through a 256-dimensional mixing layer, and then through an LSTM whose hidden state feeds an actor head over $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$ and a critic head $V(s_t)$ (Lin et al., 2023).

Training uses synchronous A2C with RMSProp at learning rate $2.5\times10^{-4}$. The reward function gives $+10$ for reaching the correct target, $-3$ for colliding with non-target objects, $-1$ for hitting walls, and $-10$ for exceeding 500 steps. The performance criterion is an average episodic return of at least 9 over 100 consecutive episodes. Concept learning is not enforced by auxiliary contrastive or classification losses; rather, it is operationalized by RL pretraining in the single-attribute $128\times128$0 environment under

$128\times128$1

followed by compositional training in $128\times128$2 under

$128\times128$3

The policy objective is

$128\times128$4

The key empirical result is that agents trained naively on 20 of the 25 possible color-shape pairs nevertheless learned to decompose and recompose those combinations, achieving zero-shot success on the 5 held-out pairs. The one-hot agent required about 67K episodes to meet the training criterion and about 95K to succeed zero-shot on held-out combinations. When pretrained on single-attribute concepts for 168K episodes and then fine-tuned on $128\times128$5, the same system required only 0.6K episodes to learn the 20 training pairs and 5.5K episodes to reach zero-shot performance on held-out combinations. Only agents trained on both concept and compositional learning solved a more complex out-of-distribution environment in zero-shot fashion, and only text encoders pretrained on image-text datasets, such as CLIP, reduced the number of episodes needed for compositional learning while also generalizing to five unseen colors—orange, cyan, pink, purple, white—with mean return about 6.9 (Lin et al., 2023).

3. Determiners, prepositions, and curriculum as a compositional scaffold

The 2025 extension generalizes the synthetic-environment program from color-shape conjunctions to richer linguistic concept classes, specifically determiners and prepositions. Navigation-from-instruction is formalized as an MDP $128\times128$6 in which the state $128\times128$7 contains the raw RGB image $128\times128$8 and a discrete instruction $128\times128$9 of length $7\times7$0 words, with $7\times7$1 in determiner environments, $7\times7$2 in preposition environments, and $7\times7$3 in combined determiner-plus-preposition environments. The action space remains $7\times7$4, transitions are deterministic physics-based rollouts in Unity, and the reward is $7\times7$5 for reaching the correct target, $7\times7$6 for reaching a wrong target, $7\times7$7 for colliding with walls, and $7\times7$8 if no target is reached by $7\times7$9 steps, with $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$0 typically set to $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$1 (Lin et al., 9 Apr 2025).

The instruction vocabulary is partitioned into determiners, prepositions, colors, and shapes. Determiners are $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$2, and prepositions are $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$3. Each word slot is one-hot encoded, mapped through a shared linear embedding to 128 dimensions, and concatenated into a 128-dimensional language representation. Visual processing again uses three convolutional layers producing a 3136-dimensional vector, followed by linear fusion to 256 dimensions and an LSTM. Actor and critic heads are defined by

$$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$4

with hidden size $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$5 (Lin et al., 9 Apr 2025).

The central intervention is curriculum learning. For determiners, tasks are ordered as 2 simple determiners, then 4 mid-level determiners, then all 8. For prepositions, the sequence is $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$6. Progression requires at least 80% success—described as average $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$7 reward—over 1000 consecutive episodes. This scaffold materially changes sample efficiency. Training all eight determiners from scratch required 0.87M episodes; the $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$8 curriculum required $$\{\text{forward}, \text{backward}, \text{turn left}, \text{turn right}\}$$9, and the $V(s_t)$0 curriculum required $V(s_t)$1. Prepositions were substantially harder: naive training on 8P failed to converge by 3.5M episodes, whereas $V(s_t)$2 required $V(s_t)$3 and $V(s_t)$4 required $V(s_t)$5, reported as about 38% fewer episodes versus naive extrapolation (Lin et al., 9 Apr 2025).

The generalization results are correspondingly stratified. Held-out determiner instructions yielded about 77–79% success on test, close to training performance. Held-out preposition instructions reached about 76% success, compared with 24% for non-converged agents. In the combined $V(s_t)$6 setting with 160,000 instructions, pretraining on prepositions with $V(s_t)$7 and then fine-tuning for only 0.1M episodes produced 53% success in a zero-shot $V(s_t)$8 test set, whereas removing curriculum or pretraining abolished convergence in preposition environments and prevented generalization to $V(s_t)$9 (Lin et al., 9 Apr 2025). A common misconception is that compositional generalization in RL is uniform across linguistic operators; these results indicate that relational concepts such as prepositions are substantially more difficult than determiner concepts under otherwise similar architectures and objectives.

4. Semantic parsing and value-function composition

CERLLA demonstrates a more explicitly symbolic variant of Ground-Compose-Reinforce. Instructions are parsed by a frozen LLM, such as GPT-4, into Boolean expressions over primitive symbols using operators $2.5\times10^{-4}$0, $2.5\times10^{-4}$1, and $2.5\times10^{-4}$2. At each episode, the parser is given a short system instruction, up to $2.5\times10^{-4}$3 retrieved in-context examples from a set $2.5\times10^{-4}$4 via BM25, and the new instruction $2.5\times10^{-4}$5, then returns $2.5\times10^{-4}$6 candidate parses. There is no gradient-based parser loss in the implemented system; parser improvement occurs by augmenting $2.5\times10^{-4}$7 with successful examples identified through rollout-based evaluation (Cohen et al., 21 Jan 2025).

Composition is performed by a bank of primitive world-value functions $2.5\times10^{-4}$8, one per primitive attribute, pretrained by DQN under an augmented reward that strongly penalizes reaching the wrong goal. Closed-form composition then yields a task $2.5\times10^{-4}$9-function:

$+10$0

$+10$1

Given a Boolean formula $+10$2, the same operator tree is applied recursively to obtain $+10$3, from which the policy is induced by softmax with temperature $+10$4 or by greedy action selection (Cohen et al., 21 Jan 2025).

Reinforcement enters twice. First, each primitive $+10$5 is learned with the temporal-difference objective

$+10$6

Second, candidate parses are filtered by policy performance: each candidate $+10$7 induces a policy $+10$8 that is evaluated over $+10$9 rollouts, producing empirical success rate $-3$0. If $-3$1, the pair $-3$2 is added to or replaces an entry in the in-context example set $-3$3 (Cohen et al., 21 Jan 2025).

The reported sample-complexity profile is highly asymmetric between primitive learning and downstream adaptation. CERLLA learns $-3$4 primitives in about 19M environment steps, then accumulates about 162 in-context examples in another 0.6M steps, for about 19.6M total steps to reach 92% average success on all 162 tasks. The best non-compositional baseline, a single Q-network over language and vision, is trained for 21M steps but reaches only about 80% success. On held-out splits of 81 training and 81 test tasks, CERLLA reaches about 90% on both train and test in about 1M steps, whereas the non-compositional baseline maintains a substantial generalization gap even after 21M steps (Cohen et al., 21 Jan 2025). Here, “compose” is not latent recombination inside a recurrent policy but explicit algebra over reusable value functions.

5. Formal-language tasking with Reward Machines

The most formal version of Ground-Compose-Reinforce is the neurosymbolic framework based on Reward Machines. It adopts Reward Machines as the task-specification language, with atomic propositions $-3$5 and propositional conditions generated by

$-3$6

An RM is $-3$7, where $-3$8 and $-3$9 define state transitions and instantaneous rewards over truth assignments to the atomic propositions. Given a learned labelling function $-1$0, the induced reward and automaton transition are

$-1$1

Grounding is thus the problem of learning $-1$2 from labelled state data (Li et al., 14 Jul 2025).

The grounding module is dataset-driven. In GeoGrid, the labeler is a two-layer convolutional network with Conv$-1$3ReLU$-1$4Conv$-1$5ReLU$-1$6flatten$-1$7MLP$-1$85-way logits. In DrawerWorld, it is a two-layer MLP $-1$9. The objective is binary cross-entropy per proposition:

$-10$0

Composition occurs at the value-function level. Primitive value functions for eventual satisfaction of literals, $-10$1 and $-10$2, are learned once; DNF decomposition then uses max over clauses and min over literals, and a high-level value iteration over the RM graph approximates $-10$3 for arbitrary Reward Machines (Li et al., 14 Jul 2025).

Reinforcement is performed on the augmented MDP with state $-10$4 under PPO and shaped reward

$-10$5

The data regime is deliberately small relative to downstream task complexity: GeoGrid uses 5,000 random-policy episodes of length 100 labelled automatically, and DrawerWorld uses 350 manually controlled episodes performing generic interactions. Downstream tasks include temporally extended sequences, loops, logical ordering, and safety constraints, some of which require behaviors never seen together in the grounding data. In these experiments, the framework is reported as the only method to consistently solve all tasks in both domains, with end-to-end baselines failing to generalize to unseen compositions; ablations further show that the compositional reward shaping is critical in sparse settings such as DrawerWorld (Li et al., 14 Jul 2025).

6. Antecedents, extensions, and interpretive boundaries

A control-theoretic antecedent appears in compositional RL for discrete-time stochastic control systems. There, the same triad can be recognized as Ground through implicit finite abstraction of each subsystem, Compose through assume-guarantee synthesis over a network of subsystems and conjunctions of scLTL specifications, and Reinforce through minimax-Q learning on the product of each abstraction with an automaton-derived reward structure. The framework provides an abstraction error bound

$-10$6

and a compositional lower bound on global satisfaction probability under decentralized policies. Sparse automaton rewards are mitigated by potential-based shaping, and multi-level discretization is used as a warm start to accelerate tabular minimax-Q learning (Lavaei et al., 2022). Although the paper does not use the term Ground-Compose-Reinforce as a formal label, it supplies a closely related structural template.

A later extension broadens the phrase into grounded visual reasoning rather than navigation or formal tasking. H-GRPO uses a pretrained vision encoder and an LLM decoder to generate grounded reasoning triplets $-10$7, where each sub-question and sub-answer is paired with a localized evidence bounding box. The RL objective uses a composite reward

$-10$8

with permutation invariance enforced by Hungarian matching over predicted and reference triplets. The training dataset contains 10K automatically synthesized grounded chains from Visual7W, Visual-CoT, A-OKVQA, and ERQA, plus a gold-standard seed set of 40 human-verified examples. Reported gains include improvements from 54.4% to 70.2% on RoboSpatial and from 46.0% to 52.4% on MMStar for Qwen2.5-VL under H-GRPO, along with an interpretability score of 4.73 on a 1–5 scale (Peh et al., 29 Jun 2026). This does not redefine the earlier RL-tasking frameworks, but it shows that the same ground-compose-reinforce motif can be transferred to evidence-grounded reasoning chains.

Two interpretive boundaries recur across the literature. First, Ground-Compose-Reinforce is not synonymous with generic language-conditioned RL. The grounding object may be perceptual invariances, logical forms, atomic propositions, or evidence regions, and the composition operator may be recurrent recombination, curriculum-mediated recomposition, Boolean min/max algebra, Reward Machine semantics, or permutation-invariant matching. Second, zero-shot or few-shot compositionality does not appear as an automatic by-product of large-scale training alone. In these studies, gains are tied to structural scaffolds: single-attribute pretraining, curriculum ordering, primitive value reuse, formal language semantics, or intermediate evidence supervision (Lin et al., 2023, Lin et al., 9 Apr 2025, Cohen et al., 21 Jan 2025, Li et al., 14 Jul 2025, Peh et al., 29 Jun 2026).

Taken together, the literature positions Ground-Compose-Reinforce as a technical program for separating atomic grounding from higher-order task composition while preserving RL as the mechanism for behavior acquisition. The most conservative reading is that this separation improves sample efficiency, strengthens out-of-distribution generalization to unseen combinations, and makes the task structure more explicit than monolithic end-to-end baselines under the experimental settings reported in these papers.