Grounding-Aware Reinforcement Learning

• Grounding-aware RL is a family of formulations that optimizes the interface between symbolic descriptions, perceptual observations, simulator dynamics, or executable actions, making these variables active in learning.
• Notably, methods like RGAT, which interleave grounding optimization with policy improvement, demonstrate near-optimal performance with concrete metrics (e.g., Hopper reaching direct-real-training performance in about three grounding steps [2008.01279]).
• The approach spans applications from language-grounded transfer and temporal/visual reasoning to GUI and clinical control, emphasizing verifiable, geometry-aware reward components to counteract reward hacking and structural limitations.

to illustrate different problem formulations. Need maintain encyclopedic tone.

Need ensure concrete claims appear verbatim in data block. Since data includes abundant wording. We can paraphrase but must not change numbers or introduce unsupported claims. For safety, quote phrasing closely.

Need maybe use arXiv ids for each paper in sentences. Search tool maybe not available? Let's check tools. Grounding-aware reinforcement learning is a family of reinforcement learning formulations in which the interface between symbolic descriptions, perceptual observations, simulator dynamics, or executable actions and the environment is itself optimized, critiqued, or reward-shaped during learning, rather than assumed as a fixed oracle. In this literature, grounding may mean modifying a simulator so that transformed actions induce more realistic transitions, learning a probabilistic mapping from raw observations to propositions used by temporal-logic tasks, selecting temporally relevant video segments or image regions before answering, aligning world models with physically verifiable geometry, or converting clinical and GUI instructions into executable spatial prompts (Karnan et al., 2020, Pannacci et al., 10 Feb 2026, He et al., 1 Dec 2025).

1. Conceptual scope and historical trajectory

The term does not denote a single algorithmic template. In 2017, language grounding for transfer was framed as learning how entity descriptions correspond to environment transitions and rewards, so that text becomes a compact intermediate representation across domains (Narasimhan et al., 2017). By 2019 and 2020, temporal grounding was recast as sequential decision making over video boundaries, both with full supervision and under weak supervision (He et al., 2019, Wu et al., 2020). In 2023, functional grounding of LLM policies was posed as online RL in a textual environment, emphasizing alignment between language-model priors and actual transition dynamics (Carta et al., 2023). Work from 2025 and 2026 generalizes the same principle to world models, sub-symbolic task semantics, GUI agents, medical VLMs, and multi-image grounding (He et al., 1 Dec 2025, Li et al., 14 Jul 2025, Wang et al., 23 Apr 2026, Zheng et al., 26 Sep 2025).

Taken together, the literature uses grounding for several distinct objects: transition dynamics in sim-to-real transfer, symbolic labels and automaton states in temporal tasking, temporally or spatially localized evidence in multimodal reasoning, and executable control targets such as click points, endoscopic motion commands, or segmentation prompts. A unifying property is that the grounded variable is no longer a passive annotation channel. It changes the effective state seen by the policy, the rollouts generated during optimization, or the reward used to compare candidate trajectories.

2. Grounding environment dynamics and learned simulators

In sim-to-real transfer, grounding-aware RL is instantiated most explicitly by Reinforced Grounded Action Transformation. The real environment is an MDP with transition kernel $T_{real}$, the simulator differs only by $T_{sim}$, and the grounding module sits between policy output and simulator execution. The target policy outputs $a_t=\pi_\theta(s_t)$; the action transformer receives $(s_t,a_t)$ and produces $\hat a_t=g_\phi(s_t,a_t)$, or equivalently a residual $\Delta a_t=\hat a_t-a_t$. Instead of GAT’s supervised composition $g(s_t,a_t)=f^{-1}_{sim}(s_t,f_{real}(s_t,a_t))$, RGAT trains $g_\phi$ with RL using reward $$R_{AT}(\hat s_{t+1},s_{t+1})=-\|f_\psi(s_t,a_t)-s_{t+1}\|^2$$, alternates grounding and policy improvement with TRPO, and uses 50 real-world trajectories per grounding step. The reported behavior is that GAT and RGAT are similar with a shallow InvertedPendulum policy, but with a deeper network GAT fails to reach optimal real-world performance whereas RGAT achieves near-optimal return; on Hopper, GAT barely improves over the non-transfer baseline, whereas RGAT reaches the direct-real-training reference in about three grounding steps (Karnan et al., 2020).

A later extension moves the same logic from action transformation to generative dynamics. Reinforcement Learning with World Grounding treats a pretrained world model $W_\theta$ as a stochastic policy over rollouts conditioned on $T_{sim}$0, samples groups of candidate futures, and scores them with frozen verifiers using translation alignment, rotation alignment, depth temporal reprojection, and video quality. GrndCtrl instantiates this with GRPO-style group-relative normalization so that world-model post-training is driven by physically verifiable spatial and temporal invariants rather than reconstruction loss alone. The clearest gains appear in counterfactual evaluation: on CODa, counterfactual translation error drops from 71.5 to 40.7; on SCAND, it drops from 315.9 to 220.1; on CityWalk, it drops from 13.1 to 4.7 (He et al., 1 Dec 2025).

These formulations share a specific conceptual shift. Grounding is no longer a pre-processing stage that corrects an environment once and then disappears. It becomes part of the RL objective itself, either as a learned action transformation that changes simulator interaction or as a reward-aligned dynamics model whose sampled rollouts are ranked by geometric consistency.

3. Language, symbols, and formal task semantics

Language-grounded transfer work treats text not as an instruction stream but as a source of dynamics and reward semantics. A state can be represented cellwise by $T_{sim}$1, where $T_{sim}$2 encodes entity identity and $T_{sim}$3 encodes an entity-aligned description; this representation feeds a VIN planner and a DQN head, and the reported gains reach up to 14% absolute improvement in average reward and 11.5% absolute improvement in initial reward (Narasimhan et al., 2017). GLAM moves a related intuition into online LLM control: the environment is a goal-augmented partially observable textual RL problem, the LLM scores admissible text actions by conditional likelihood, a value head is added, and PPO fine-tunes the full FLAN-T5 policy so that misaligned priors about action semantics, spatial language, or domain-specific vocabulary are corrected through interaction (Carta et al., 2023).

In temporally structured tasks, grounding-aware RL centers on learning the observation-to-symbol interface that earlier LTL and reward-machine methods assumed. In sub-symbolic environments, the grounder is a probabilistic map $T_{sim}$4, the most likely proposition progresses the current LTL formula, and Neural Reward Machines provide a differentiable reward-prediction loss $T_{sim}$5 that trains the grounder from raw observations and sparse rewards. The reported performance is comparable to using the true symbol grounding on most settings, and the grounder often converges within roughly the first million frames (Pannacci et al., 10 Feb 2026).

Ground-Compose-Reinforce adopts a more explicitly neurosymbolic route. It learns $T_{sim}$6 from a fixed trajectory-label dataset, compiles tasks into Reward Machines, and uses the learned labels to self-generate reward and compositional shaping. The online agent acts in the surrogate RM-MDP $T_{sim}$7, while primitive value functions over $T_{sim}$8 and $T_{sim}$9 supply a shaping potential for sparse tasks. In GeoGrid, GCR reaches $a_t=\pi_\theta(s_t)$0 on Sequence and $a_t=\pi_\theta(s_t)$1 on Safety; in DrawerWorld it reaches $a_t=\pi_\theta(s_t)$2 on Pickup-Each-Box although the offline dataset contains mean 0 and max 0 for that task (Li et al., 14 Jul 2025).

The common structural move in these papers is to internalize grounding into the task interface. Symbols, propositions, and formal task states are not externally supplied annotations; they are learned objects that determine reward execution, temporal progression, or value composition.

4. Temporal and visual grounding in multimodal reasoning

Temporal grounding in video was among the earliest explicit grounding-as-control formulations. “Read, Watch, and Move” models the current hypothesis as state $a_t=\pi_\theta(s_t)$3, uses 7 discrete actions to move start and end boundaries or STOP, and rewards actions according to whether tIoU improves, with the policy observing 10 or less clips per video. It reports $a_t=\pi_\theta(s_t)$4 of 36.9 on DenseCaption and 36.7 on Charades-STA (He et al., 2019). BAR extends the same idea to weak supervision by replacing unavailable temporal-IoU rewards with a cross-modal alignment evaluator and using $a_t=\pi_\theta(s_t)$5; the method abandons sliding windows, uses adaptive action amplitude, and outperforms prior weakly supervised methods on Charades-STA and ActivityNet (Wu et al., 2020).

Recent video MLLM work makes grounding variables explicit in reward design rather than only in the policy state. MUSEG trains Qwen2.5-VL with GRPO on timestamp-aware multi-segment grounding, using segment matching reward $a_t=\pi_\theta(s_t)$6, timestamp reward $a_t=\pi_\theta(s_t)$7, and format reward $a_t=\pi_\theta(s_t)$8, together with a phased recipe that keeps $a_t=\pi_\theta(s_t)$9 only for the first 400 of 900 steps. On THUMOS14 it reaches 29.7, on THUMOS15 29.3, and on E.T. Bench subset average 35.9 (Luo et al., 27 May 2025). ViaRL instead treats frame selection as a rule-based RL problem: a selector outputs > and <index>, a stronger answer model receives the selected frames, and answer correctness becomes the main reward. On MLVU Needle QA, Qwen2.5-VL rises from 58.6 to 73.5 using only 8 frames (Xu et al., 21 May 2025).

Grounded visual reasoning papers generalize the same principle from time to space and from single images to multi-image contexts. Ground-R1 separates a grounding phase that produces evidence boxes from an answering phase conditioned on the resulting crops; despite training without box supervision, it raises VisCoT average from 72.5 to 81.7 and its box GIoU reaches about 67.5 during training (Cao et al., 26 May 2025). MGPO makes grounding a multi-turn action in RL: the first turn outputs coordinates, the environment returns a crop from the original high-resolution image, and the second turn answers. With only binary final-answer reward, it improves over GRPO by 5.4 on MME-Realworld and 5.2 on V* Bench, and Qwen2.5-VL-7B with 21K samples surpasses OpenAI’s o1 and GPT-4o on V* Bench (Huang et al., 8 Jul 2025). UniVG-R1 couples cold-start CoT grounding SFT with IoU-plus-format GRPO and difficulty-aware weighting, producing a 9.1% improvement over the previous method on MIG-Bench and an average improvement of 23.4% in zero-shot performance across four image and video reasoning grounding benchmarks (Bai et al., 20 May 2025). MIRG-RL extends the same agenda to multi-image grounding through a two-stage training paradigm that combines supervised fine-tuning with annotated trajectories and image-aware reinforcement learning optimization, together with dual reward functions for objects and images (Zheng et al., 26 Sep 2025).

Across these works, grounding is not merely an auxiliary explanation. It changes what visual evidence is seen next, what segments are retained, or how candidate trajectories are ranked.

5. GUI, medical, and instruction-conditioned control

GUI grounding makes the grounding-aware character of RL especially explicit because the final output must be an executable pixel coordinate. “Measure Twice, Click Once” replaces static consistency strategies with a co-evolving proposer and visual critic: the proposer samples $(s_t,a_t)$0, the candidates are rendered on the screenshot, the critic ranks them, and GRPO uses proposer rewards built from Gaussian candidate quality and coverage and critic rewards built from top-1 quality and NDCG, modulated by maturity-aware co-evolution. The reported effect is a simultaneous improvement in Oracle@5 and Top-1, with up to 17.2% relative improvement in grounding capability (Wang et al., 23 Apr 2026). GUI-C$(s_t,a_t)$1 simplifies the control problem further: GUI-D keeps rollout-informative samples and assigns difficulty scores, while the policy predicts click point and box, and area-gated coarse-to-fine refinement plus improvement-aware stage rewards determine whether to crop again. The method reaches 46.4 on ScreenSpot-Pro, 85.8 on ScreenSpot, and 87.8 on ScreenSpot-v2; replacing threshold-based crop decisions with model self-decision raises inference time from 3.05 s to 24.81 s and reduces accuracy (Li et al., 29 May 2026).

In medical and robotic control, grounding-aware RL often optimizes structured spatial outputs that mediate action or diagnosis. BiliVLA conditions a VLA policy on stage-specific instructions and jointly predicts target category, grounded bounding box, and discrete 3-DoF action; GRPO uses $(s_t,a_t)$2, and the full system reaches total mIoU 0.9625, action precision 91.96%, and overall success rate 84.85% in real-world phantom experiments (Lin et al., 22 Jun 2026). MedReasoner similarly decouples reasoning from segmentation: a clinical reasoning module emits a JSON answer containing bbox, points_1, and points_2, a frozen MedSAM2 converts these prompts into a mask, and GRPO rewards format and spatial accuracy. The reported test results are IoU 32.42, pDice 26.55, and Dice 37.78, while RL-Soft eliminates grounding refusals (Yan et al., 11 Aug 2025). In chest X-ray grounding, GRPO with a Hungarian-matched soft-F1 reward turns coordinate-string prediction into overlap-aware optimization; Qwen3-VL improves from 11.1 to 79.0 on anatomy grounding and from 19.0 to 79.4 on MS-CXR phrase grounding, while RL also improves strong in-domain RadVLM baselines and explicit thinking does not further improve results (Gundersen et al., 11 Dec 2025).

These control-oriented papers make a distinctive design choice. The grounded output is directly executable: a click point, a crop target, a bounding box paired with an action, or a segmentation prompt that a frozen expert consumes. That executable status sharply constrains reward design and makes grounding failure immediately task-relevant.

6. Methodological regularities, limitations, and open problems

A recurring pattern is that grounding moves from a frozen interface to a learnable control variable. In RGAT, the simulator correction map is itself a policy; in sub-symbolic LTL, the grounder determines formula progression; in ViaRL, the selector chooses frames before answer generation; in MedReasoner, the reasoner emits prompts that a frozen segmenter can execute (Karnan et al., 2020, Pannacci et al., 10 Feb 2026, Xu et al., 21 May 2025, Yan et al., 11 Aug 2025). A second pattern is the preference for verifiable, geometry-aware rewards and relative ranking. Recent work repeatedly uses GRPO or related group-relative objectives with IoU, soft-F1, NDCG, pose alignment, or answer correctness as reward components rather than unconstrained learned reward models (He et al., 1 Dec 2025, Wang et al., 23 Apr 2026, Gundersen et al., 11 Dec 2025).

Several papers also show that RL rarely replaces representation learning from scratch. Cold-start CoT grounding SFT is decisive in UniVG-R1, the best radiology RL results come from RL on top of a domain-specific RadVLM, the pretrained frozen LTL module performs best in sub-symbolic LTL grounding, and MGPO needs a fixed multi-turn template because the base model does not autonomously trigger grounding during rollout (Bai et al., 20 May 2025, Gundersen et al., 11 Dec 2025, Pannacci et al., 10 Feb 2026, Huang et al., 8 Jul 2025). This pattern indicates that grounding-aware RL is typically most effective as post-training or alternating optimization over an already competent multimodal backbone, not as a replacement for perceptual or semantic pretraining.

The limitations are likewise consistent. Reward misspecification and reward hacking appear when verifiers are exploitable or too narrow; the GrndCtrl appendix reports reward hacking and noisy generations, and report-generation RL in radiology can optimize a single metric while shortening reports drastically (He et al., 1 Dec 2025, Gundersen et al., 11 Dec 2025). Some formulations still require online real interaction or validate only on surrogate environments, as in RGAT’s modified simulators rather than physical robots (Karnan et al., 2020). Others operate under strong structural assumptions, such as fixed proposition vocabularies, co-safe LTL, and the Declare assumption (Pannacci et al., 10 Feb 2026), or are evaluated only in phantom settings with sample-level rather than trajectory-level rewards (Lin et al., 22 Jun 2026). Taken together, these results indicate that grounding-aware reinforcement learning is most mature when the grounded variable is executable, the reward is externally verifiable, and the task interface admits a useful decomposition between reasoning, grounding, and action.