Language-Driven Reward Specification

• Language-driven reward specification is a paradigm that uses natural language inputs to define reinforcement learning reward functions, enabling intuitive goal alignment.
• It employs techniques such as LLM-based code synthesis, semantic encoders, and symbolic specification to translate linguistic directives into executable rewards.
• This approach enhances interpretability, reduces manual reward engineering, and has wide applications in robotics, gaming, and multi-agent systems.

Language-driven reward specification is the process of using natural language descriptions, instructions, or preferences to define reward functions for reinforcement learning (RL) agents. This paradigm shifts reward engineering from hand-coded, numerical signals to more interpretable and accessible specifications, often leveraging LLMs or other language-grounding architectures to automatically translate linguistic inputs into executable reward mechanisms. The field encompasses techniques spanning reward shaping, symbolic specification languages, code generation, preference learning, and semantic alignment across domains from robotics and games to multi-agent coordination.

1. Conceptual Foundations and Motivations

Traditional reinforcement learning depends on precise, manually crafted reward functions to drive agent behavior. However, numerically encoding complex, high-level, or multifaceted objectives is labor intensive and error-prone, limiting scalability and alignment with user intent. Language, as the natural modality for expressing goals and task criteria, offers a semantically rich, flexible, and user-accessible interface for specifying RL objectives.

Language-driven reward specification enables (1) non-expert users to rapidly articulate desired agent behaviors; (2) representation of complex, compositional, or non-Markovian objectives; and (3) dynamic adaptation of rewards in response to shifting requirements or environment changes. This is realized either by directly grounding language in executable reward models, or by using language to parameterize, shape, or interpret scalar rewards used in policy optimization. Early symbolic approaches used compositional task languages (Jothimurugan et al., 2020), while more recent methods employ LLMs and multimodal foundation models for code synthesis or reward function inference (Sun et al., 2024, Han et al., 2024, Rocamonde et al., 2023, Goyal et al., 2019).

2. Model Architectures and Algorithmic Pipelines

Several system designs operationalize language-driven reward specification. The core architectural motifs are:

• Code Synthesis via LLMs: LLMs are prompted with environment code, task-specific constraints, and verbal instructions to generate Python functions implementing reward logic (Han et al., 2024, Baek et al., 15 Feb 2025, Mukherjee et al., 20 Nov 2025, Baek et al., 2024). These pipelines often include iterative refinement loops where quantitative RL performance metrics are reflected back to the LLM for self-improvement [CARD, (Sun et al., 2024)], or reasoning-based prompt engineering (chain/tree-of-thought) to enhance reward quality (Baek et al., 15 Feb 2025).
• Semantic Encoders and Scoring Models: Other architectures learn joint representations of language and state/action histories, using neural networks to compute alignment or “relatedness” scores, often as potential-based shaping functions (Goyal et al., 2019). In vision-based domains, VLMs such as CLIP compute reward as the cosine similarity between an image of the current state and the embedding of a language prompt (Rocamonde et al., 2023).
• Object-centric and Symbolic Specifications: Methods such as OCALM extract object-centric abstractions from state observations and use LLMs to synthesize interpretable, relational reward code (Kaufmann et al., 2024). Symbolic specification languages, e.g., SPECTRL or RML, support highly expressive, compositional reward structures, encompassing temporal logic, sequencing, counting, and parameterization (Jothimurugan et al., 2020, Donnelly et al., 17 Oct 2025).
• Preference and Feedback-Based Reward Induction: Reward models can also be trained from language-based preference data, either via human-annotated success/failure pairs, automatically mined follow-up responses (“Follow-up Likelihood as Reward”) (Zhang et al., 2024), or LLM-generated trajectory rankings (Lin et al., 2024). Learned reward models are integrated into RL as scalar critics or via potential-difference shaping.

3. Formalisms, Mathematical Criteria, and Reward Guarantees

Language-driven reward specification is formalized via mappings from natural language $L$ (or contextual utterances) and environment state/action space $(S,A)$ to scalar reward $R$. Key formulations include:

• Potential-Based Shaping:

$$r_{\text{lang}}(f_t) = \gamma \,\phi(f_t) - \phi(f_{t-1})$$

where $\phi(f_t)$ scores action histories for alignment with instruction $I$ (Goyal et al., 2019). Such shaping preserves policy invariance under certain conditions.

• Vision-Language Similarity:

$$R_\mathrm{CLIP}(s) = \frac{\mathrm{CLIP}_L(l) \cdot \mathrm{CLIP}_I(\psi(s))}{\|\mathrm{CLIP}_L(l)\|\|\mathrm{CLIP}_I(\psi(s))\|}$$

where reward is the cosine similarity between text and image embeddings (Rocamonde et al., 2023).

• Preference-Based Rewards:

$r_t = \sigma_\psi(s_t) - \sigma_\psi(s_{t-1})$

with $\sigma_\psi$ learned from LLM or human preference queries (Lin et al., 2024).

• Specification Compilation: Logical specifications $\varphi$ are compiled to automata or reward machines, endowing atomic predicates with quantitative semantics and enabling reward shaping that is policy invariant and preserves subgoal structure (Jothimurugan et al., 2020, Donnelly et al., 17 Oct 2025).
• Feedback-Driven Iteration: Card-style frameworks formalize dynamic adaptation loops, wherein rewards are evolved based on process (e.g., success rates), trajectory, or preference feedback using precise order-preservation criteria (Sun et al., 2024).

4. Empirical Evaluations and Benchmark Comparisons

Language-driven reward specification has been validated across diverse task domains, with the following empirical highlights:

Paper/Framework	Domain(s)	Core Metric	Main Finding
LEARN (Goyal et al., 2019)	Atari (Montezuma's)	Avg successes at 500k steps	+60% relative gain (1529 vs 903), 30% faster learning
Highway LLM (Han et al., 2024)	Driving (HWY-env)	Avg. success rate	+22% gain vs human-crafted baseline across densities
CARD (Sun et al., 2024)	Meta-World, ManiSkill2	Success rate, token use	Matches/exceeds Oracle on 10/12 tasks, 10–40× lower token usage
FLR (Zhang et al., 2024)	LLM preference alignment	Pairwise/RM benchmarks	Matches GPT-4-pairwise RM (no human data), boosts DPO alignment
VLC (Alakuijala et al., 2024)	Robotics (Meta-World)	Sample efficiency, success	2× sample efficiency vs sparse, +20% final success
SPECTRL (Jothimurugan et al., 2020)	Robotics/sim control	Rollout return, subgoal progress	Outperforms baselines, provides interpretable reward shaping
OCALM (Kaufmann et al., 2024)	Atari	Final returns, correlations	Matches ground-truth rewards in most games, transparent code

Consistent themes are rapid convergence, improved alignment with user-specified objectives, and reduced dependency on RL-specific engineering. However, success depends critically on environment observability, reward model capacity (e.g., VLM scale), and the robustness of prompt engineering or feedback protocols.

5. Interpretability, Expressivity, and Practical Constraints

A principal advantage of language-driven reward specification is enhanced interpretability. In frameworks such as OCALM or RML-based reward machines, the resulting reward code is plain, human-readable Python or declarative monitor syntax (Kaufmann et al., 2024, Donnelly et al., 17 Oct 2025). This enables domain experts to audit, debug, and refine reward logic without black-box dependence. Specification languages allow concise, parameterized definitions that generalize across instance families (e.g., collect $K$ wheels and $L$ engines) (Donnelly et al., 17 Oct 2025).

However, this expressivity comes with practical constraints:

• Black-box neural reward models (e.g., video-language critics) can be difficult to debug.
• Reward code generated by LLMs may fail syntactic or semantic checks and often needs iterative refinement.
• Quality of generated rewards is highly sensitive to prompt and context design.
• Pipeline overhead (LLM inference, code validation, or reward function execution) introduces computational cost, motivating reward distillation or offline evaluation schemes (Su et al., 13 Jan 2026).
• Scaling to hierarchical or extremely high-dimensional settings requires careful modularization and potentially new forms of specification languages or distributed reward modeling.

6. Domains of Application and Emerging Directions

Language-driven reward specification methods have been applied in robotics (manipulation, locomotion, drone and warehouse navigation) (Yu et al., 2023, Perez et al., 2023), multi-agent systems (Su et al., 13 Jan 2026), procedural content generation in games (Baek et al., 15 Feb 2025, Baek et al., 2024), negotiation/dialogue (Kwon et al., 2023), and simulated cybersecurity defense (Mukherjee et al., 20 Nov 2025). Recent research demonstrates generalization from offline open-embodiment datasets to new task configurations (Alakuijala et al., 2024), as well as semantic adaptation of rewards in response to nonstationary environments (Sun et al., 2024).

Key open frontiers include: fully automated, vision-to-reward pipelines that require no manual coding; integrating active preference elicitation and clarification queries; reward-critic architectures for safety and robustness; and development of standardized benchmarks for language-driven multi-agent RL.

7. Limitations and Future Perspectives

While language-driven reward specification marks a paradigm shift away from brittle, hand-designed numerical signals, several limitations remain:

• Ambiguity in language can yield unintended behaviors; prompt engineering and clarification strategies remain active research areas (Su et al., 13 Jan 2026).
• Computational cost and latency of LLM-based reward models presents scaling challenges.
• Safe reward generation—including avoidance of reward hacking and misbehavior—often requires additional verification layers or human oversight.
• In some domains, especially those with complex visual input or physics, induced reward models may fail to generalize without sufficient grounding or large-scale multimodal pretraining.
• Universal, domain-agnostic reward specification remains elusive; hierarchical decomposition and protocol design are needed for scalability (Su et al., 13 Jan 2026).

Nevertheless, ongoing advances in LLM architectures, multimodal foundation models, and formal specification languages continue to expand the capabilities and reliability of language-driven reward systems, with increasing impact in both research and real-world deployment across diverse RL settings.