Parent-Guided Semantic Reward Model

• PGSRM is a lightweight reward framework for RL alignment that uses cosine similarity between parent and child model outputs to generate continuous, semantically meaningful rewards.
• It employs a teacher–student paradigm with frozen text embeddings (e.g., Numberbatch and text-embedding-3-large) to provide reliable reward signals without additional training.
• Experimental results show that PGSRM stabilizes PPO policy updates and improves sample efficiency compared to binary reward baselines.

The Parent-Guided Semantic Reward Model (PGSRM) is a lightweight reward framework for reinforcement learning (RL) alignment of transformer LLMs. PGSRM eschews traditional binary correctness checks, human preference datasets, and trained neural reward models, instead utilizing a single, dense reward: the cosine similarity between a parent model’s reference output embedding and a child model’s generated output embedding for the same prompt. This approach offers annotation-free, semantically meaningful reward signals for policy-gradient RL algorithms without additional model training (Plashchinsky, 7 Dec 2025).

1. Formalization of Semantic Reward Function

PGSRM operates on a teacher–student paradigm involving a fixed “parent” policy $\pi_p$ (e.g., GPT-4-class) and a trainable “child” policy $\pi_\theta$ (e.g., GPT-2 variant). Given a text prompt $s$, the parent generates a reference answer $a_p = \pi_p(s)$; the child policy stochastically generates $a_c \sim \pi_\theta(\cdot|s)$. Both outputs are embedded using a frozen text-embedding function $f: \mathcal{A} \rightarrow \mathbb{R}^d$, and then $\ell_2$-normalized: $e_p = f(a_p)/\|f(a_p)\|_2$, $e_c = f(a_c)/\|f(a_c)\|_2$.

The reward is defined by the cosine similarity between $e_p$ and $\pi_\theta$0, truncated at zero, and with an optional sharpening exponent $\pi_\theta$1 controlling density and curvature:

$\pi_\theta$2

This yields a continuous reward $\pi_\theta$3, where increasing $\pi_\theta$4 penalizes partial matches, emphasizing high semantic overlap.

2. Embedding Models and Data Processing

PGSRM leverages off-the-shelf, frozen embeddings:

• Numberbatch (ConceptNet 5.5): 300-dimensional token-level vectors.
• text-embedding-3-large (OpenAI): 1536-dimensional sequence-level vectors.

Output sequences—typically the entire model response, possibly JSON-wrapped—are fed to the embedding function $\pi_\theta$5, and pooling yields a single embedding vector. Both parent and child outputs are thus mapped into the same semantic space, allowing the cosine similarity to serve as a proxy for meaning overlap. No further fine-tuning of the embedding models is performed.

3. Integration into Proximal Policy Optimization (PPO)

PGSRM is implemented within conventional single-step PPO. Each training iteration proceeds as follows:

1. Prompt Sampling: Sample a minibatch of $\pi_\theta$6 prompts $\pi_\theta$7.
2. Reference Generation: For each $\pi_\theta$8, retrieve cached parent output $\pi_\theta$9.
3. Child Response: Sample $s$0.
4. Reward Calculation: Embed, normalize, and compute $s$1.
5. Value/Advantage Estimation: Estimate $s$2; form $s$3.
6. Actor Update: Minimize PPO surrogate (with no ratio clipping), plus light KL penalty and entropy regularization:

$s$4

Typical hyperparameters: actor learning rate $s$5, critic $s$6, max-grad-norm $s$7, batch size $s$8 (GPT-2 Small), $s$9 (GPT-2 Large), $a_p = \pi_p(s)$0 single-step episodes per run. Adaptive $a_p = \pi_p(s)$1, entropy regularization $a_p = \pi_p(s)$2.

1. Critic Update: Minimize $a_p = \pi_p(s)$3.
2. KL Adaptation: Optionally tune $a_p = \pi_p(s)$4 to keep $a_p = \pi_p(s)$5 within $a_p = \pi_p(s)$6–$a_p = \pi_p(s)$7.

4. Experimental Results and Diagnostics

Five single-step alignment tasks were evaluated with pre-cached GPT-4-class parent outputs and GPT-2 child models:

Task	Prompt Example	Parent Output	PGSRM Reward	Binary Baseline
Color mixing	"red + blue"	"purple"	≈0.8 final	~0
Antonym generation	"Opposite of 'hot'"	"cold"	≈0.75 plateau	~0
Word categorization	"Category of 'apple'"	"fruit"	0.1→0.45	~0
Exact-string copying	"Copy this sentence: ..."	same string	0.6–0.8 mid	~0
Sentiment inversion	"Rewrite as sad: ..."	"I feel empty."	≈0.4 stable	~0

Key experimental findings:

• Binary reward runs remain near-zero across all episodes—exact matches are too sparse for gradient-based learning.
• PGSRM produces rapid phase transitions in average reward: e.g., color mixing jumps to ≈0.7 after ∼10$a_p = \pi_p(s)$8 episodes, plateauing around ≈0.8.
• PPO with PGSRM yields stable policy entropy ($a_p = \pi_p(s)$9–$a_c \sim \pi_\theta(\cdot|s)$0 nats), bounded KL divergence ($a_c \sim \pi_\theta(\cdot|s)$1–$a_c \sim \pi_\theta(\cdot|s)$2), and smooth critic loss decrease.
• Binary reward experiments display oscillatory entropy and KL, failing to establish a learning signal.

5. Qualitative Analysis and Case Studies

PGSRM’s semantic reward surface supports gradual improvement and refinement. For color mixing, a child model that first generates “green” ($a_c \sim \pi_\theta(\cdot|s)$3 with “light green”) incrementally adapts toward the parent’s “light green” ($a_c \sim \pi_\theta(\cdot|s)$4), whereas binary reward only grants signal for exact matches. In antonym tasks, intermediate proposals (“tired” vs. “sluggish”) receive partial credit ($a_c \sim \pi_\theta(\cdot|s)$5), facilitating convergence. For sentiment inversion, near-miss responses (“I feel down”) are scored within $a_c \sim \pi_\theta(\cdot|s)$6–$a_c \sim \pi_\theta(\cdot|s)$7, allowing movement toward parent style without stagnation at zero reward.

6. Limitations and Implementation Guidance

PGSRM is fundamentally an imitation mechanism: the child model cannot surpass parent response quality and inherits any parent biases and safety flaws. Embedding model quality is critical; poor embeddings produce noisy, misaligned rewards. Recommended practices include selecting high-capacity, domain-relevant embedders (e.g., text-embedding-3-large for long text), ensuring $a_c \sim \pi_\theta(\cdot|s)$8-normalization, and verifying that cosine correlates with task success. The sharpening exponent $a_c \sim \pi_\theta(\cdot|s)$9 (typically $f: \mathcal{A} \rightarrow \mathbb{R}^d$0–$f: \mathcal{A} \rightarrow \mathbb{R}^d$1) should be tuned: lower values yield smoother gradients, higher emphasize near-perfect matches.

Reward hacking is a notable risk: generic responses may optimize the embedding-based reward but fail downstream checks. Hybrid reward designs—combining semantic similarity with task-specific constraints—may mitigate such exploits.

PGSRM’s PPO configuration omits ratio clipping and uses light KL penalty; this accelerates response to dense rewards, though clipping may be reintroduced for sensitive or larger models.

7. Summary and Future Prospects

PGSRM provides an annotation-free approach for leveraging high-capacity teacher models to semantically guide reinforcement learning of transformer policies. Dense, embedding-based rewards yield stable PPO dynamics and improved sample efficiency across diverse synthetic alignment tasks. The principal constraints are strict teacher-imitation and dependence on embedding fidelity. This suggests PGSRM is a practical alternative to RLHF-style reward modeling for smaller models, with clear recommendations for embedding selection and algorithmic tuning (Plashchinsky, 7 Dec 2025).