Parent-Guided Semantic Reward Model

• PGSRM is a lightweight reward framework for RL alignment that uses cosine similarity between parent and child model outputs to generate continuous, semantically meaningful rewards.
• It employs a teacher–student paradigm with frozen text embeddings (e.g., Numberbatch and text-embedding-3-large) to provide reliable reward signals without additional training.
• Experimental results show that PGSRM stabilizes PPO policy updates and improves sample efficiency compared to binary reward baselines.

The Parent-Guided Semantic Reward Model (PGSRM) is a lightweight reward framework for reinforcement learning (RL) alignment of transformer LLMs. PGSRM eschews traditional binary correctness checks, human preference datasets, and trained neural reward models, instead utilizing a single, dense reward: the cosine similarity between a parent model’s reference output embedding and a child model’s generated output embedding for the same prompt. This approach offers annotation-free, semantically meaningful reward signals for policy-gradient RL algorithms without additional model training (Plashchinsky, 7 Dec 2025).

1. Formalization of Semantic Reward Function

PGSRM operates on a teacher–student paradigm involving a fixed “parent” policy $\pi_p$ (e.g., GPT-4-class) and a trainable “child” policy $\pi_\theta$ (e.g., GPT-2 variant). Given a text prompt $s$, the parent generates a reference answer $a_p = \pi_p(s)$; the child policy stochastically generates $a_c \sim \pi_\theta(\cdot|s)$. Both outputs are embedded using a frozen text-embedding function $f: \mathcal{A} \rightarrow \mathbb{R}^d$, and then $\ell_2$-normalized: $e_p = f(a_p)/\|f(a_p)\|_2$, $e_c = f(a_c)/\|f(a_c)\|_2$.

The reward is defined by the cosine similarity between $e_p$ and $e_c$, truncated at zero, and with an optional sharpening exponent $\alpha \geq 1$ controlling density and curvature:

$$\operatorname{cos}(e_p, e_c) = e_p^\top e_c \in [-1, 1], \quad R_\text{PGSRM}(s, a_c, a_p) = [\max(0, \operatorname{cos}(e_p, e_c))]^\alpha$$

This yields a continuous reward $R\in [0,1]$, where increasing $\alpha$ penalizes partial matches, emphasizing high semantic overlap.

2. Embedding Models and Data Processing

PGSRM leverages off-the-shelf, frozen embeddings:

• Numberbatch (ConceptNet 5.5): 300-dimensional token-level vectors.
• text-embedding-3-large (OpenAI): 1536-dimensional sequence-level vectors.

Output sequences—typically the entire model response, possibly JSON-wrapped—are fed to the embedding function $f$, and pooling yields a single embedding vector. Both parent and child outputs are thus mapped into the same semantic space, allowing the cosine similarity to serve as a proxy for meaning overlap. No further fine-tuning of the embedding models is performed.

3. Integration into Proximal Policy Optimization (PPO)

PGSRM is implemented within conventional single-step PPO. Each training iteration proceeds as follows:

1. Prompt Sampling: Sample a minibatch of $N$ prompts $\{s_i\}$.
2. Reference Generation: For each $s_i$, retrieve cached parent output $a_{p,i} = \pi_p(s_i)$.
3. Child Response: Sample $a_{c,i} \sim \pi_\theta(\cdot|s_i)$.
4. Reward Calculation: Embed, normalize, and compute $r_i = R_\text{PGSRM}(s_i, a_{c,i}, a_{p,i})$.
5. Value/Advantage Estimation: Estimate $V_\phi(s_i)$; form $A_i = r_i - V_\phi(s_i)$.
6. Actor Update: Minimize PPO surrogate (with no ratio clipping), plus light KL penalty and entropy regularization:

$$L_\text{actor} = -\mathbb{E}_i[\log \pi_\theta(a_{c,i}|s_i)\cdot A_i] + \lambda_\text{KL}\cdot D_\text{KL}[\pi_\theta(\cdot|s_i)\|\pi_\text{ref}(\cdot|s_i)] - \beta_\text{ent}\cdot \mathcal{H}[\pi_\theta(\cdot|s_i)]$$

Typical hyperparameters: actor learning rate $1\times10^{-5}$, critic $1\times10^{-4}$, max-grad-norm $1.0$, batch size $50$ (GPT-2 Small), $10$ (GPT-2 Large), $100\,000$ single-step episodes per run. Adaptive $\lambda_\text{KL}\sim 5\times 10^{-5}$, entropy regularization $\beta_\text{ent}=0.01$.

1. Critic Update: Minimize $$L_\text{critic}=0.5\,\mathbb{E}_i[(V_\phi(s_i)-r_i)^2]$$.
2. KL Adaptation: Optionally tune $\lambda_\text{KL}$ to keep $D_\text{KL}$ within $0.5$–$0.8$.

4. Experimental Results and Diagnostics

Five single-step alignment tasks were evaluated with pre-cached GPT-4-class parent outputs and GPT-2 child models:

Task	Prompt Example	Parent Output	PGSRM Reward	Binary Baseline
Color mixing	"red + blue"	"purple"	≈0.8 final	~0
Antonym generation	"Opposite of 'hot'"	"cold"	≈0.75 plateau	~0
Word categorization	"Category of 'apple'"	"fruit"	0.1→0.45	~0
Exact-string copying	"Copy this sentence: ..."	same string	0.6–0.8 mid	~0
Sentiment inversion	"Rewrite as sad: ..."	"I feel empty."	≈0.4 stable	~0

Key experimental findings:

• Binary reward runs remain near-zero across all episodes—exact matches are too sparse for gradient-based learning.
• PGSRM produces rapid phase transitions in average reward: e.g., color mixing jumps to ≈0.7 after ∼10$^4$ episodes, plateauing around ≈0.8.
• PPO with PGSRM yields stable policy entropy ($0.5$–$1.5$ nats), bounded KL divergence ($\approx 0.01$–$0.1$), and smooth critic loss decrease.
• Binary reward experiments display oscillatory entropy and KL, failing to establish a learning signal.

5. Qualitative Analysis and Case Studies

PGSRM’s semantic reward surface supports gradual improvement and refinement. For color mixing, a child model that first generates “green” ($\cos\approx0.85$ with “light green”) incrementally adapts toward the parent’s “light green” ($\cos\to 0.98$), whereas binary reward only grants signal for exact matches. In antonym tasks, intermediate proposals (“tired” vs. “sluggish”) receive partial credit ($\cos\approx0.7$), facilitating convergence. For sentiment inversion, near-miss responses (“I feel down”) are scored within $0.6$–$0.8$, allowing movement toward parent style without stagnation at zero reward.

6. Limitations and Implementation Guidance

PGSRM is fundamentally an imitation mechanism: the child model cannot surpass parent response quality and inherits any parent biases and safety flaws. Embedding model quality is critical; poor embeddings produce noisy, misaligned rewards. Recommended practices include selecting high-capacity, domain-relevant embedders (e.g., text-embedding-3-large for long text), ensuring $\ell_2$-normalization, and verifying that cosine correlates with task success. The sharpening exponent $\alpha$ (typically $2$–$8$) should be tuned: lower values yield smoother gradients, higher emphasize near-perfect matches.

Reward hacking is a notable risk: generic responses may optimize the embedding-based reward but fail downstream checks. Hybrid reward designs—combining semantic similarity with task-specific constraints—may mitigate such exploits.

PGSRM’s PPO configuration omits ratio clipping and uses light KL penalty; this accelerates response to dense rewards, though clipping may be reintroduced for sensitive or larger models.

7. Summary and Future Prospects

PGSRM provides an annotation-free approach for leveraging high-capacity teacher models to semantically guide reinforcement learning of transformer policies. Dense, embedding-based rewards yield stable PPO dynamics and improved sample efficiency across diverse synthetic alignment tasks. The principal constraints are strict teacher-imitation and dependence on embedding fidelity. This suggests PGSRM is a practical alternative to RLHF-style reward modeling for smaller models, with clear recommendations for embedding selection and algorithmic tuning (Plashchinsky, 7 Dec 2025).