---
title: Fixed-effect Semantic Reward (FSR)
url: https://www.emergentmind.com/topics/fixed-effect-semantic-reward-fsr
type: topic
---

# Fixed-effect Semantic Reward (FSR)

Fixed-effect Semantic Reward (FSR) is a reward modulation scheme introduced in the Trade-R1 framework to address the challenge of training reinforcement learning (RL) agents, specifically Large Language Models (LLMs), for financial decision-making in stochastic environments. FSR augments raw, noisy market returns with a constant semantic alignment incentive computed via a structured process-level reasoning verification metric. This approach aims to stabilize reasoning quality by decoupling semantic fidelity from return volatility, preserving reasoning alignment even under adverse market conditions [2601.03948].

## 1. Formal Definition and Mathematical Formulation

FSR operates in a contextual RL setting, formalized as follows. Let $x$ denote the market context, comprising financial news and data, and let $\pi_\theta(y | x)$ be a policy that generates a reasoning chain $c$ and a final decision $d$. The observed market reward is given by $r \in \mathbb{R}$, typically a multi-day excess return from historical backtesting. A semantic similarity score $s \in [0,1]$, computed via the triangular consistency metric, quantifies the alignment between evidence, reasoning, and decision.

FSR is defined by the reward integration function:
$$
G_{\mathrm{FSR}}(r, s) = r + \lambda \cdot s
$$
where $\lambda$ is a fixed hyperparameter controlling the weight of semantic alignment. In Trade-R1, $\lambda$ is set to $2$, yielding
$$
G_{\mathrm{FSR}}(r, s) = r + 2s
$$
The RL objective becomes maximizing the expectation:
$$
J(\theta) = \mathbb{E}_{x \sim D, y \sim \pi_\theta(.|x)} [G_{\mathrm{FSR}}(r(x, y), s(x, y))]
$$

## 2. Triangular Consistency Metric and Semantic Scoring

FSR leverages a triangular consistency metric to compute the semantic similarity score $s$. For each sample, a concise evidence snippet $E$ relevant to the stock pick is retrieved. An LLM-based judge then scores the following terms, each in $[0, 1]$:
- **Factuality:** $S_{E \leftrightarrow c}$ (“Does the reasoning $c$ faithfully reflect the facts in $E$?”)
- **Deduction:** $S_{c \leftrightarrow d}$ (“Does the final decision $d$ logically follow from $c$?”)
- **Consistency:** $S_{E \leftrightarrow d}$ (“Is $d$ supported by $E$?”)

The overall semantic score is the arithmetic mean:
$$
s = \frac{S_{E \leftrightarrow c} + S_{c \leftrightarrow d} + S_{E \leftrightarrow d}}{3}
$$
In FSR, this score is scaled by the fixed coefficient and added to the raw market return, providing a constant step-wise semantic incentive regardless of return magnitude or sign.

## 3. Rationale for Fixed-Effect Formulation

FSR's fixed-effect design is motivated by multiple factors:

- **Stability under Volatility:** Financial returns $r$ exhibit noisy behavior that can obscure semantic alignment signals. The fixed bonus $\lambda s$ ensures the agent consistently receives semantic grounding encouragement, even when $|r|$ is small or highly volatile.
- **Decoupled Alignment Gradient:** The policy gradient $\nabla_\theta G$ separates into $\nabla_\theta r$ (traditional policy gradient) and a constant $\nabla_\theta s$ for reasoning, preventing market noise from distorting the semantic alignment update.
- **Simplicity:** A single hyperparameter $\lambda$ controls the trade-off between economic returns and reasoning fidelity. There is no conditional scaling or piecewise treatment based on $r$ or $s$.

*A plausible implication is that this simplicity makes the system robust to hyperparameter tuning and easier to analyze, but may limit expressiveness in settings where adaptivity is required.*

## 4. Integration with Trade-R1 RL Workflow

FSR is incorporated into Trade-R1 via a grouped-sampling variant of Proximal Policy Optimization (GRPO). The training workflow proceeds as follows:

```text
Initialize policy πθ
for each training batch do
  sample a batch of contexts {xᵢ}
  for each xᵢ:
    generate G rollouts yᵢ₁,…,yᵢG ∼ π_{θ_old}(.|xᵢ)
    for each sample yᵢⱼ:
      compute market return rᵢⱼ (backtest)
      retrieve evidence Eᵢⱼ via embedding & reranking
      score (S_{E↔c}, S_{c↔d}, S_{E↔d}) with LLM judge → sᵢⱼ=(…)/3
      Gᵢⱼ = rᵢⱼ + 2·sᵢⱼ     // Fixed-effect semantic reward
    end
    compute group-normalized advantages:
      Aᵢⱼ = (Gᵢⱼ − meanₖ Gᵢₖ)/stdₖ Gᵢₖ
    take PPO-style gradient step on θ using {Aᵢⱼ, log πθ(yᵢⱼ|xᵢ)}
  end
end
```

Key points:
- The additive term $2s_{ij}$ applies uniformly, irrespective of return volatility or sign.
- Group normalization mitigates non-stationary trends in market returns prior to advantage computation, ensuring stable updates.

## 5. Empirical Performance and Illustrative Case Study

FSR's efficacy is demonstrated in the China A-Share market (July–October 2025):

- **Market-Only Policy:** Cumulative return 37.62%; Sharpe ratio 3.028; Semantic Similarity $\approx 0.44$; Hallucination rate $0.225$.
- **FSR-Augmented Policy:** Cumulative return 39.38%; Sharpe ratio $\approx 3.07$; Mean triangle-score $s \approx 0.956$; Hallucination rate $0.004$.

The net asset value (NAV) curve under FSR remains consistently higher than market-only, with dramatically improved reasoning fidelity and reduced hallucination (unsupported reasoning chains). This suggests FSR is highly effective for in-distribution semantic grounding. 

## 6. Benefits, Limitations, and Cross-Market Observations

**Benefits:**
- Strong, stable in-distribution returns, paired with extremely high semantic alignment ($s \approx 0.956$).
- The constant semantic term prevents the model from disregarding reasoning integrity in periods of low or negative returns.

**Limitations:**
- The additive nature of the semantic reward can be under-weighted when $r < 0$: models may exploit this by lowering $s$ to reduce cumulative penalties, impairing out-of-distribution robustness.
- Reduced cross-market generalization: In US market tests, FSR yields lower returns than both Market-Only and Dynamic-effect Semantic Reward (DSR), coupled with only moderate gains in reasoning consistency.
- FSR does not suppress excess variance from noisy positive returns, in contrast to DSR's adaptive regularization.

**Empirical Comparison Table**

| Setting                | Strategy      | Cumulative Return (%) | Sharpe Ratio | Semantic Similarity | Hallucination Rate |
|------------------------|--------------|----------------------|--------------|---------------------|--------------------|
| China A-Share (test)   | Market-Only  | 37.62                | 3.028        | 0.437               | 0.225              |
|                        | FSR          | 39.38                | 3.065        | 0.956               | 0.0039             |
| US Market (OOD)        | Market-Only  | 12.63                | 1.712        | 0.659               | 0.141              |
|                        | FSR          | 11.40                | 1.473        | 0.758               | 0.092              |

*A plausible implication is that FSR is best suited to environments where high reasoning fidelity is prioritized and the reward distribution is not adversarial; its lack of adaptivity may constrain performance in more volatile or distribution-shifting markets.*

## 7. Significance and Theoretical Implications

Fixed-effect Semantic Reward establishes a straightforward mechanism for enforcing reasoning integrity in RL agents operating under noisy reward regimes. By championing process-level verification, it supports semantic alignment at each decision step, unaffected by economic reward fluctuations. Its major trade-off is simplicity versus adaptivity: FSR delivers high in-distribution fidelity but falls short as environments change or stochasticity increases. Dynamic-effect alternatives, such as DSR, may provide superior robust generalization and regularize variance, though with greater operational complexity.

FSR marks an important step in integrating retrieval-based reasoning verification with RL reward design, especially for high-stakes, stochastic applications such as financial asset selection [2601.03948].

Source: https://www.emergentmind.com/topics/fixed-effect-semantic-reward-fsr