Reward Model Based Counting Loss

• Reward Model Based Counting Loss is an umbrella term for methods integrating counting into reward-centric learning through internal automata, pseudo-count regression, and normalization adjustments.
• In Counting Reward Automata, counting is embedded within the reward function, while MERCI uses explicit pseudo-count loss to estimate token-level uncertainty, and NormBT normalizes pairwise losses to correct representation bias.
• These techniques expand task memory, enable intrinsic reward generation, and rebalance gradient influence, promoting improved sample efficiency and refined reward modeling.

“Reward model based counting loss” does not denote a single standardized objective in the recent literature. Across current arXiv usage, the phrase spans at least three technically distinct constructions: counting embedded directly into reward semantics through Counting Reward Automata (CRAs), a learned pseudo-count module trained by an explicit supervised regression loss and converted into count-based intrinsic reward for LLM reasoning, and reward-model loss redesign that addresses pairwise weighting pathologies without introducing a counting objective (Bester et al., 2023, Zhang et al., 18 Oct 2025, Xie et al., 6 Dec 2025). The common thread is that counting, or the allocation of update mass that can be interpreted as a form of counting influence, is coupled to reward-centric learning; the underlying mathematical objects, however, differ substantially.

1. Terminological scope

In the cited works, the relevant meanings of “reward model,” “counting,” and “loss” are not interchangeable. In CRAs, the “reward model” is a finite-state reward automaton extended with counters; counting is internal task memory; and there is no explicit counting-specific supervised loss (Bester et al., 2023). In MERCI, the “reward model” analogue is a learned uncertainty estimator, the Coin Flipping Network (CFN); counting is realized as pseudo-count estimation over token-level hidden states; and the counting mechanism is trained by an explicit mean-squared regression loss (Zhang et al., 18 Oct 2025). In NormBT, the object is a standard RLHF reward model trained from pairwise preferences; the contribution is a normalization of Bradley–Terry loss to offset representation-distance bias; and the paper explicitly does not introduce a counting-based loss (Xie et al., 6 Dec 2025).

Framework	Counting mechanism	Role of loss
CRA	Internal counters in reward automata	No explicit counting loss
MERCI	Pseudo-counts from CFN outputs	Explicit CFN regression loss
NormBT	No literal counting; pair reweighting by representation distance	Modified BT objective

A plausible implication is that the phrase is best treated as an umbrella description rather than a formal term of art. The literature distinguishes reward semantics counting, intrinsic-reward counting, and pairwise reward-model reweighting.

2. Counting as reward semantics in Counting Reward Automata

“Counting Reward Automata: Sample Efficient Reinforcement Learning Through the Exploitation of Reward Function Structure” defines counting as persistent internal memory inside the reward specification itself rather than as exploration counts or external state features (Bester et al., 2023). The paper starts from a $k$-counter counter machine with transition function

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$

configuration

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$

and zero-test map

$Z: \mathbb{N}^k \mapsto \{0,1\}^k.$

Its essential counting mechanism is that the automaton does not branch on exact counter values, only on whether each counter is zero or nonzero, while transitions can increment or decrement counters.

A CRA extends that machine with reward outputs. Given environment states $S$, actions $A$, and propositional event symbols $\mathcal{P}$, a $k$-counter CRA is

$$\langle U, F, \Sigma, \Delta, \delta, \lambda, u_0 \rangle,$$

with $\Sigma = 2^\mathcal{P}$ and

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$0

Operationally, after observing

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$1

the next configuration is

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$2

and the reward is

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$3

The paper is explicit that counting enters the reward model because both the next automaton state and the reward function can depend on the automaton state together with whether counters are zero/nonzero, while the counters themselves are updated through transition-specific integer increments and decrements.

This construction strictly generalizes standard reward machines. A reward machine is

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$4

with

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$5

and

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$6

so transition dynamics depend only on automaton state and input symbol, and reward depends only on automaton state. In a CRA, both transition and reward can additionally depend on counter-status bits derived from internal memory, and counters are updated on transitions. The paper states the theorem that any reward machine can be emulated by a CRA with an equivalent number of machine states and transitions, making reward machines strict special cases of CRAs with unused counters.

The work is explicit that it does not introduce a supervised or differentiable “counting loss” in the usual sense of a reward-model training objective. It proposes a counting-structured reward model, not an explicit counting-specific loss function. The closest analogue to a counting loss is the Bellman-style Q-update performed on automaton-augmented states, especially in the counterfactual learning algorithm. The induced reward process becomes Markov in the Automaton-Augmented MDP (AAMDP), whose state space is

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$7

with

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$8

This gives the central trade-off: counting increases representational power without increasing the number of automaton control states, but introduces multiplicative dependence on bounded counter ranges.

The paper’s examples make the semantics concrete. In LetterEnv, the target language includes context-free forms such as

$$\delta \colon Q \times \{ \Sigma \cup \{ \varepsilon \} \} \times \{0,1\}^k \to Q \times \{+m : m \in \mathbb{Z}\}^k,$$9

and experimentally

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$0

In Office Gridworld, the agent must collect an arbitrary number of mail items, then make exactly that many coffees, then deliver them. The CRA requires only three intuitive non-terminal states—collect mail, make coffee, deliver—while the counters carry the task multiplicity. The paper argues that this enables more compact reward specification and supports counterfactual experience generation, where one real transition can generate many synthetic updates over hypothetical prior automaton configurations.

3. Explicit counting loss in MERCI

“Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards” introduces an explicit counting-derived auxiliary loss, but not in the form of a classical scalar reward model that predicts preference or correctness (Zhang et al., 18 Oct 2025). The paper’s central problem is sparse outcome reward in LLM reasoning. Its solution is MERCI, which adds a count-based intrinsic exploration bonus built from a learned pseudo-count estimator called the Coin Flipping Network.

The CFN is trained on state-label pairs

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$1

where each visit to a state $$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$2 is paired with an independent random Rademacher vector $$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$3. The explicit counting loss is

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$4

This is the most direct instance of a “counting loss” in the cited material. It is not a reward loss; it is a supervised regression loss whose purpose is to make the model output the average of multiple random coin-flip labels attached to repeated visits of the same state.

The derivation yields the pseudo-count relation

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$5

where $$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$6 denotes the visitation count of state $$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$7. In the LLM implementation, the state is instantiated at the token position via the hidden representation of that token in the trajectory. The counted object is therefore best described as token-state pseudo-counts over hidden states. The paper states that each state, i.e. token, within the trajectory is processed by the separate CFN network to extract its hidden representation, and the local reward variance is computed as

$$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$8

MERCI then converts token-level uncertainty into a trajectory-level intrinsic bonus. A key technical point is that the method does not simply add token-wise uncertainty bonuses independently. Motivated by the simplified Uncertainty Bellman Equation, it aggregates local variances along the trajectory and only then takes a square root. Before normalization, three filtering stages are applied: percentile filtering, spatial coherence filtering, and noise-suppression filtering. The normalized exploration bonus is then added to GRPO’s advantage through a clipped augmented advantage. The paper distinguishes two training signals: the GRPO or DAPO policy objective, which updates policy parameters, and the CFN supervised regression loss, which updates the CFN network.

The empirical results isolate the effect of the counting-derived intrinsic reward. On vanilla GRPO for math, the GRPO baseline avg pass@k is $$\langle q, \boldsymbol{c} \rangle \in Q \times \mathbb{N}^k,$$9, and GRPO + MERCI is $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$0. Without percentile and spatial filtering it is $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$1, without noise filtering $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$2, with cumulative standard deviation $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$3, and with token integration $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$4. For avg mean@k, the GRPO baseline is $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$5, GRPO + MERCI is $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$6, without percentile and spatial filtering $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$7, without noise filtering $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$8, with cumulative standard deviation $Z: \mathbb{N}^k \mapsto \{0,1\}^k.$9, and with token integration $S$0. These ablations show that the contribution is not merely the addition of a novelty bonus, but specifically the combination of pseudo-count estimation, uncertainty interpretation, trajectory-level aggregation, within-group normalization, and aggressive filtering.

4. Reward-model loss redesign without counting: BT-loss and NormBT

“When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models” addresses a different question: how pairwise preference losses allocate gradient magnitude in RLHF reward-model training (Xie et al., 6 Dec 2025). The standard Bradley–Terry objective is

$S$1

For a sequence-classifier reward model

$S$2

the per-sample gradient is

$S$3

with $S$4. The paper shows that the gradient norm scales with both prediction error and representation distance. At the score head, the relation

$S$5

is exact.

The diagnosis is representation distance bias. Pairs with small representation distance often receive vanishingly weak updates, even when misranked, while pairs with large distance receive disproportionately strong updates. The paper argues that this is especially harmful for fine-grained judgments such as reasoning and code correctness, where chosen and rejected responses may differ by only a subtle logical or semantic error. Reasoning has the smallest average representation distance and also the smallest average gradient norm under BT in the reported RewardBench category analysis.

NormBT is proposed as an adaptive pair-wise normalization scheme. The ideal inverse-sensitivity weight is

$S$6

but the practical proxy is based on final-layer representation distance,

$S$7

With exponential moving average stabilization, the pair weight is

$S$8

and the final objective is

$S$9

This is a normalization-based pairwise reweighting objective, not a counting-based one. The paper explicitly states that there is no objective that counts errors, counts pair types, counts tokens, or uses count-based balancing in a direct combinatorial sense.

The empirical pattern is consistent across backbones and datasets. Trained on Unified-Feedback $A$0K, gemma-2b-it improves from a BT baseline of $A$1 average RewardBench score to $A$2 with NormBT, while Reasoning rises from $A$3 to $A$4. Under the same dataset, Llama-3.2-3B-Instruct improves from $A$5 to $A$6, and Reasoning rises from $A$7 to $A$8. The paper states that Reasoning improves by more than $A$9 on average. In conceptual relation to “counting,” NormBT can be read as correcting a hidden imbalance in how BT counts pair influence in gradient space, but it is not a literal counting loss.

5. Comparative anatomy of counting and loss

The three works distinguish three different loci at which counting can enter reward-centric learning.

In CRA, counting is part of the reward function structure itself. The counted quantities are task-relevant event occurrences, such as how many $\mathcal{P}$0’s have been observed, how many mail items have been collected, or how many coffees have been made. The counters live inside the reward model, and the RL agent accesses them through the augmented state $\mathcal{P}$1. The learning signal remains the standard RL objective over automaton-augmented states, with counterfactual Q-updates as the operational mechanism (Bester et al., 2023).

In MERCI, counting is not part of the task reward semantics. It is a learned pseudo-count over token-state hidden representations, trained by a standalone regression loss and converted into intrinsic reward. The counting-derived path is

$\mathcal{P}$2

so the explicit loss is auxiliary, while the policy is still updated through the RL objective (Zhang et al., 18 Oct 2025).

In NormBT, the issue is neither event counting nor pseudo-count estimation. The paper studies how the standard reward-model loss implicitly reweights samples by representation distance and introduces a normalization to rebalance pair contributions. This suggests a broader conceptual distinction between counting occurrences, counting uncertainty, and counting influence. Only the second of these corresponds to an explicit counting loss in the narrow optimization sense (Xie et al., 6 Dec 2025).

6. Limitations, misconceptions, and research significance

A common misconception is to equate CRA with count-based exploration. The paper is explicit that its notion of counting is reward semantics counting, not exploration-counting. The sample-efficiency gain comes from count-aware experience reuse, enabled by a reward model that has internal counters. Another misconception is to read MERCI’s CFN as a standard reward model. The paper states that MERCI is not a reward model in the RLHF sense that predicts preference or correctness; it introduces a learned pseudo-count and uncertainty estimator whose output is later converted into intrinsic reward. A third misconception is to classify NormBT as a counting-based objective. The paper explicitly rejects that interpretation.

Each approach also has clear technical caveats. For CRAs, the main cost is AAMDP state-space expansion,

$\mathcal{P}$3

so compact automaton control can coexist with multiplicative dependence on bounded counter ranges (Bester et al., 2023). For MERCI, raw count-based novelty alone is too noisy: without noise filtering, performance degrades substantially and sometimes falls below baseline; token-level integration is significantly worse than trajectory-level integration (Zhang et al., 18 Oct 2025). For NormBT, the theory focuses on gradient magnitude rather than gradient direction, there may be a slight trade-off on larger-distance regions, and if a dataset contains many truly uninformative near-duplicate pairs, upweighting all small-distance pairs could be less desirable (Xie et al., 6 Dec 2025).

The broader significance lies in the fact that the cited works locate counting at different levels of the learning stack. One line of work extends reward expressiveness from regular temporal patterns to tasks with matching, repetition, and unbounded counting. Another uses learned pseudo-counts to make exploration reliable for long-horizon reasoning under sparse outcome rewards. A third shows that pairwise reward-model training can suffer from hidden geometry-dependent imbalance and proposes a normalization-based correction. Taken together, they show that “reward model based counting loss” is not a single method class but a family of reward-centered mechanisms in which counting may appear as internal reward memory, auxiliary pseudo-count supervision, or a diagnostic lens on how optimization allocates pairwise influence.