Contrastive Policy Learning (ConPoLe)

• The paper introduces contrastive policy learning that bypasses value estimation by employing an InfoNCE loss to align state-successor pairs, enhancing policy selection in sparse, symbolic tasks.
• Contrastive Policy Learning is a paradigm that reframes reinforcement learning by maximizing mutual information between states and their correct successors, leading to robust and semantically rich representations.
• Empirical evaluations show that ConPoLe significantly outperforms traditional methods in text-based mathematical, logical, and combinatorial domains by effectively filtering noise from transient errors.

Contrastive Policy Learning (ConPoLe) is a reinforcement learning (RL) paradigm that recasts RL policy optimization as a contrastive representation learning problem, with the objective of maximizing mutual information between states and their correct successors in environments characterized by sparse, binary rewards. The core insight of ConPoLe and its extensions is to sidestep value estimation by applying a contrastive InfoNCE loss to policy selection, yielding policies and representations that excel in symbolic reasoning tasks—including text-based mathematics, logic, and structured combinatorial domains—where traditional RL methods struggle. ConPoLe and its subsequent generalizations, such as group-based contrastive policy optimization for LLMs, have demonstrated significant empirical gains in reasoning robustness and generalization, particularly under sparse supervision (Poesia et al., 2021, Cui et al., 10 Mar 2026).

1. Problem Formulation and Conceptual Foundations

In the canonical ConPoLe framework (Poesia et al., 2021), an environment comprises states and actions described as unstructured text strings, a deterministic transition function $T(s,a) = s'$, and a reward function $R(s)$ that is $1$ if $s$ is a terminal “solved” state and $0$ otherwise. The learning objective is to discover a stochastic policy $\pi_\theta(a|s)$ maximizing the probability of eventually reaching a solved state from an initial state sampled from a distribution $P_I(s)$. The high sparsity of reward signals in such symbolic domains impedes classical value-based and Monte Carlo approaches.

Contrastive Policy Learning reframes the RL goal: instead of learning state or action values, it optimizes the expected InfoNCE loss over positive and negative successor states generated during search, thereby maximizing a lower bound on the mutual information between the current state and those correct successors that continue on a solution trajectory.

2. InfoNCE Objective and Contrastive Loss Formulation

Central to ConPoLe is the InfoNCE contrastive loss, defined over representations of states and their candidate successors. For each encountered trajectory $$s_0 \rightarrow s_1 \rightarrow \dots \rightarrow s_T$$ ending in a solved state, and at each step $t$, the true next state $s^+ = s_{t+1}$ serves as the positive example, while other beam candidates $R(s)$0 at depth $R(s)$1 form the negatives. State embeddings $R(s)$2 are computed via a character-level, bidirectional LSTM encoder. The compatibility between $R(s)$3 and $R(s)$4 is given by $R(s)$5.

The InfoNCE loss per step is:

$R(s)$6

Minimizing $R(s)$7 maximizes a lower bound on the mutual information $R(s)$8, encouraging the representation to capture features predictive of successful problem progressions (Poesia et al., 2021). In group-based RL for LLMs, the framework is generalized to cluster embeddings of multiple complete trajectories sharing the same prompt and correct outcome, further tightening mutual information bounds (Cui et al., 10 Mar 2026).

3. Sampling, Optimization, and Algorithmic Workflow

ConPoLe employs a beam search procedure to efficiently generate solution trajectories and collect both positives (the actual solution successors) and negatives (competing candidate states/actions explored but not leading to the solution). Beam elements are prioritized according to cumulative log-probability under $R(s)$9. As learning advances, the search depth is incrementally increased (iterative deepening), creating an implicit curriculum over solution length and complexity.

The learning algorithm proceeds by harvesting (state, correct successor, negatives) tuples into a replay buffer during search episodes. Periodically, contrastive updates are performed using batches from this buffer:

• Sample batch $1$0.
• Compute InfoNCE loss and derive gradient updates for policy and encoder parameters.
• Optionally, increase depth limit after accumulating $1$1 new solved problems.

For LLM policies, CLIPO (Cui et al., 10 Mar 2026) adapts this approach by sampling $1$2 trajectory rollouts per prompt $1$3, embedding each via a learned contrastive head, and applying a group-level InfoNCE loss across correct trajectories. The auxiliary reward derived from the contrastive loss is added to the original RLVR reward, and policy optimization proceeds with normalized advantages.

4. Policy Architecture and Representation Learning

In symbolic domains, ConPoLe represents each state and successor as a linearized character string, embedding these via a stack of bidirectional LSTM layers (512-dim vector output). The learned compatibility matrix $1$4 aligns state embeddings such that correct successors are maximally similar—under the contrastive loss—with their predecessor states, and negative candidates are repelled. This implicit structure learning leads to representations that are semantically meaningful and aligned with solution pathways.

For LLM-driven policies, group contrastive heads (single linear layers on top of mean-pooled last hidden states) map each trajectory to a vector, enabling computation of cross-trajectory similarities and contrastive rewards. This mechanism clusters embeddings of correct completions and repels incorrect or hallucinated ones, denoising the reward signal in trajectory space.

5. Theoretical Insights and Mutual Information

Contrastive Policy Learning is grounded in the information-theoretic guarantee that minimizing InfoNCE yields a tight lower bound on mutual information between current and successor states (or between correct trajectories for LLMs). At the optimum, the learned policy approaches the Bayes-optimal successor selection, as the density over correct moves along a solution is matched. The quality and number of negative samples affect the tightness and variance of the mutual information estimation; larger beam sizes and richer negative sets enhance both the bound and sample efficiency (Poesia et al., 2021, Cui et al., 10 Mar 2026).

In the group setting, maximizing mutual information between representations of successful trajectories promotes invariance to incidental reasoning steps and focuses training on structural features genuinely predictive of correct problem solving.

6. Empirical Evaluation and Representational Outcomes

ConPoLe’s effectiveness has been established across a suite of symbolic reasoning environments inspired by the Mathematics Common Core Curriculum, including sorting, addition, multiplication, fractions, and equations. In comparison to behavioral cloning (BC), deep RL for text (DRRN), Autodidactic Iteration (ADI), Deep Approximate Value Iteration (DAVI), the hand-coded MathSteps solver, and the closely related Contrastive Value Iteration (CVI), ConPoLe achieves state-of-the-art or near-perfect solution rates (up to 100% in several domains) with substantial margins over baselines (Poesia et al., 2021). On challenging combinatorial domains (Rubik’s Cube), ConPoLe solves all test scrambles of length 1000 and requires fewer node expansions than DeepCubeA.

The benefit of maximal and diverse negatives in contrastive sampling is corroborated by ablations: full ConPoLe (all beam negatives) substantially outperforms its local-negative variant, particularly in domains with complex solution structure (e.g., fractions, equations). Embeddings derived from ConPoLe cluster problems by curricular category (as shown by PCA and 1-NN evaluations), reflecting high semantic content.

In LLM domains, CLIPO consistently improves Pass@1 accuracy on mathematical and QA benchmarks over strong RLVR baselines. CLIPO’s group contrastive regularization especially enhances generalization on perturbed and out-of-distribution problems (Cui et al., 10 Mar 2026). t-SNE visualization confirms that CLIPO training clusters successful trajectories tightly in representation space, while incorrect or hallucinatory paths are separated.

Method	Sorting	Add	Mult	Frac	Eqns
BC	92.0%	64.5%	10.0%	52.5%	4.5%
DRRN	29.5%	40.0%	10.5%	20.0%	2.5%
DAVI	99.5%	54.0%	18.5%	57.5%	8.0%
CVI	77.0%	72.0%	100.0%	84.5%	89.0%
ConPoLe-loc	100.0%	98.5%	99.5%	86.0%	76.5%
ConPoLe	100.0%	100.0%	99.5%	96.0%	92.5%

Representation	1-NN Accuracy (Khan Acad. classes)
edit-distance	71.4%
BC embedding	61.9%
DRRN embedding	64.2%
CVI embedding	64.2%
ConPoLe embedding	90.5%

7. Regularization Effects and Robustness to Reasoning Errors

Contrastive regularization mitigates the central weakness of outcome-only RL supervision: spurious trajectories that arrive at the correct answer through incorrect or hallucinated reasoning. By clustering correct reasoning paths and repelling inconsistent or off-manifold ones in representation space, ConPoLe and CLIPO filter out transient token-level mistakes and prevent overfitting to accidental solution chains. This cross-trajectory denoising endows policies with increased resistance to both step-level and distributional errors and is particularly effective in environments with complex semantic structure or ambiguous intermediate solutions (Poesia et al., 2021, Cui et al., 10 Mar 2026).

8. Extensions to LLMs and General Symbolic Policy Optimization

Recent developments extend contrastive policy learning to the regime of large auto-regressive models under the RLVR framework. CLIPO (Cui et al., 10 Mar 2026) introduces group-level contrastive heads into policy optimization, leveraging similarities among successful LLM rollouts to form auxiliary rewards that complement sparse, binary final-answer checks. This methodology is directly compatible with prevalent group-based RLVR policy optimizers (GRPO, GSPO, DAPO) and scales to competition-level mathematical domains.

Empirically, CLIPO yields consistent generalization and robustness gains across diverse models and benchmarks, underscoring the modularity and effectiveness of contrastive objectives as a foundation for policy learning in symbolic and text-based domains.

---

Contrastive Policy Learning coherently unifies RL with contrastive representation learning, providing a principled and empirically validated approach for difficult, sparse-reward symbolic sequences. By maximizing mutual information between solution-supporting states (or trajectories), ConPoLe and its successors enhance both search efficacy and semantic understanding, demonstrating versatility across algorithmic domains and large-scale language modeling (Poesia et al., 2021, Cui et al., 10 Mar 2026).