Bilevel Natural Language Actor-Critic (Bi-NAC)

• The paper introduces Bi-NAC as a bilevel actor-critic framework where the critic generates natural language feedback to refine the actor’s policy, improving verified rewards.
• In reasoning tasks, Bi-NAC employs a Stackelberg formulation where the actor adapts its output based on critic-provided language feedback to mitigate sparse reward challenges.
• For recommendation, Bi-NAC instantiates an LLM-guided reference policy with anchoring and temporal-difference regularization to balance accuracy and novelty.

Bilevel Natural Language Actor-Critic (Bi-NAC) is a class of bilevel actor-critic formulations in which natural language enters the optimization loop as either learnable textual feedback, an LLM-derived reference policy, or a natural-language RLHF interface. In the Stackelberg formulation introduced for reasoning tasks, the critic is trained to generate feedback that improves the actor’s downstream verified reward after adaptation (Singh et al., 23 May 2026). In recommendation, LAAC instantiates a Bi-NAC by using a prompted LLM as a reference policy inside a bilevel adversarial actor-critic game, with critic-side anchoring and temporal-difference regularization to control overestimation (Woo et al., 28 Jul 2025). Related single-loop bilevel actor-critic methods for RLHF and earlier two-timescale bilevel natural actor-critic analyses provide the optimization framework, first-order surrogates, and convergence perspective that place Bi-NAC within broader bilevel reinforcement learning (Zeng et al., 23 Jan 2026, Hong et al., 2020).

1. Scope and defining structure

Bi-NAC is characterized by a leader-follower or upper-lower coupling between critic-side language generation or guidance and actor-side policy improvement. In the reasoning formulation, the actor $\pi^L_\theta$ first produces an initial attempt $y_0$, the critic $\pi^H_\phi$ generates natural-language feedback $z$ conditioned on $(x,y_0)$, and the actor then refines its answer to $y_1$ under the feedback-conditioned context. The usefulness of $z$ depends on whether the actor can learn from it, while the optimal critic policy depends on the actor’s learning dynamics; this is the basis for the Stackelberg bilevel program (Singh et al., 23 May 2026).

In recommendation, the same bilevel pattern is implemented differently. A prompted LLM induces a reference policy $\pi_{\mathrm{ref}}(\cdot\mid s)$ over candidate items, and a lightweight actor $\pi_\theta$ is trained to outperform that reference under a critic $f_\phi$ that is selectively optimistic toward LLM-suggested actions but grounded on dataset actions through anchoring and Bellman regularization (Woo et al., 28 Jul 2025). In bilevel RLHF, the upper-level decision variable can parameterize the reward of the lower-level MDP, while the lower level is solved by an entropy-regularized actor-critic; this supplies a more abstract Bi-NAC blueprint for NLP and RLHF (Zeng et al., 23 Jan 2026).

Paper	Natural-language component	Bilevel coupling
(Singh et al., 23 May 2026)	Critic-generated feedback $y_0$0	Critic anticipates actor adaptation
(Woo et al., 28 Jul 2025)	LLM reference policy $y_0$1	Actor competes against LLM guidance
(Zeng et al., 23 Jan 2026)	RLHF over language outputs	Upper-level reward learning, lower-level actor-critic
(Hong et al., 2020)	Not language-specific	Two-timescale bilevel actor-critic antecedent

A central feature across these variants is that the critic is not merely evaluative. It actively shapes the actor’s learning signal, either by producing feedback in language or by biasing value estimation toward language-derived actions.

2. Stackelberg Bi-NAC with learnable textual feedback

In "RL with Learnable Textual Feedback: A Bilevel Approach" (Singh et al., 23 May 2026), Bi-NAC is formalized for tasks with verifiable rewards such as MATH-500, MBPP, and GPQA. Problems are sampled as $y_0$2, the actor generates $y_0$3, the critic emits $y_0$4, the actor refines to $y_0$5, and the final outcome is scored by a binary verifier $y_0$6.

The lower-level and upper-level objectives are both the expected verified reward after refinement:

$y_0$7

The bilevel program is

$y_0$8

The constrained form is relaxed with a Lagrange multiplier $y_0$9, yielding a first-order training objective without implicit differentiation or Hessian inverses.

For the actor, the gradient reduces to a REINFORCE update scaled by $\pi^H_\phi$0:

$\pi^H_\phi$1

The corresponding minimization-form loss is

$\pi^H_\phi$2

For the critic, the key term is reward-weighted likelihood of the feedback text under the current actor, offset by a second term evaluated under a proxy for the optimal actor:

$\pi^H_\phi$3

The $\pi^H_\phi$4 term is approximated in practice by a recent actor checkpoint $\pi^H_\phi$5, producing a two-term policy-gradient estimator with a leader-follower flavor.

This formulation directly addresses sparse-reward failure modes. On MBPP with LLaMA-3.2-1B, GRPO exhibits advantage collapse in $\pi^H_\phi$6 of groups and reward collapse in $\pi^H_\phi$7 of responses when all sampled candidates fail, whereas Bi-NAC densifies supervision through feedback that is optimized for its downstream effect rather than for standalone plausibility (Singh et al., 23 May 2026).

3. LLM-guided recommendation as a Bi-NAC instance

LAAC, described as "LLM-guided Adversarial Actor-Critic," instantiates Bi-NAC for offline sequential recommendation (Woo et al., 28 Jul 2025). The environment is modeled as an MDP

$\pi^H_\phi$8

where the state is a user’s recent interaction context and, in experiments, is represented as

$\pi^H_\phi$9

with $z$0 a GRU encoder applied to the last five items. Learning is entirely off-policy from a historical dataset $z$1; no online exploration is performed.

The natural-language component enters through a prompted LLM reference policy. For each state, the system provides the titles of the last five items and a random candidate set $z$2 of size $z$3, asks the LLM for $z$4 recommendations, and defines $z$5 to be uniform over the returned set $z$6:

$z$7

The paper instantiates $z$8 with Llama3-8B-Instruct and Claude3-Haiku.

The actor-critic game is

$z$9

with

$(x,y_0)$0

where

$(x,y_0)$1

The actor is parameterized by a softmax over item scores:

$(x,y_0)$2

The critic’s loss combines three terms:

$(x,y_0)$3

The first term is adversarial alignment, the second is anchoring or grounding, and the third is the TD Bellman residual. Anchoring forces critic estimates for LLM-suggested, low-frequency, or unseen items to remain close to well-estimated in-sample values, thereby mitigating overestimation of unreliable LLM suggestions.

LAAC therefore differs from LLM-only recommendation and from end-to-end LLM recommenders. The LLM supplies candidate novel or diverse items through prompting, but the actor learns a dataset-aligned policy that can refine or override those suggestions without any LLM fine-tuning.

4. Optimization mechanics and algorithmic patterns

A recurring algorithmic theme in Bi-NAC is the use of first-order bilevel surrogates rather than explicit second-order differentiation. In the RLHF-oriented framework of "A Regularized Actor-Critic Algorithm for Bi-Level Reinforcement Learning" (Zeng et al., 23 Jan 2026), the upper-level variable $(x,y_0)$4 parameterizes the reward $(x,y_0)$5 of a lower-level MDP, and the lower level is optimized by an entropy-regularized actor-critic. The regularized lower-level objective is

$(x,y_0)$6

with entropy term $(x,y_0)$7, while the upper-level objective is $(x,y_0)$8. Instead of differentiating through $(x,y_0)$9, the method introduces the penalized objective

$y_1$0

and updates the upper level by a sample-based hyper-gradient estimator that uses two trajectories, one under $y_1$1 and another under $y_1$2.

The method is explicitly single-loop. At each iteration it updates the upper-level variable, two actor parameters, and two critics simultaneously. For the unregularized target, the attenuation schedule is

$y_1$3

This construction is intended to obtain asymptotically unbiased upper-level hyper-gradients while progressively removing lower-level entropy bias.

The task-specific Bi-NAC variants adopt the same first-order philosophy. The textual-feedback formulation avoids KL regularization in the core objective and relies instead on the bilevel formulation, learning-rate control, and decoding settings; stability is supported by actor and critic learning rates of $y_1$4 and $y_1$5, decoding temperature $y_1$6, and two-turn training with concise, instruction-style feedback (Singh et al., 23 May 2026). LAAC likewise uses a two-timescale actor-critic procedure with larger critic learning rate $y_1$7, smaller actor learning rate $y_1$8, defaults $y_1$9, $z$0, $z$1, minibatch size $z$2, and $z$3 training steps, while caching $z$4 offline to avoid runtime LLM overhead (Woo et al., 28 Jul 2025).

These methods share an implementation pattern: the critic or critic-side object evolves on the faster or more expressive channel, while the actor learns to exploit critic-provided structure without requiring nested solves, importance sampling, or expensive LLM fine-tuning.

5. Empirical findings

On reasoning benchmarks, Bi-NAC is reported to improve both parameter efficiency and sample efficiency over RL baselines with sparse terminal rewards (Singh et al., 23 May 2026).

Setting	Bi-NAC	Baseline
MATH-500 (2B vs 3B)	46.56%	41.43%
MBPP (2B vs 3B)	66.73%	61.55%
GPQA (6B vs 7B)	49.30%	43.60%

The same study reports that a 6B Bi-NAC model achieves $z$5 versus $z$6 on MATH-500 and $z$7 versus $z$8 on MBPP against a 7B GRPO baseline. Across 1B, 3B, and 8B scales, Bi-NAC reaches $z$9, $\pi_{\mathrm{ref}}(\cdot\mid s)$0, and $\pi_{\mathrm{ref}}(\cdot\mid s)$1 on MATH, $\pi_{\mathrm{ref}}(\cdot\mid s)$2, $\pi_{\mathrm{ref}}(\cdot\mid s)$3, and $\pi_{\mathrm{ref}}(\cdot\mid s)$4 on MBPP, and $\pi_{\mathrm{ref}}(\cdot\mid s)$5, $\pi_{\mathrm{ref}}(\cdot\mid s)$6, and $\pi_{\mathrm{ref}}(\cdot\mid s)$7 on GPQA. It also avoids the long zero-advantage plateau seen in GRPO: on MBPP with LLaMA-3.2-1B, GRPO improves minimally for about $\pi_{\mathrm{ref}}(\cdot\mid s)$8 steps, whereas Bi-NAC reaches high accuracy within about $\pi_{\mathrm{ref}}(\cdot\mid s)$9 steps. Learned feedback achieves up to $\pi_\theta$0 higher Feedback Optimality after $\pi_\theta$1 steps than fixed or auxiliary feedback. On MATH-500 with Qwen3-1.7B, multi-turn inference improves from Turn 1 $\pi_\theta$2 to Turn 2 $\pi_\theta$3 and Turn 3 $\pi_\theta$4, after which it saturates. Cross-family pairings between Qwen and LLaMA at about 1B scale show minimal degradation, and on AIME 2024 with Qwen3-1.7B, a single critic-guided refinement increases accuracy from $\pi_\theta$5 to $\pi_\theta$6.

In recommendation, LAAC is evaluated on MovieLens-1M and is reported to improve accuracy and diversity or novelty jointly over classical baselines (Woo et al., 28 Jul 2025).

Metric	LAAC (Llama3)	GRU4Rec
HR@10	0.0720	0.0644
NDCG@10	0.0387	0.0339
R@10	1109	994
CV@20	0.7674	0.7350
NCV@20	0.5464	0.4782
NC@1	268	219

LAAC with Claude3 performs similarly, with HR@10 $\pi_\theta$7, NDCG@10 $\pi_\theta$8, R@10 $\pi_\theta$9, CV@20 $f_\phi$0, NCV@20 $f_\phi$1, and NC@1 $f_\phi$2. By contrast, LLM-only policies achieve high diversity but poor accuracy; for example, $f_\phi$3 has HR@10 $f_\phi$4. Under imbalanced male-only training data, LAAC tested on the original distribution reaches NDCG@20 $f_\phi$5 (best), CV@20 $f_\phi$6 (best), and NC@1 $f_\phi$7 (best), while GRU4Rec’s CV@20 is $f_\phi$8 and SMORL’s diversity declines markedly. The ablation pattern is explicit: increasing anchoring $f_\phi$9 raises R@10 from $y_0$00 to $y_0$01 while reducing NCV@10 from $y_0$02 to $y_0$03, and removing TD regularization by setting $y_0$04 harms performance on poor-quality data.

These results collectively indicate that Bi-NAC can use language-derived guidance to recover reward signal, improve diversity or novelty, or reduce sample requirements, depending on the domain-specific critic design.

6. Relation to natural actor-critic, guarantees, and limitations

The term Bi-NAC can be confused with classical Natural Actor-Critic. In the classical formulation, the policy update uses a Fisher-preconditioned natural gradient,

$y_0$05

with

$y_0$06

LAAC explicitly states that it does not compute a Fisher-based natural gradient; there, “Natural Language” refers to the LLM-guided reference policy rather than to natural-gradient optimization (Woo et al., 28 Jul 2025). Earlier two-timescale work, however, shows that a two-timescale natural actor-critic proximal policy optimization algorithm is a special case of a general bilevel stochastic approximation framework, with fast critic and slow actor updates and an $y_0$07 rate for the gap in expected discounted reward relative to a global optimal policy (Hong et al., 2020).

The strongest explicit convergence guarantees among the cited Bi-NAC-related methods come from the penalty-based single-loop framework of (Zeng et al., 23 Jan 2026). Under its assumptions, the algorithm achieves

$y_0$08

for the original unregularized bilevel objective and

$y_0$09

for fixed $y_0$10. The corresponding sample complexities are $y_0$11 and $y_0$12. By contrast, the textual-feedback Bi-NAC provides policy-gradient derivations and empirical stability but does not claim formal convergence guarantees (Singh et al., 23 May 2026).

The limitations are domain-specific but structurally related. Bi-NAC with textual feedback relies on verifiable rewards, is sensitive to feedback that is too long or generic, incurs overhead from two-turn decoding and dual-model training, and approximates $y_0$13 by a recent checkpoint rather than by an exact lower-level solve (Singh et al., 23 May 2026). LAAC can be hurt by misaligned LLM guidance under domain shift or niche cold-start interests, requires efficient candidate generation or caching for very large catalogs, and may fail to capture temporal drift when prompts are static (Woo et al., 28 Jul 2025). The broader bilevel actor-critic theory relies on assumptions such as invertible lower-level Hessians, regularization-dependent Polyak-Lojasiewicz structure, sufficient exploration, exact linear value approximation, or concentrability-type conditions, which may be restrictive for large LLM systems and complex environments (Zeng et al., 23 Jan 2026, Hong et al., 2020).

Bi-NAC therefore occupies a precise position in contemporary RL and LLM research: it is a bilevel actor-critic paradigm in which natural language is not auxiliary decoration but part of the optimization object itself, whether as feedback to be learned, guidance to be refined, or reward-facing structure to be integrated into a single-loop bilevel RL procedure.