--- title: Bilevel Natural Language Actor-Critic (Bi-NAC) url: https://www.emergentmind.com/topics/bilevel-natural-language-actor-critic-bi-nac type: topic --- # Bilevel Natural Language Actor-Critic (Bi-NAC) Bilevel Natural Language Actor-Critic (Bi-NAC) is a class of bilevel actor-critic formulations in which natural language enters the optimization loop as either learnable textual feedback, an LLM-derived reference policy, or a natural-language RLHF interface. In the Stackelberg formulation introduced for reasoning tasks, the critic is trained to generate feedback that improves the actor’s downstream verified reward after adaptation [2605.24547]. In recommendation, LAAC instantiates a Bi-NAC by using a prompted LLM as a reference policy inside a bilevel adversarial actor-critic game, with critic-side anchoring and temporal-difference regularization to control overestimation [2507.21274]. Related single-loop bilevel actor-critic methods for RLHF and earlier two-timescale bilevel natural actor-critic analyses provide the optimization framework, first-order surrogates, and convergence perspective that place Bi-NAC within broader bilevel reinforcement learning [2601.16399, 2007.05170]. ## 1. Scope and defining structure Bi-NAC is characterized by a leader-follower or upper-lower coupling between critic-side language generation or guidance and actor-side policy improvement. In the reasoning formulation, the actor $\pi^L_\theta$ first produces an initial attempt $y_0$, the critic $\pi^H_\phi$ generates natural-language feedback $z$ conditioned on $(x,y_0)$, and the actor then refines its answer to $y_1$ under the feedback-conditioned context. The usefulness of $z$ depends on whether the actor can learn from it, while the optimal critic policy depends on the actor’s learning dynamics; this is the basis for the Stackelberg bilevel program [2605.24547]. In recommendation, the same bilevel pattern is implemented differently. A prompted LLM induces a reference policy $\pi_{\mathrm{ref}}(\cdot\mid s)$ over candidate items, and a lightweight actor $\pi_\theta$ is trained to outperform that reference under a critic $f_\phi$ that is selectively optimistic toward LLM-suggested actions but grounded on dataset actions through anchoring and Bellman regularization [2507.21274]. In bilevel RLHF, the upper-level decision variable can parameterize the reward of the lower-level MDP, while the lower level is solved by an entropy-regularized actor-critic; this supplies a more abstract Bi-NAC blueprint for NLP and RLHF [2601.16399]. | Paper | Natural-language component | Bilevel coupling | |---|---|---| | [2605.24547] | Critic-generated feedback $z$ | Critic anticipates actor adaptation | | [2507.21274] | LLM reference policy $\pi_{\mathrm{ref}}$ | Actor competes against LLM guidance | | [2601.16399] | RLHF over language outputs | Upper-level reward learning, lower-level actor-critic | | [2007.05170] | Not language-specific | Two-timescale bilevel actor-critic antecedent | A central feature across these variants is that the critic is not merely evaluative. It actively shapes the actor’s learning signal, either by producing feedback in language or by biasing value estimation toward language-derived actions. ## 2. Stackelberg Bi-NAC with learnable textual feedback In "RL with Learnable Textual Feedback: A Bilevel Approach" [2605.24547], Bi-NAC is formalized for tasks with verifiable rewards such as MATH-500, MBPP, and GPQA. Problems are sampled as $x\sim\mathcal{D}$, the actor generates $y_0\sim\pi^L_\theta(\cdot\mid x)$, the critic emits $z\sim\pi^H_\phi(\cdot\mid x,y_0)$, the actor refines to $y_1\sim\pi^L_\theta(\cdot\mid x,y_0,z)$, and the final outcome is scored by a binary verifier $R(x,y_1)\in\{0,1\}$. The lower-level and upper-level objectives are both the expected verified reward after refinement: $$ L(\phi,\theta)=\mathbb{E}[R(x,y_1)],\qquad U(\phi,\theta)=\mathbb{E}[R(x,y_1)]. $$ The bilevel program is $$ \max_{\phi}\;U\big(\phi,\theta^\star(\phi)\big) \quad\text{s.t.}\quad \theta^\star(\phi)\in\arg\max_\theta L(\phi,\theta). $$ The constrained form is relaxed with a Lagrange multiplier $\lambda\ge 0$, yielding a first-order training objective without implicit differentiation or Hessian inverses. For the actor, the gradient reduces to a REINFORCE update scaled by $(1+\lambda)$: $$ \nabla_{\theta}\mathcal{L} = (1+\lambda)\; \mathbb{E}_{\tau\sim\pi_{\theta,\phi}} \Big[ \big( \nabla_{\theta}\log \pi^L_\theta(y_0\mid x) + \nabla_{\theta}\log \pi^L_\theta(y_1\mid x,y_0,z) \big)\;R(x,y_1) \Big]. $$ The corresponding minimization-form loss is $$ \mathcal{J}_{\text{actor}}(\theta;\phi) = -\,(1+\lambda)\; \mathbb{E}_{\tau\sim\pi_{\theta,\phi}} \Big[ \big( \log \pi^L_\theta(y_0\mid x)+ \log \pi^L_\theta(y_1\mid x,y_0,z) \big)\;R(x,y_1) \Big]. $$ For the critic, the key term is reward-weighted likelihood of the feedback text under the current actor, offset by a second term evaluated under a proxy for the optimal actor: $$ \nabla_{\phi}\mathcal{L} = (1+\lambda)\; \mathbb{E}_{\tau\sim\pi_{\theta,\phi}} \big[ \nabla_{\phi}\log \pi^H_\phi(z\mid x,y_0)\;R(x,y_1) \big] - \lambda\; \mathbb{E}_{\tau^\star\sim\pi_{\theta^\star,\phi}} \big[ \nabla_{\phi}\log \pi^H_\phi(z\mid x,y_0)\;R(x,y_1) \big]. $$ The $\theta^\star$ term is approximated in practice by a recent actor checkpoint $\theta^k$, producing a two-term policy-gradient estimator with a leader-follower flavor. This formulation directly addresses sparse-reward failure modes. On MBPP with LLaMA-3.2-1B, GRPO exhibits advantage collapse in $74\%$ of groups and reward collapse in $83\%$ of responses when all sampled candidates fail, whereas Bi-NAC densifies supervision through feedback that is optimized for its downstream effect rather than for standalone plausibility [2605.24547]. ## 3. LLM-guided recommendation as a Bi-NAC instance LAAC, described as "LLM-guided Adversarial Actor-Critic," instantiates Bi-NAC for offline sequential recommendation [2507.21274]. The environment is modeled as an MDP $$ M=(\mathcal{S},\mathcal{A},P,r,\rho_0,\gamma), $$ where the state is a user’s recent interaction context and, in experiments, is represented as $$ s_t = G(x_{t-5:t}), $$ with $G$ a GRU encoder applied to the last five items. Learning is entirely off-policy from a historical dataset $\mathcal{D}=\{(s,a,r,s')\}$; no online exploration is performed. The natural-language component enters through a prompted LLM reference policy. For each state, the system provides the titles of the last five items and a random candidate set $A_c\subseteq\mathcal{A}$ of size $n_c=100$, asks the LLM for $n_r=10$ recommendations, and defines $\pi_{\mathrm{ref}}(\cdot\mid s)$ to be uniform over the returned set $A_r$: $$ \pi_{\mathrm{ref}}(a\mid s)= \begin{cases} \frac{1}{|A_r|}, & a\in A_r,\\ 0, & a\notin A_r. \end{cases} $$ The paper instantiates $\pi_{\mathrm{ref}}$ with Llama3-8B-Instruct and Claude3-Haiku. The actor-critic game is $$ \max_\theta\;\mathcal{J}_{\mathrm{actor}}(\theta,\phi^\star) \quad\text{s.t.}\quad \phi^\star\in\arg\min_\phi\;\mathcal{L}_{\mathrm{critic}}(\theta,\phi), $$ with $$ \mathcal{J}_{\mathrm{actor}}(\theta,\phi) = \mathbb{E}_{\mathcal{D}} \Big[ f_\phi(s,\pi_\theta)-f_\phi(s,\pi_{\mathrm{ref}}) \Big], $$ where $$ f_\phi(s,\pi)\triangleq \mathbb{E}_{a\sim\pi(\cdot\mid s)}[f_\phi(s,a)]. $$ The actor is parameterized by a softmax over item scores: $$ \pi_\theta(a\mid s)= \frac{\exp(f_\theta(s,a))} {\sum_{a'\in\mathcal{A}}\exp(f_\theta(s,a'))}. $$ The critic’s loss combines three terms: $$ \mathcal{L}_{\mathrm{critic}}(\theta,\phi) = \mathbb{E}_{\mathcal{D}} \Big[ f_\phi(s,\pi_\theta)-f_\phi(s,\pi_{\mathrm{ref}}) \Big] + \alpha\, \mathbb{E}_{\mathcal{D}} \Big[ \big(f_\phi(s,a)-f_\phi(s,\pi_{\mathrm{ref}})\big)^2 \Big] + \beta\, \mathbb{E}_{\mathcal{D}} \Big[ \big(f_\phi(s,a)-(r+\gamma f_\phi(s',\pi_\theta))\big)^2 \Big]. $$ The first term is adversarial alignment, the second is anchoring or grounding, and the third is the TD Bellman residual. Anchoring forces critic estimates for LLM-suggested, low-frequency, or unseen items to remain close to well-estimated in-sample values, thereby mitigating overestimation of unreliable LLM suggestions. LAAC therefore differs from LLM-only recommendation and from end-to-end LLM recommenders. The LLM supplies candidate novel or diverse items through prompting, but the actor learns a dataset-aligned policy that can refine or override those suggestions without any LLM fine-tuning. ## 4. Optimization mechanics and algorithmic patterns A recurring algorithmic theme in Bi-NAC is the use of first-order bilevel surrogates rather than explicit second-order differentiation. In the RLHF-oriented framework of "A Regularized Actor-Critic Algorithm for Bi-Level Reinforcement Learning" [2601.16399], the upper-level variable $x$ parameterizes the reward $r_x$ of a lower-level MDP, and the lower level is optimized by an entropy-regularized actor-critic. The regularized lower-level objective is $$ J_\tau(x,\pi) = \mathbb{E}_{s\sim\rho}[V^{x,\pi}_\tau(s)], $$ with entropy term $E(\pi,s)=-\sum_a \pi(a\mid s)\log\pi(a\mid s)$, while the upper-level objective is $\Phi(x)=f(x,\pi^\ast(x))$. Instead of differentiating through $\pi^\ast(x)$, the method introduces the penalized objective $$ \mathcal{L}_{w,\tau}(x,\pi) = f(x,\pi)+\frac{1}{w}\big(J_\tau(x,\pi_\tau^\ast(x))-J_\tau(x,\pi)\big), $$ and updates the upper level by a sample-based hyper-gradient estimator that uses two trajectories, one under $\pi_k\approx \pi_\tau^\ast(x_k)$ and another under $\pi_k^L\approx \pi_{w_k,\tau_k}^\ast(x_k)$. The method is explicitly single-loop. At each iteration it updates the upper-level variable, two actor parameters, and two critics simultaneously. For the unregularized target, the attenuation schedule is $$ \zeta_k=\frac{\zeta_0}{(k+1)^{9/10}},\quad \alpha_k=\frac{\alpha_0}{(k+1)^{1/2}},\quad \beta_k=\frac{\beta_0}{(k+1)^{1/2}},\quad w_k=\frac{w_0}{(k+1)^{3/20}},\quad \tau_k=\frac{\tau_0}{(k+1)^{1/20}}. $$ This construction is intended to obtain asymptotically unbiased upper-level hyper-gradients while progressively removing lower-level entropy bias. The task-specific Bi-NAC variants adopt the same first-order philosophy. The textual-feedback formulation avoids KL regularization in the core objective and relies instead on the bilevel formulation, learning-rate control, and decoding settings; stability is supported by actor and critic learning rates of $5\mathrm{e}{-6}$ and $1\mathrm{e}{-4}$, decoding temperature $0.7$, and two-turn training with concise, instruction-style feedback [2605.24547]. LAAC likewise uses a two-timescale actor-critic procedure with larger critic learning rate $\eta_{\mathrm{critic}}=0.01$, smaller actor learning rate $\eta_{\mathrm{actor}}=0.001$, defaults $\gamma=0.99$, $\alpha=1.0$, $\beta=1.0$, minibatch size $128$, and $10{,}000$ training steps, while caching $\pi_{\mathrm{ref}}(s)$ offline to avoid runtime LLM overhead [2507.21274]. These methods share an implementation pattern: the critic or critic-side object evolves on the faster or more expressive channel, while the actor learns to exploit critic-provided structure without requiring nested solves, importance sampling, or expensive LLM fine-tuning. ## 5. Empirical findings On reasoning benchmarks, Bi-NAC is reported to improve both parameter efficiency and sample efficiency over RL baselines with sparse terminal rewards [2605.24547]. | Setting | Bi-NAC | Baseline | |---|---:|---:| | MATH-500 (2B vs 3B) | 46.56% | 41.43% | | MBPP (2B vs 3B) | 66.73% | 61.55% | | GPQA (6B vs 7B) | 49.30% | 43.60% | The same study reports that a 6B Bi-NAC model achieves $51.40\%$ versus $48.40\%$ on MATH-500 and $75.00\%$ versus $72.20\%$ on MBPP against a 7B GRPO baseline. Across 1B, 3B, and 8B scales, Bi-NAC reaches $46.6$, $51.4$, and $60.2$ on MATH, $66.4$, $75.0$, and $79.8$ on MBPP, and $40.6$, $49.3$, and $56.3$ on GPQA. It also avoids the long zero-advantage plateau seen in GRPO: on MBPP with LLaMA-3.2-1B, GRPO improves minimally for about $600$ steps, whereas Bi-NAC reaches high accuracy within about $400$ steps. Learned feedback achieves up to $2.1\times$ higher Feedback Optimality after $1600$ steps than fixed or auxiliary feedback. On MATH-500 with Qwen3-1.7B, multi-turn inference improves from Turn 1 $62.4\%$ to Turn 2 $70.2\%$ and Turn 3 $71.6\%$, after which it saturates. Cross-family pairings between Qwen and LLaMA at about 1B scale show minimal degradation, and on AIME 2024 with Qwen3-1.7B, a single critic-guided refinement increases accuracy from $6.7\%$ to $13.3\%$. In recommendation, LAAC is evaluated on MovieLens-1M and is reported to improve accuracy and diversity or novelty jointly over classical baselines [2507.21274]. | Metric | LAAC (Llama3) | GRU4Rec | |---|---:|---:| | HR@10 | 0.0720 | 0.0644 | | NDCG@10 | 0.0387 | 0.0339 | | R@10 | 1109 | 994 | | CV@20 | 0.7674 | 0.7350 | | NCV@20 | 0.5464 | 0.4782 | | NC@1 | 268 | 219 | LAAC with Claude3 performs similarly, with HR@10 $0.0720$, NDCG@10 $0.0389$, R@10 $1109$, CV@20 $0.7646$, NCV@20 $0.5410$, and NC@1 $278$. By contrast, LLM-only policies achieve high diversity but poor accuracy; for example, $\pi_{\mathrm{Llama3}}$ has HR@10 $=0.0064$. Under imbalanced male-only training data, LAAC tested on the original distribution reaches NDCG@20 $0.0416$ (best), CV@20 $0.7444$ (best), and NC@1 $264$ (best), while GRU4Rec’s CV@20 is $0.6913$ and SMORL’s diversity declines markedly. The ablation pattern is explicit: increasing anchoring $\alpha$ raises R@10 from $1066$ to $1358$ while reducing NCV@10 from $0.408$ to $0.309$, and removing TD regularization by setting $\beta=0$ harms performance on poor-quality data. These results collectively indicate that Bi-NAC can use language-derived guidance to recover reward signal, improve diversity or novelty, or reduce sample requirements, depending on the domain-specific critic design. ## 6. Relation to natural actor-critic, guarantees, and limitations The term Bi-NAC can be confused with classical Natural Actor-Critic. In the classical formulation, the policy update uses a Fisher-preconditioned natural gradient, $$ \nabla_\theta^{\mathrm{nat}} J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta), $$ with $$ F(\theta)= \mathbb{E}_{s\sim d} \Big[ \mathbb{E}_{a\sim\pi_\theta} [ \nabla_\theta\log\pi_\theta(a\mid s)\nabla_\theta\log\pi_\theta(a\mid s)^\top ] \Big]. $$ LAAC explicitly states that it does not compute a Fisher-based natural gradient; there, “Natural Language” refers to the LLM-guided reference policy rather than to natural-gradient optimization [2507.21274]. Earlier two-timescale work, however, shows that a two-timescale natural actor-critic proximal policy optimization algorithm is a special case of a general bilevel stochastic approximation framework, with fast critic and slow actor updates and an $O(K^{-1/4})$ rate for the gap in expected discounted reward relative to a global optimal policy [2007.05170]. The strongest explicit convergence guarantees among the cited Bi-NAC-related methods come from the penalty-based single-loop framework of [2601.16399]. Under its assumptions, the algorithm achieves $$ \min_{t