Linguistic Unit Policy Optimization

• Linguistic unit policy optimization is a design principle that targets explicit language structures (tokens, reasoning trajectories, full documents) to address task-specific supervision bottlenecks.
• Methodologies vary from token-level updates to mitigate localized errors to trajectory-level pairwise comparisons and document-level rewriting for privacy-utility trade-offs.
• Empirical studies confirm that structured linguistic optimization enhances multilingual performance, reduces errors, and improves control over generation outcomes.

Searching arXiv for the specified papers and related terminology to ground the article in current preprints. Linguistic unit policy optimization can be understood, in recent arXiv work, as policy optimization in which the optimized object is an explicitly linguistic unit rather than an undifferentiated response-level signal. Depending on the setting, that unit may be a rollout language, a token at a confusion point, a full reasoning trajectory, a linguistically expressed experience repository, or an entire rewritten document. Across these formulations, the common move is to expose structure that sequence-level optimization would otherwise collapse, and to use that structure for more targeted credit assignment, routing, or control (Guo et al., 25 May 2026, Choo et al., 29 Apr 2026, Wen et al., 2024, Yuan et al., 19 May 2026, Leng et al., 1 Jul 2026, Loiseau et al., 2024).

1. Scope of the concept

The literature does not present a single canonical algorithm under the label. Instead, papers instantiate the idea at different granularities, each tied to a different failure mode or supervision bottleneck. In multilingual alignment, the unit is the response language; in language-confusion mitigation and language-agent RL, it is the token; in reasoning alignment, it is the generated trajectory; in video anomaly reasoning, it is an editable linguistic prior; and in authorship obfuscation, it is the rewritten document (Guo et al., 25 May 2026, Choo et al., 29 Apr 2026, Wen et al., 2024, Yuan et al., 19 May 2026, Leng et al., 1 Jul 2026, Loiseau et al., 2024).

Paper	Optimized linguistic unit	Core mechanism
"Learning to Route Languages for Multilingual Policy Optimization" (Guo et al., 25 May 2026)	Language	Contextual multi-armed bandit over rollout languages
"TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in LLMs" (Choo et al., 29 Apr 2026)	Token at confusion point	Top-$N$ candidate exploration with token-level PPO-style updates
"Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement" (Wen et al., 2024)	Token within textual action	Per-token soft Bellman update and per-token policy improvement
"LambdaPO: A Lambda Style Policy Optimization for Reasoning LLMs" (Yuan et al., 19 May 2026)	Reasoning trajectory	Pairwise decomposed advantage over rollout cohorts
"Linguistic Relative Policy Optimization for Video Anomaly Reasoning" (Leng et al., 1 Jul 2026)	Linguistic experience repository	Group-wise reflection and context injection without parameter updates
"TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods" (Loiseau et al., 2024)	Entire rewritten text	PPO or DPO for privacy-utility trade-off

This range is itself significant. It shows that the phrase is less a single optimizer than a design principle: make the relevant linguistic structure explicit, then optimize at that level. This suggests that “linguistic unit” is best treated as a variable abstraction layer whose exact meaning depends on the task.

2. Language as an optimization variable

"Learning to Route Languages for Multilingual Policy Optimization" introduces Language-Routed Policy Optimization (LRPO), an online policy optimization framework that treats language as a selectable variable rather than a fixed attribute of the input example (Guo et al., 25 May 2026). The motivation is that multilingual LLMs encode complementary knowledge across languages, while prior RLHF/GRPO-style methods often keep each question confined to one language or rely on a fixed dominant-language anchor, typically English. LRPO instead asks which rollout language is most informative for a given question.

The router is formulated as a contextual multi-armed bandit over languages. For a training question $x$ with input language $\ell_x$, topic $t(x)$, and optional region $g(x)$, the routing distribution is

$$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$

The matrices $\mathbf{A}$ and $\mathbf{B}$ encode topic-language and region-language preferences, and the router uses $\epsilon$-greedy exploration with temperature annealing. The update rule is an exponential moving average of observed rewards,

$$A_{t(x),\ell} \leftarrow (1-\alpha)\,A_{t(x),\ell} + \alpha\,\bar{r}_{t,g,\ell}, \qquad B_{g(x),\ell} \leftarrow (1-\alpha)\,B_{g(x),\ell} + \alpha\,\bar{r}_{t,g,\ell},$$

where $x$0 is the empirical mean reward for language $x$1 under topic $x$2 and region $x$3. The paper explicitly interprets this as an online estimator of expected utility per linguistic channel.

The rollout group is multilingual. LRPO constructs a group of size $x$4, reserves an on-policy quota $x$5 for the input language, and samples the remaining rollout languages from the router distribution. Each rollout is generated as $x$6. The policy update is not a new optimizer in isolation; it is a multilingual rollout-routing framework wrapped around GRPO. Reward is built from cross-lingual semantic similarity to a reference response, but the paper stresses that raw similarity is not comparable across language pairs. It therefore introduces mean-based and quantile-based calibration, then gates reward by language consistency with

$x$7

and uses the final reward $x$8.

The empirical results support the claim that adaptive language routing is useful. Across Qwen2.5-1.5b-it, LLaMA3.2-1b-it, and Gemma3-4b-it, LRPO improves multilingual performance, especially on open-ended generation tasks such as CARE, CARE-pro, and mGSM-v2. On Qwen2.5-1.5b-it, it raises the average score on mGSM-v2 from 24.87 to 38.25 and improves the overall score to 32.15 versus 30.42 for GRPO. It also improves seen-language averages across five benchmarks by +5.08 over the initial instruction-tuned model and +2.85 over GRPO. The fixed-router ablation shows that input-dominant and English-dominant mixtures help, but the learned router performs best overall, particularly on region-sensitive tasks. The paper’s case study is especially pointed: a Japanese question about Ferragamo’s flagship store is answered incorrectly in Japanese and English, but correctly in French. That example directly challenges the assumption that the source language or English is always the best supervision anchor.

3. Token-level optimization

A second line of work argues that sequence-level optimization is too coarse when the failure mode is localized. "TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in LLMs" states that language confusion in multilingual LLMs is often caused by localized token-level errors rather than globally defective responses (Choo et al., 29 Apr 2026). Sequence-level methods such as DPO, ORPO, and GRPO operate on whole responses and can suppress valid surrounding content along with the error. TLPO therefore tries to identify the exact point where confusion begins.

For a prompt $x$9, the model generates a response $\ell_x$0. If language confusion occurs, TLPO detects the confusion point $\ell_x$1, defined as the first token decoded in a language other than the target language. At that position it performs probability-ranked exploration over the top-$\ell_x$2 next-token candidates,

$\ell_x$3

where $\ell_x$4 returns the top-$\ell_x$5 tokens under $\ell_x$6. Each candidate is evaluated via a short lookahead rollout of length $\ell_x$7, and a token-level reward is assigned according to whether that candidate leads to confusion after detokenization. The advantage is probability-weighted and normalized,

$\ell_x$8

with $\ell_x$9 the probability-weighted average reward over the candidate set. The paper’s argument is that this preserves the relative ordering among valid tokens while pushing down confusion-inducing ones.

The reported results are consistent with that claim. On four target languages—Chinese, Arabic, Korean, and Japanese—TLPO is evaluated on WPR and RPR for language consistency and on multiple zero-shot CoT benchmarks for general ability. When English is treated as neutral, TLPO reaches RPR 99.19, compared with 96.68 for the baseline, 99.14 for SFT, 98.31 for DPO, and 97.27 for ORPO; its general accuracy is 58.08, close to the baseline 58.35 and above the fine-tuned alternatives. When English is treated as language confusion, TLPO reaches RPR 77.59 and general accuracy 56.17, again the best among the fine-tuned methods. The paper also reports that GRPO was excluded from the main comparison because it caused response-length collapse during training.

"Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement" develops a different token-level formulation for interactive decision-making by language agents (Wen et al., 2024). Here the concern is not language confusion but the mismatch between action-level RL and token-level generation. ETPO models the task as a language-augmented MDP $t(x)$0, uses entropy-regularized RL with a reference policy $t(x)$1, and decomposes the soft Bellman update to the token level. For an action $t(x)$2, the token-conditioned soft $t(x)$3-update distinguishes non-final tokens from the final token that receives the environment reward. The paper emphasizes that discounting applies between actions, not within a token sequence.

The point of that decomposition is both computational and statistical. The action space for textual actions scales as $t(x)$4, whereas ETPO reduces exploration to $t(x)$5. It also replaces uniform action-level credit assignment with token-by-token credit. In a simulated data-science code-generation environment using 14 datasets, CodeLlama-7B trained with ETPO reaches average validation ROC AUC 0.8090, compared with 0.8005 for PPO-KL and 0.7973 for Reflection. The paper additionally reports very small perplexity changes on Github, Wikitext, and CodeLlama-7B at 2.0633 / 5.9850 / 3.5768 versus ETPO at 2.0648 / 5.9874 / 3.5787. This is presented as evidence that token-level RL can improve interactive performance without materially damaging base language modeling behavior.

4. Trajectory-level preference decomposition

"LambdaPO: A Lambda Style Policy Optimization for Reasoning LLMs" addresses a different granularity problem: the loss of pairwise structure when GRPO reduces a group of trajectories to a scalar baseline (Yuan et al., 19 May 2026). The paper calls this GRPO’s “relational bottleneck.” In GRPO, the sequence-level advantage is centered around the group mean reward,

$t(x)$6

which only indicates whether a trajectory is above or below the cohort average. LambdaPO replaces that scalar summary with a LambdaRank-inspired pairwise decomposition.

For a prompt $t(x)$7, the model samples a group $t(x)$8. The Pairwise Decomposed Advantage for trajectory $t(x)$9 is

$g(x)$0

where $g(x)$1 and $g(x)$2. The reward is also augmented with a semantic density term,

$g(x)$3

and the composite reward used in experiments is described as $g(x)$4. The optimization objective remains PPO/GRPO-like at the token level, but the advantage signal is constructed at the trajectory level through pairwise ranking comparisons.

That distinction matters. The paper explicitly states that LambdaPO is not token-level in reward construction and not merely sequence-level mean-normalized RL. Its primary optimization unit is the trajectory or reasoning trace; the comparison unit is the pairwise relation between trajectories; the gradient application unit is still the token, because the LM remains autoregressive. This preserves the pairwise structure that GRPO discards while retaining a critic-free implementation.

The experimental setup uses OpenR1-Math-220k and GSM8K for training, AIME24, AIME25, MATH-500, and GPQA-Diamond for evaluation, and Qwen3-1.7B, Qwen3-4B, and Phi-4-mini as models. The reported improvements over GRPO are consistent across models. For Qwen3-4B, the GRPO average is 75.40 and the LambdaPO average is 76.49, an improvement of +1.45. For Phi-4-mini, the GRPO average is 55.35 and the LambdaPO average is 56.38, an improvement of +1.86. The paper also reports stability improvements and notes that performance in one temperature sweep is best around $g(x)$5. In this formulation, the linguistic unit is the coherent reasoning trajectory, and the policy signal is derived from its relative position inside a cohort rather than from an absolute scalar baseline.

5. Linguistically expressed experience priors

"Linguistic Relative Policy Optimization for Video Anomaly Reasoning" shifts the notion of policy optimization further away from parameter updates and toward editable textual memory (Leng et al., 1 Jul 2026). The setting is weakly supervised video anomaly detection with multimodal LLMs. A frozen vision-LLM with fixed parameters $g(x)$6 generates anomaly reasoning under prompt $g(x)$7, and LRPO changes the inference distribution from $g(x)$8 to $g(x)$9, where $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$0 is an experience context selected from an editable linguistic experience repository $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$1.

The pipeline has three stages. First, for each sample the frozen learner generates a group of $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$2 reasoning trajectories under the same video, prompt, and current experience context. Second, each trajectory receives an anomaly alignment reward. Third, an optimizer LLM performs group-wise reflection: it summarizes the trajectories, contrasts high-reward and low-reward outputs, distills group-relative semantic advantages, and converts them into explicit edits over the repository. The repository can be updated by adding, modifying, deleting, or keeping experience items rather than by rewriting the whole memory.

The repository has two complementary components. General experience, $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$3, captures transferable anomaly preferences across scenes and is capped at $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$4. Scenario experience, $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$5, encodes scene-dependent rules using a template of the form “When in <scene type>, if <cue/event> happens, judge it as <anomaly type>.” Scenario items are indexed by visual and textual embeddings, and inference retrieves Top-$$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$6 relevant entries through a dual-branch visual-semantic score. The selector returns $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$7, and the retrieved text is concatenated into the model context. The paper’s theoretical note argues that modifying the text context causes a first-order perturbation in the hidden state and can shift the output distribution analogously to a small parameter update.

The anomaly alignment reward combines accuracy, preference alignment, and temporal dependency. Accuracy aligns the parsed anomaly score $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$8 with the coarse label $$p(\ell \mid x) \propto \exp\!\left(\frac{A_{t(x),\ell} + \mathbb{I}[g(x)\neq\varnothing]\,B_{g(x),\ell}}{\tau}\right).$$9. Preference alignment compares the output embedding to a positive scenario experience $\mathbf{A}$0 and to hard negatives $\mathbf{A}$1 using a temperature-scaled contrastive form. Temporal dependency compares ordered and shuffled frame sequences and rewards experience that improves ordered reasoning more than shuffled reasoning. In the appendix, the paper writes a PPO-like objective with within-group normalized advantages, but the object being updated is the experience repository rather than model weights.

The results emphasize low-annotation and tuning-free operation. Using InternVL3_5-8B as the learner and GPT-OSS-120B as the optimizer, LRPO is trained with only 100 training videos per dataset for the main setting, corresponding to 2.5% of XD-Violence and 6% of UCF-Crime. It reaches 73.17% AP on XD-Violence with only 100 videos and 74.09% with the full training set, 85.36% AUC on UCF-Crime with 100 videos and 86.59% with the full set, and 75.81% AUC on UBnormal when transferring experiences learned on XD-Violence, with 76.24% in the same full-set setting. The ablations are also diagnostic: without experience injection the baseline on XD-Violence is 59.93% AP; adding learned general experience with only accuracy reward raises it to 68.48%; adding scenario experience further raises it to 70.58%; adding preference reward yields 69.78%, temporal reward 70.06%, both together 71.91%, and adding scenario experience on top of the full reward setup gives the best 73.17%. This formulation broadens the meaning of linguistic-unit policy optimization: the optimized “policy” can be a persistent, human-readable textual prior rather than a weight tensor.

6. Document-level rewriting and authorship obfuscation

"TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods" applies policy optimization to whole-document rewriting under a privacy-utility trade-off (Loiseau et al., 2024). The task is to generate an obfuscated text $\mathbf{A}$2 from an original document $\mathbf{A}$3 written by author $\mathbf{A}$4, such that authorship attribution fails while a downstream task label is preserved. Privacy is defined by requiring $\mathbf{A}$5, and utility by requiring $\mathbf{A}$6.

TAROT uses a two-stage training strategy: supervised fine-tuning to initialize a rewriting model, then policy optimization with either PPO or DPO. The PPO reward is

$\mathbf{A}$7

and the reward model itself combines a utility term and a privacy term,

$\mathbf{A}$8

with final reward $\mathbf{A}$9. The starting point is the Keep It Simple simplification model, a fine-tuned GPT2-medium trained on Newsela, and policy optimization is conducted on Yelp reviews in an unsupervised manner. For DPO, preference pairs are synthesized by sampling two generations from the SFT model, computing reward components, and filtering them with $\mathbf{B}$0 and $\mathbf{B}$1.

The empirical picture is explicitly trade-off driven. On IMDb-20, the original text has utility 79.46 and attribution 99.80, while TAROT-DPO reaches utility 60.72 and attribution 17.34; the paper highlights this as a drop of 82.46% in attacker accuracy. On AMT-20, TAROT-PPO yields utility 72.22 and attribution 17.86, while TAROT-DPO yields utility 64.18 and attribution 16.67. The paper’s summary is that policy optimization improves the privacy-utility balance beyond text-editing baselines, DPO is generally more private than PPO, and PPO often preserves utility better than DPO. It also reports that generation-based methods such as GPT-3.5, SFT, TAROT-PPO, and TAROT-DPO are more resistant to adversarial retraining than Synonyms and ALISON.

The appendix’s content-preservation table on IMDb-10 makes the same balance visible in another metric space. TAROT-DPO has Rouge-1 42.52, Rouge-2 17.27, Rouge-L 29.14, BLEU 10.77, METEOR 30.04, BERTScore 80.56, and CoLA 81.10. The paper notes that text-editing methods preserve content best, generation methods produce more fluent and acceptable text, and TAROT-DPO has the best CoLA score among the generation-based methods. In this setting, the optimized linguistic unit is the entire regenerated text, and policy optimization is used to control a document-level attribute trade-off rather than a local decoding error.

7. Cross-cutting patterns, misconceptions, and significance

Several common patterns recur across these otherwise heterogeneous methods. First, the papers repeatedly reject undifferentiated sequence-level optimization when the task structure is more localized or more relational. TLPO argues that sequence-level DPO, ORPO, and GRPO are too coarse for language confusion; LambdaPO argues that GRPO’s scalar baseline erases pairwise structure; multilingual LRPO argues that fixed input-language or English-anchored supervision overlooks complementary knowledge across languages (Choo et al., 29 Apr 2026, Yuan et al., 19 May 2026, Guo et al., 25 May 2026).

Second, policy optimization in this literature is not synonymous with model-weight updates. The video-anomaly LRPO paper explicitly optimizes a linguistic experience repository and injects it into the context without any parameter updates, while still writing the method in PPO-like terms. That point matters because it separates “policy optimization” as a control-theoretic idea from “fine-tuning” as a specific implementation choice (Leng et al., 1 Jul 2026).

Third, reward design is consistently tied to the linguistic unit. Multilingual LRPO calibrates cross-lingual semantic similarity and gates it by language consistency; TLPO uses short lookahead to define token-local rewards at confusion points; ETPO regularizes toward the base LM and preserves token-level language modeling structure; LambdaPO adds semantic density reward through $\mathbf{B}$2; TAROT combines GTE-based utility with LUAR-based privacy. This suggests that the success of these methods depends at least as much on unit-specific reward shaping and calibration as on the outer optimization algorithm (Guo et al., 25 May 2026, Wen et al., 2024, Yuan et al., 19 May 2026, Loiseau et al., 2024).

A common misconception is that finer linguistic granularity is always preferable. The papers do not support that claim. TLPO and ETPO both argue for token-level updates, but they do so because their target problems—language confusion and language-agent action decomposition—are intrinsically token-sensitive. LambdaPO, by contrast, explicitly treats the reasoning trajectory as the meaningful unit, and the video-anomaly LRPO paper treats reusable textual experience as the optimized object. A plausible implication is that the correct unit is task-dependent rather than universally minimal.

Another misconception is that multilingual policy optimization should always anchor to English or to the source language. The multilingual LRPO results directly contest that view, including the Ferragamo case study in which French is the informative rollout language for a Japanese query. Similarly, the video-anomaly LRPO transfer result on UBnormal suggests that a language-expressed prior can generalize across datasets, while TAROT’s results suggest that document-level generation can be more robust to stronger attackers than small edit-based methods (Guo et al., 25 May 2026, Leng et al., 1 Jul 2026, Loiseau et al., 2024).

Taken together, these works suggest that linguistic unit policy optimization is best understood as a research program rather than a single method. Its central claim is that policy learning for language systems improves when the optimized object is aligned with the structure of the failure mode or supervision source: languages for cross-lingual knowledge routing, tokens for localized decoding errors and language-agent actions, trajectories for rank-sensitive reasoning, editable textual priors for tuning-free multimodal reasoning, and whole documents for privacy-preserving rewriting.