Directed-Token Approach in Code Watermarking

Updated 9 July 2026

Directed-Token Approach is a method that embeds statistically detectable watermarks in code by biasing token predictions using a learned, context-sensitive policy.
The approach uses a reinforcement learning framework with execution feedback and reward shaping to ensure both code correctness and watermark robustness.
It employs Gumbel Top-k for discrete green-list selection, balancing hard token decisions with differentiable relaxation for effective watermark placement.

The Directed-Token Approach, as introduced in CodeTracer, is a policy-driven code watermarking method in which a small watermarking policy $\pi_\phi$ is interposed alongside a frozen base code LLM $\pi_\theta$ and used to bias next-token prediction only at selected decoding positions. The method is designed for the highly structured and syntactically constrained environment of code generation: it aims to embed a statistically detectable watermark while preserving code functionality, and it is trained by reinforcement learning with execution feedback, watermark embedding signals, and a differentiable relaxation for discrete token-selection decisions (Guo et al., 16 Aug 2025).

1. Policy-driven token biasing

At decoding step $t$ , the frozen model $\pi_\theta$ produces unmodified logits

$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$

where $V$ is the full vocabulary. The watermarking policy $\pi_\phi$ observes the same context

$c_t=(x,y_{(t-c)\dots(t-1)})$

and outputs two objects: a binary switch $w_t\in\{0,1\}$ indicating whether watermarking should be applied at that position, and a subset $G_t\subset V$ of size $\pi_\theta$ 0, called the green list (Guo et al., 16 Aug 2025).

The composite policy $\pi_\theta$ 1 biases the logits by adding a fixed positive bias $\pi_\theta$ 2 only to vocabulary items that belong to the selected green list and only when the switch is active: $\pi_\theta$ 3 Sampling then proceeds from $\pi_\theta$ 4. In this formulation, watermarking is not a uniform perturbation over all decoding steps. Because $\pi_\theta$ 5 can set $\pi_\theta$ 6, no bias is added at positions judged unsafe for watermarking. When $\pi_\theta$ 7, the green list is chosen by a learned subpolicy so that the biased tokens remain syntactically valid in context (Guo et al., 16 Aug 2025).

Semantic preservation is enforced operationally rather than by a static constraint. During training, outputs that fail compilation or test cases are punished through the reward system. A common misconception is that the method simply forces “green” tokens whenever possible; the actual mechanism is selective and context-conditioned, with the policy learning when to abstain and how to choose a green list that is compatible with valid code generation.

2. Reinforcement-learning formulation

Learning the watermark policy parameters $\pi_\theta$ 8 is cast as a policy-gradient problem under Group Relative Policy Optimization (GRPO). In each rollout, the system generates $\pi_\theta$ 9 completions under $t$ 0, collects execution and watermarking feedback, forms advantages, and updates only $t$ 1; the base model parameters $t$ 2 remain frozen (Guo et al., 16 Aug 2025).

The reward decomposes into three terms. The execution reward $t$ 3 evaluates functional correctness: $t$ 4 The outcome-based watermark reward $t$ 5 measures statistical detectability through the watermark z-score: $t$ 6 Here $t$ 7 is the number of positions where $t$ 8, and $t$ 9 counts how many sampled tokens at those positions fall in the corresponding green lists. The process-based watermark reward $\pi_\theta$ 0 supplies token-level credit: $\pi_\theta$ 1 These are combined through a weighted sum

$\pi_\theta$ 2

with $\pi_\theta$ 3 chosen by hyperparameter sweep (Guo et al., 16 Aug 2025).

The full objective maximizes discounted reward under a KL regularizer against a lagged reference policy $\pi_\theta$ 4, which is a lagged copy of $\pi_\theta$ 5. The regularization term is explicitly described as a small penalty to prevent drifting too far from $\pi_\theta$ 6 (Guo et al., 16 Aug 2025). This coupling of execution reward and watermark reward is central: correctness is treated as a first-class optimization target rather than as a post hoc filter.

3. Discrete green-list selection and Gumbel Top-k

Choosing $\pi_\theta$ 7 is a discrete top- $\pi_\theta$ 8 decision over a vocabulary-sized score vector $\pi_\theta$ 9, which is non-differentiable. To address this, the method adopts Gumbel Top-$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$0 reparameterization. The perturbed logits are

$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$1

and the green list is defined by

$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$2

This provides a hard set in the forward pass (Guo et al., 16 Aug 2025).

For backpropagation, CodeTracer uses a differentiable relaxation via a Gumbel-Softmax soft indicator vector $l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$3: $l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$4 where $l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$5 denotes stop-gradient. A temperature $l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$6 may be applied inside $l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$7 to control sharpness. The result is a straight-through construction: the forward pass uses the hard top-$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$8 set, while the backward pass propagates through the continuous relaxation (Guo et al., 16 Aug 2025).

This component is technically consequential because the watermark policy does not merely choose whether to watermark. It also learns which subset of the vocabulary should receive the bias at each step. The Gumbel Top-$l_j=\bigl[\logit(\pi_\theta(y_t=v_j\mid x,\,y_{<t}))\bigr]_{j=1}^{|V|},$9 mechanism therefore turns green-list construction into an optimized, context-sensitive control problem rather than a fixed heuristic.

4. Architecture and generation procedure

The watermark policy is implemented as a small transformer encoder with 6 layers, 8 heads, $V$ 0, and $V$ 1. Its input is the last $V$ 2 tokens of context plus embeddings. It outputs a scalar logit $V$ 3 for the binary watermark switch, using a straight-through sigmoid, and a $V$ 4-dimensional vector $V$ 5 for green-list selection (Guo et al., 16 Aug 2025).

The generation loop follows a fixed pattern. At each step, the policy reads the recent context, predicts the watermark switch and green-list scores, thresholds the switch by $V$ 6, and either leaves the base logits untouched or adds $V$ 7 to produce biased logits. The next token is then sampled from the resulting softmax. The RL loop further reconstructs $V$ 8 for each generated token, computes $V$ 9, $\pi_\phi$ 0, and $\pi_\phi$ 1, aggregates advantages, normalizes group rewards, subtracts reference-policy log-probabilities, and adds the KL penalty before updating $\pi_\phi$ 2 (Guo et al., 16 Aug 2025).

Two implementation details matter for interpretation. First, the policy has access only to a short local context window, not the full sequence history. Second, the base model remains frozen. This suggests that the Directed-Token Approach is intended as an adaptive watermarking layer over an existing code generator rather than as a retraining strategy for the generator itself.

5. Empirical performance and attack resilience

The reported evaluation compares CodeTracer with a no-watermark base model and practical watermark baselines WLLM and EXP-edit. The reproduced summary reports the following values (Guo et al., 16 Aug 2025):

Method	Pass@1/10	AUROC / TPR@5%FPR
Base	65.42 / 79.17	—
WLLM	58.05 / 70.35	70.17% / 20.73%
EXP-edit	59.29 / 72.41	66.50% / 25.61%
CodeTracer	60.82 / 79.72	82.95% / 46.34%

These results are described as demonstrating significant superiority over state-of-the-art baselines in both watermark detectability and preservation of generated code functionality (Guo et al., 16 Aug 2025). The functional metrics show that CodeTracer narrows the gap to the unwatermarked base model relative to the other practical watermark baselines, while the detection metrics show materially stronger separability.

Under adversarial code-transformation attacks, CodeTracer is reported to retain the highest AUROC and TPR. The supplied example is renaming, where CodeTracer achieves 73.36% AUROC versus 62.02% for EXP-edit (Guo et al., 16 Aug 2025). Scaling to the larger 8B model yields 78.69% AUROC with only an approximately 0.4% drop in Pass@1, indicating that the approach is not limited to the specific smaller model configuration used in the main experiments.

A common misunderstanding is that stronger detectability necessarily requires aggressive distortion of generated programs. The reported results suggest a different operating point: token-level, policy-conditioned biasing can improve detectability while remaining competitive in Pass@1 and Pass@10 relative to practical watermark baselines.

6. Position within the broader “directed-token” vocabulary

The phrase “Directed-Token Approach” is not standardized across arXiv, and its meaning depends strongly on context. In CodeTracer, it denotes policy-driven token biasing for code watermarking (Guo et al., 16 Aug 2025). In TGDPO, a related phrase refers to token-level reward guidance for Direct Preference Optimization, where different tokens are assigned varying degrees of deviation from a reference policy through token-wise weights (Zhu et al., 17 Jun 2025). In large language-vision models, “directed tokens” denotes learnable embeddings appended to the sequence and used to reconstruct the order of shuffled visual inputs (Truong et al., 19 Aug 2025). In efficient LLM inference, OrthoRank uses a directed-token interpretation based on tokens moving toward a sink token in hidden-state space and ranks them by orthogonality to that sink (Shin et al., 5 Jul 2025).

This suggests that the shared label identifies a broader design pattern rather than a single technique: token-level control is made explicit, and token behavior is directed by an auxiliary mechanism rather than left entirely to the base model dynamics. Within that broader family, CodeTracer is specifically an adaptive watermarking framework in which the directed-token mechanism operates at next-token logit level, is trained by GRPO, and is grounded in execution-aware reward shaping (Guo et al., 16 Aug 2025).

The main conceptual significance of the CodeTracer variant is therefore not merely that it biases token choice, but that it turns watermark placement into a learned control problem over discrete generation decisions. The method’s operational safeguards—selective application through $\pi_\phi$ 3, context-dependent green-list construction, execution reward, and KL regularization—define the distinctive content of the Directed-Token Approach in code watermarking.