Expert-guided Bag Reward Transformer (EBaReT)

• The paper introduces a framework that models auto-bidding as sequence decision-making using expert trajectories, PU learning, and bag reward redistribution.
• EBaReT constructs expert trajectories and applies a Transformer-based model to mitigate sub-optimal bids and high reward variance in sparse conversion environments.
• Empirical results show EBaReT outperforms state-of-the-art baselines, demonstrating its effectiveness in handling long-term dependencies and low-quality logged data.

Expert-guided Bag Reward Transformer (EBaReT) denotes, in its named form, a generative reinforcement learning framework for automated bidding that formalizes bidding as sequence decision-making and combines expert trajectory generation, a Positive–Unlabeled (PU) learning-based discriminator, a Bag Decision Transformer, and bag-based reward redistribution to address low-quality logged data and uncertainty rewards (Li et al., 22 Jul 2025). In broader arXiv usage, the same label also serves as a precise conceptual description for methods that learn dense or redistributed rewards from expert guidance while aggregating over bags of transitions, traces, or expert references, even when the original paper does not use the name explicitly (Fanconi et al., 2 Oct 2025, Plashchinsky, 7 Dec 2025).

1. Terminology and conceptual scope

In the auto-bidding literature, EBaReT is introduced as a framework for automated bidding in which long-term dependencies, data quality, and sparse conversion feedback are handled jointly through sequence modeling, expert guidance, and bagged rewards (Li et al., 22 Jul 2025). The framework is explicitly motivated by two difficulties: logged historical data contain many sub-optimal actions, and conversion rates are $<0.01$, so winning auctions still rarely yield reward feedback.

Across related work, the term acquires a broader technical meaning. In "Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning," the paper does not mention EBaReT by name, but the method can naturally be viewed as an “Expert-guided Reward Transformer” with a “bag” aggregation at both training and inference time: the discriminator is expert-guided, Transformer-based, and trained over per-prompt bags of expert and policy traces, while inference reranking operates over a bag of sampled trajectories (Fanconi et al., 2 Oct 2025). In "Parent-Guided Semantic Reward Model (PGSRM)," the EBaReT label appears as a structured generalization from a single parent reference to a bag of experts whose embedding similarities are aggregated into a reward (Plashchinsky, 7 Dec 2025).

This variation is important because the word bag does not denote a single invariant object across the literature. In the auto-bidding setting, a bag is a contiguous time window of transitions; in the reasoning setting, it is a per-prompt set of expert and policy traces or a bag of sampled candidate traces; in the PGSRM-to-EBaReT mapping, it is a bag of expert references. A common misconception is therefore that EBaReT names one fixed architecture. The literature instead supports a narrower claim: it names one concrete bidding framework and also functions as a family resemblance concept for expert-guided, bag-aggregated reward transformers.

2. Sequence decision-making formulation for automated bidding

EBaReT models auto-bidding as conditional sequence modeling rather than as a conventional Markov Decision Process. The motivation is that advertisers’ outcomes depend on long-horizon constraints such as budget and RoS, on market diurnal cycles, and on strategic interactions that violate short-horizon assumptions and make stepwise credit assignment difficult (Li et al., 22 Jul 2025). The framework follows the Decision Transformer paradigm in which the agent predicts actions conditioned on future returns-to-go and history.

The return-to-go is defined as

$R_t = \sum_{k>t} r_k.$

To exploit pronounced diurnal patterns, each day is partitioned into consecutive time intervals called bags. A bag $B_i$ of length $B$ contains a subsequence

$$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$

and a full trajectory is written as

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$

Tokens carry content embeddings plus bag position embeddings, allowing the model to encode within-bag locality and across-bag long-term dependencies.

The core supervised training problem is dual-headed. EBaReT predicts the current step’s RTG from state context,

$\hat{R}_t = f_{\theta}(\ldots, s_t),$

and predicts the next action from state and current RTG,

$\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$

The training loss is the sum of mean-squared errors: $$\mathcal{L}(\theta) = \sum_{(s,a) \in \mathcal{D}} (\hat{R}_t - R_t)^2 \;+\; \sum_{(s,a) \in \mathcal{D}} (\hat{a}_t - a_t)^2.$$

This formulation directly targets the two bottlenecks emphasized in the paper. First, pure supervised imitation of noisy logs can reproduce sub-optimal bids. Second, sparse conversion signals induce high-variance RTG targets. EBaReT addresses the former by constructing expert trajectories and the latter by bagging transitions and redistributing rewards within each bag.

3. Architecture, expert guidance, and bag reward design

EBaReT integrates three mechanisms: expert trajectory generation, a PU learning-based discriminator, and bag reward redistribution within a Transformer sequence model (Li et al., 22 Jul 2025). Expert trajectories are generated for each advertiser under truthful auction assumptions, starting from the standard auto-bidding linear program with budget and RoS constraints. Under auctions satisfying truthful optimality, the dual LP yields expert bids

$$b_j = \frac{1 + \alpha_{c} \cdot C}{\alpha_{b} + \alpha_{c} \cdot v_j},$$

where $R_t = \sum_{k>t} r_k.$0 captures the budget constraint’s shadow price, $R_t = \sum_{k>t} r_k.$1 captures the RoS constraint’s shadow price, and $R_t = \sum_{k>t} r_k.$2 is the RoS target. These trajectories serve as labeled positive examples and as anchors for sequence modeling.

To avoid treating all logged transitions as negatives, EBaReT trains a discriminator $R_t = \sum_{k>t} r_k.$3 with Positive–Unlabeled learning. With expert positives $R_t = \sum_{k>t} r_k.$4, unlabeled logs $R_t = \sum_{k>t} r_k.$5, and positive prior $R_t = \sum_{k>t} r_k.$6, the non-negative PU objective is

$R_t = \sum_{k>t} r_k.$7

The trained discriminator produces an expert-likeness score that is discretized into $R_t = \sum_{k>t} r_k.$8 expert levels. During training, transitions carry their level token; during inference, the highest expert level token $R_t = \sum_{k>t} r_k.$9 is used: $B_i$0

Reward uncertainty is handled by bag reward design. For a bag $B_i$1, the total reward is

$B_i$2

Reward is then redistributed across transitions using the discriminator: $B_i$3

$B_i$4

The RTG update becomes

$B_i$5

This gives each step a shaped credit signal consistent with expert-likeness and the bag’s total reward.

The paper reports the following hyperparameters: bag size $B_i$6, PU class prior $B_i$7, redistribution sharpness $B_i$8, and expert levels $B_i$9. It does not specify optimizer, learning rate, batch size, context length, or number of layers and heads. In computational terms, multi-head attention scales as $B$0 per layer for sequence length $B$1, and inference latency is described as comparable to Decision Transformer because no beam search or post-filtering is required.

4. Empirical performance and ablation evidence

The experiments are conducted in the AuctionNet (Alibaba) offline environment. The dataset contains 21 advertising periods indexed 7 to 27; each period contains over 500,000 impression opportunities divided into 48 steps, each opportunity includes 50 agents, and the dataset has over 500 million records with predicted conversion value, bid, auction, and impression outcomes. Training uses periods 7–13 and testing uses periods 14–20. The evaluation metric is cumulative conversions per period (Li et al., 22 Jul 2025).

The offline comparison against CQL, IQL, BC, DT, and DiffBid reports the following cumulative conversions over test periods $B$2–$B$3:

Method	P14–P20 cumulative conversions	Note
CQL	[31.09, 29.99, 29.88, 29.73, 27.97, 33.45, 27.99]	Offline RL baseline
IQL	[32.75, 25.43, 27.05, 35.09, 30.38, 34.07, 28.39]	Offline RL baseline
BC	[28.53, 27.37, 30.55, 28.64, 29.13, 31.70, 26.43]	Generative RL baseline
DT	[30.01, 29.79, 30.31, 31.92, 29.52, 34.42, 29.70]	Generative RL baseline
DiffBid	[31.95, 28.62, 30.65, 35.59, 27.72, 34.36, 29.79]	Diffusion-based bidder
EBaReT	[37.89, 33.31, 36.09, 40.40, 33.42, 40.66, 33.71]	Best in all seven periods

EBaReT achieves the best performance in all seven periods, with gains of several conversions per period over the strongest baselines. The paper does not report statistical significance tests.

Ablation studies isolate the contribution of expert data, PU learning, expert-guided inference, and bag reward redistribution:

Variant	P14–P20 cumulative conversions	Interpretation stated in the paper
EBaReT (full)	[37.89, 33.31, 36.09, 40.40, 33.42, 40.66, 33.71]	Full method
EBaReT$B$4	[30.44, 30.53, 32.62, 31.75, 31.18, 34.66, 28.80]	No expert data; degenerate to DT
EBaReT$B$5	[37.26, 33.71, 33.25, 36.12, 32.99, 38.37, 31.14]	Treat all offline data as negative
EBaReT$B$6	[34.92, 31.86, 30.80, 34.32, 30.78, 33.97, 29.79]	Remove expert-guided inference
EBaReT$B$7	[36.17, 35.41, 34.89, 35.91, 33.41, 38.98, 34.23]	Remove reward redistribution

These ablations confirm the contributions claimed by the paper, while also showing sensitivity: $B$8 attains the highest value in $B$9 and $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$0, suggesting that bag reward performance depends on bag settings. The online test, conducted for one week from 2024-11-19 to 2024-11-25 against a baseline similar to IQL, reports equal budget $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$1 and plan $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$2 for both systems, with conversions increasing from $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$3 to $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$4 and revenue from $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$5 to $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$6, corresponding to $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$7 conversions and $$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$8 revenue.

5. Theoretical lineage in bagged reward reinforcement learning

The theoretical substrate for EBaReT-style methods is supplied by Reinforcement Learning from Bagged Reward (RLBR), which formalizes Bagged Reward Markov Decision Processes (BRMDPs) in settings where the environment supplies rewards over bags rather than immediately per action (Tang et al., 2024). A bag is a contiguous subsequence

$$B_i=\left\langle s_t, R_t, a_t, \ldots, s_{t+B-1}, R_{t+B-1}, a_{t+B-1} \right\rangle,$$9

with bag reward

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$0

If bags form a disjoint partition of the trajectory timesteps and each bag reward is the sum of the underlying per-step rewards, then the paper’s Theorem 1 states that the sets of optimal policies in the standard MDP and the BRMDP coincide.

This policy-equivalence result motivates reward redistribution. If a redistributed reward $$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$1 satisfies

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$2

for each bag, then optimizing the expected sum of $$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$3 is equivalent to optimizing the bagged-return objective. The paper then proposes the Reward Bag Transformer (RBT), which uses a causal Transformer encoder followed by bidirectional self-attention to predict per-step redistributed rewards $$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$4 and a dynamics head to predict $$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$5. Its losses are a bag-sum reward consistency term

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$6

and a state-prediction term

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$7

The exact term EBaReT is not used in RLBR. However, the structured extension associated with that work constructs an expert-guided Bag Reward Transformer by adding expert demonstrations, optional expert value functions, cross-attention to expert memory, expert ranking losses, and Q- or advantage-regression losses on top of the RBT redistribution mechanism (Tang et al., 2024). This suggests a clear genealogy: EBaReT in auto bidding operationalizes expert guidance and bag-level reward smoothing in a domain-specific way, while RLBR supplies the general return-equivalence logic for bagged rewards.

6. EBaReT-style reward transformers for language-model reasoning and semantic reward modeling

In language-model reasoning, the closest EBaReT-style construction is the adversarial inverse-reinforcement-learning framework of "Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning" (Fanconi et al., 2 Oct 2025). The paper reframes adversarial IRL for LLM reasoning and learns a dense, token-level reward model directly from expert demonstrations rather than by style imitation through supervised fine-tuning. The autoregressive policy is

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$8

and the discriminator objective is

$$\tau=\left\langle \ldots, B_i, \ldots \right\rangle.$$9

The token-level reward is the discriminator logit,

$\hat{R}_t = f_{\theta}(\ldots, s_t),$0

which yields dense token-level feedback.

The bag structure appears twice. During training, for each prompt $\hat{R}_t = f_{\theta}(\ldots, s_t),$1, the method forms a group $\hat{R}_t = f_{\theta}(\ldots, s_t),$2 and standardizes rewards with GRPO-style statistics computed across the bag. During inference, the policy samples $\hat{R}_t = f_{\theta}(\ldots, s_t),$3 candidate traces, with default $\hat{R}_t = f_{\theta}(\ldots, s_t),$4, the reward model scores each by a mean discounted token-level reward over answer tokens, and a top-$\hat{R}_t = f_{\theta}(\ldots, s_t),$5 reranker selects the best under a fixed compute budget. On GSM8K with Llama3.1-8B policy and Llama3.2-1B discriminator, the Pearson correlation between learned reward on answer tokens and correctness is $\hat{R}_t = f_{\theta}(\ldots, s_t),$6 for raw rewards and $\hat{R}_t = f_{\theta}(\ldots, s_t),$7 for discounted rewards with $\hat{R}_t = f_{\theta}(\ldots, s_t),$8. Correlations with formatting-based signals are much smaller. Reward-guided reranking improves pass@$\hat{R}_t = f_{\theta}(\ldots, s_t),$9 over random ranking for Llama3-based policies across $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$0, whereas for Qwen2.5 models the separation is weaker and reranking gains are not statistically significant. The paper further reports that token-level reward heatmaps localize errors, with incorrect solutions showing a sharp negative band at the first error.

A different but related generalization appears in PGSRM, where the starting point is a single parent-guided semantic reward

$\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$1

with $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$2 in reported experiments, L2-normalized embeddings, and single-step PPO training (Plashchinsky, 7 Dec 2025). The EBaReT mapping extends this to a bag of experts or references with aggregators such as weighted sum, max, or softmax-weighted cosine, optionally followed by truncation and sharpening: $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$3 This suggests that EBaReT is not limited to reward redistribution over time. It can also designate aggregation over multiple expert reference outputs, with the reward remaining sequence-level and lightweight because expert outputs and embeddings are cached offline.

7. Limitations, sensitivities, and open directions

The concrete auto-bidding formulation has several stated limitations. Expert trajectories rely on truthful auction assumptions and dual LP multipliers $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$4; if auctions deviate from truthfulness or those estimates are biased, expert trajectories may misguide training and inference (Li et al., 22 Jul 2025). Fixed bag length $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$5 may not align with true reward dynamics across advertisers or days. The fixed PU prior $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$6 can miscalibrate $\hat{a}_t = f_{\theta}(\ldots, s_t, R_t).$7 if the true positive fraction differs. Staged training simplifies optimization but does not provide end-to-end joint gradients. The paper therefore suggests future work toward a more unified framework that redistributes both long-term and short-term rewards without manual priors and potentially with joint learning of expert guidance and sequence generation.

The reasoning variant inherits a different failure profile. Adversarial co-training can lag SFT on pass@1 for some backbones; reward signals may inherit biases or noise from demonstrations and are non-stationary; learned critics can pick up spurious cues; reward hacking remains a concern; compute and latency increase because of alternating updates and inference-time scoring; and evaluation is limited to GSM8K arithmetic reasoning rather than long-context, multimodal, or high-stakes domains (Fanconi et al., 2 Oct 2025). The paper notes that perturbations such as sign flips, number corruption, and answer swaps help train the reward to prefer semantic correctness over stylistic similarity, but do not eliminate reward hacking.

The semantic-reward generalization has its own constraints. PGSRM inherits parent bias, may suffer reward-proxy mismatch when cosine similarity fails to capture task goals, and can under-reward solutions that genuinely surpass the expert reference manifold (Plashchinsky, 7 Dec 2025). This suggests that multi-expert bagging can reduce single-expert bias, but it does not remove the need for calibration, aggregation design, or auxiliary safeguards against trivial similarity.

Taken together, these results position EBaReT as a general pattern rather than a single closed recipe: expert guidance supplies a quality prior, bag structure supplies a unit for reward smoothing or aggregation, and the Transformer supplies a sequence model that can condition on long-horizon context. The precise operational meaning of those three components depends on domain—auction trajectories, reasoning traces, or expert reference sets—but the central objective remains the same: to convert weak, sparse, or noisy supervision into a reward signal that is smoother, more discriminative, and more useful for downstream policy optimization.