FedPOB-Pref: Federated Prompt Optimization

• The paper introduces FedPOB-Pref, a federated dueling bandit method that leverages binary preference feedback to optimize prompt selection in black-box LLMs.
• It employs a dynamic regularization technique with parameter and gradient aggregation, enabling efficient convergence with significantly fewer communication rounds.
• Empirical results show FedPOB-Pref outperforms baselines by achieving an average performance of 0.8482 on Instruction Induction using 30× fewer rounds than comparable methods.

FedPOB with Preference Feedback (FedPOB-Pref) is a federated linear dueling bandit algorithm for prompt optimization when only comparative user feedback is available. Introduced as an extension of FedPOB, it addresses sample-efficient federated prompt optimization for black-box LLMs by letting each agent compare two prompts, observe a binary preference, and collaborate through parameter aggregation with dynamic regularization rather than sharing prompts, responses, or raw feedback (Lu et al., 29 Sep 2025).

1. Problem setting and scope

FedPOB-Pref targets federated prompt optimization in a setting with a shared task $\mathbb{D} = (\mathbb{X}, \mathbb{Y})$, a black-box LLM accessed via API, and a set of agents $\mathbb{A} = \{1,\dots,N\}$. Each agent $a$ has its own prompt space $\mathbb{P}_a$, and these prompt spaces are overlapping but not identical across agents. At round $t$, agent $a$ selects a pair of prompts $(p_{t,a}^1, p_{t,a}^2)\in\mathbb{P}_a^2$, queries the LLM, and receives preference feedback

$\omega_{t,a} = \mathds{1}(p_{t,a}^1 \succ p_{t,a}^2) \in \{0,1\}.$

The feedback channel is therefore comparative rather than scalar: users answer which of two prompts produced a better response, not how good a response was on an absolute scale (Lu et al., 29 Sep 2025).

The motivation is threefold. First, the method is designed for black-box LLMs, where gradients and internal states are unavailable. Second, it is explicitly sample-efficient, because LLM API calls are costly. Third, it is federated and privacy-preserving: agents never share prompts, responses, or raw feedback—only model parameters or sufficient statistics. This combination distinguishes FedPOB-Pref from centralized prompt optimization and from score-based federated bandit formulations (Lu et al., 29 Sep 2025).

A central misconception is that preference-only feedback is too weak for prompt optimization. FedPOB-Pref is built on the opposite premise: comparative feedback is often more natural and cheaper to elicit than calibrated scalar scores, especially in RLHF-style or human-in-the-loop settings. The tradeoff is that each query is noisier and less informative than numeric reward, which makes collaborative exploration across agents essential (Lu et al., 29 Sep 2025).

2. Preference model and bandit formulation

FedPOB-Pref inherits the linear representation used in FedPOB. Each prompt $p$ is mapped to an embedding

$u(p) \in \mathbb{R}^d,$

using a pre-trained text encoder; in the reported experiments, MPNet is used and $\mathbb{A} = \{1,\dots,N\}$0. A shared global parameter $\mathbb{A} = \{1,\dots,N\}$1 defines the latent utility of a prompt through the linear score $\mathbb{A} = \{1,\dots,N\}$2 (Lu et al., 29 Sep 2025).

Preferences are modeled with a Bradley–Terry–Luce logistic link. For two prompts $\mathbb{A} = \{1,\dots,N\}$3 and $\mathbb{A} = \{1,\dots,N\}$4, with embeddings $\mathbb{A} = \{1,\dots,N\}$5 and $\mathbb{A} = \{1,\dots,N\}$6,

$\mathbb{A} = \{1,\dots,N\}$7

where $\mathbb{A} = \{1,\dots,N\}$8. The observed preference is therefore

$\mathbb{A} = \{1,\dots,N\}$9

This places FedPOB-Pref in the class of linear dueling bandits, specialized to federated prompt optimization (Lu et al., 29 Sep 2025).

For each agent $a$0, the local negative log-likelihood over previously observed duels is the pairwise logistic loss

$a$1

Conceptually, the arms are prompts, each round chooses a duel rather than a single arm, and the objective is to minimize dueling regret by converging to a prompt that is approximately optimal under the preference distribution induced by the BTL model. The paper states that formal regret analysis for FedPOB-Pref is only sketched and relies on theory from linear dueling bandits and federated optimization (Lu et al., 29 Sep 2025).

3. Client–server algorithm and dynamic regularization

FedPOB-Pref is organized as a synchronous round-based protocol. At round $a$2, the server broadcasts the current global parameter $a$3 and a synchronized information matrix $a$4. Each agent then performs one local dueling-bandit step, returns an updated parameter, a gradient term, and a covariance contribution, after which the server aggregates them into the next global model (Lu et al., 29 Sep 2025).

On the client side, prompt selection is asymmetric in function but not in objective. The first prompt is chosen by pure exploitation: $a$5 The second prompt is chosen by an exploration–exploitation rule in the difference space: $a$6 with

$a$7

The first term estimates the utility difference relative to the current best prompt, while the second is a UCB bonus using global second-order information accumulated across agents (Lu et al., 29 Sep 2025).

After obtaining $a$8, agent $a$9 computes its local model by solving

$\mathbb{P}_a$0

The first term is the local pairwise logistic loss. The linear correction term uses the previous local gradient. The quadratic proximal term anchors the local model to the previous global model and limits client drift. The stored gradient is then updated as

$\mathbb{P}_a$1

and the local covariance contribution is

$\mathbb{P}_a$2

In practice, the paper reports gradient-based optimization with learning rate $\mathbb{P}_a$3 and regularization $\mathbb{P}_a$4 (Lu et al., 29 Sep 2025).

The server receives

$\mathbb{P}_a$5

and updates the global model by

$\mathbb{P}_a$6

while the synchronized information matrix evolves as

$\mathbb{P}_a$7

The first term is a FedAvg-style mean of local models. The second subtracts the average scaled gradient and is the distinctive dynamic-regularization correction. The paper emphasizes that this parameter-and-gradient aggregation improves the performance–communication tradeoff relative to gradient-only or plain FedAvg-style baselines (Lu et al., 29 Sep 2025).

4. Theoretical grounding and methodological context

The theoretical rationale for FedPOB-Pref combines linear dueling bandits with an augmented-Lagrangian view of federated optimization. The local objectives are $\mathbb{P}_a$8, and the consensus problem is written as

$\mathbb{P}_a$9

The appendix introduces the augmented Lagrangian

$t$0

At optimum, the dual variables satisfy

$t$1

which motivates the linear gradient-correction term used in the local objective and the gradient subtraction in the server update. The paper’s claim is not a closed-form regret theorem for FedPOB-Pref, but an argument that, under standard strong convexity and smoothness assumptions for logistic regression in the linear embedding space, the federated dynamic-regularization scheme converges to the same optimum as centralized training on all pairwise data (Lu et al., 29 Sep 2025).

FedPOB-Pref also belongs to a broader lineage of online learning with pairwise or improvement-based feedback. “Online Learning with Preference Feedback” formalized a setting where the learner predicts an object $t$2, receives an improved object $t$3, and updates a linear utility model through the Preference Perceptron update

$t$4

with average regret bounded by

$t$5

under $t$6-informative feedback (Shivaswamy et al., 2011). FedPOB-Pref replaces improved structured outputs by pairwise prompt duels, but it shares the central principle that learning can proceed without observed numeric reward.

Adjacent preference-optimization literature clarifies the surrounding design space. Active DPO uses a last-layer linearization of the DPO objective and D-optimal design to select informative preference pairs, with a maximum logit error bound

$t$7

under log-linear assumptions (Kveton et al., 3 Mar 2025). Preferential Bayesian optimization with Thompson sampling achieves

$t$8

regret under a dueling-kernel model (Lazzaro et al., 27 Apr 2026), and DOPL establishes $t$9 regret for restless bandits with preference feedback (Xiong et al., 2024). These results do not analyze FedPOB-Pref directly, but they show that pairwise feedback can support provable sequential optimization in several regimes.

5. Empirical evaluation and observed behavior

The experimental study uses Instruction Induction and BIG-Bench Hard. The main LLM is GPT-3.5-turbo, with GPT-4o-mini and Qwen3-235B used in ablations. Prompt embeddings are produced by MPNet. Every method is evaluated under the same total validation query budget; for preference-based methods, each iteration uses two queries, one per prompt in the duel, and 50 iterations correspond to 100 validation queries (Lu et al., 29 Sep 2025).

The preference-feedback baselines are APOHF, Double-TS, FLDB-GD, and FLDB-OGD. The reported average performances are as follows.

Benchmark	Method	Average performance
Instruction Induction	APOHF	0.7681
Instruction Induction	Double-TS	0.7859
Instruction Induction	FLDB-GD (10 agents)	0.8244
Instruction Induction	FLDB-OGD (10 agents)	0.8123
Instruction Induction	FedPOB-Pref (10 agents)	0.8482
BBH	APOHF	0.5838
BBH	Double-TS	0.5983
BBH	FLDB-GD (10 agents)	0.6457
BBH	FLDB-OGD (10 agents)	0.6271
BBH	FedPOB-Pref (10 agents)	0.6583

These results carry a strong communication interpretation. FLDB-GD uses 1500 communication rounds, whereas FLDB-OGD and FedPOB-Pref use 50 communication rounds. The paper therefore states that FedPOB-Pref dominates FLDB-OGD at the same communication budget and even outperforms FLDB-GD despite using 30× fewer communication rounds. It also reports that performance monotonically increases with the number of agents, from 1 to 3 to 10 agents, on both Instruction Induction and BBH (Lu et al., 29 Sep 2025).

The ablation study attributes much of this behavior to dynamic regularization. Comparing FedPOB-Pref with and without dynamic regularization, the paper reports faster convergence, less oscillation in performance, and better final accuracy when the dual-corrected update is retained. Prompt pools generated by different LLMs, including GPT-3.5 and GPT-4o-mini, still yield strong performance, which the paper interprets as robustness across prompt domains (Lu et al., 29 Sep 2025).

6. Limitations, misconceptions, and extensions

FedPOB-Pref rests on a narrow but explicit modeling stack. It assumes that preferences are well modeled by a linear function of MPNet embeddings, that the BTL logistic link is an adequate generative model, that federation is synchronous with all agents participating, and that optimization takes place inside a predefined prompt pool generated offline via APE. The paper also states that it does not provide a closed-form regret rate for FedPOB-Pref under federated noise, heterogeneity, and limited communication (Lu et al., 29 Sep 2025).

These limitations are important because several nearby research threads point to plausible extensions. FedPref studies federated learning under heterogeneous multi-objective preferences and uses weighted aggregation plus recursive clustering based on top-$a$0 cosine similarity between model updates, suggesting a route toward personalized or clustered variants when agents differ systematically in prompt preferences rather than merely contributing noisy pairwise labels (Hartmann et al., 23 Jan 2025). This suggests that a personalized FedPOB-Pref could combine dueling-bandit prompt search with similarity-based clustering or per-client scalarization.

Another limitation concerns noisy preferences. Robust DPO analyzes symmetric random preference flips and derives an rDPO objective whose sub-optimality gap scales as

$a$1

showing explicitly how label noise inflates sample complexity (Chowdhury et al., 2024). For privacy-sensitive deployments, label-differential privacy under a linear BTL reward model incurs additional estimation cost $a$2 in the local model and $a$3 in the central model (Chowdhury et al., 2023). FedPOB-Pref itself does not incorporate these mechanisms, but they define concrete robustness and privacy directions for preference-based federation.

Finally, FedPOB-Pref should not be conflated with direct preference optimization for model alignment. DPO-family methods optimize model likelihoods from preference pairs, while FedPOB-Pref optimizes prompt selection under black-box access. PRO revisits DPO through a decomposed optimizer-plus-regularizer formulation and extends direct alignment to pairwise, binary, and scalar feedback, addressing likelihood underdetermination through a proximal regularizer (Guo et al., 29 May 2025). That line is methodologically adjacent rather than identical: it concerns policy alignment, whereas FedPOB-Pref concerns prompt search. The commonality is the use of comparative feedback as the primary supervisory signal.

In this sense, FedPOB-Pref occupies a specific intersection: federated learning, dueling bandits, prompt optimization, and black-box LLM interaction. Its defining contribution is not merely the use of preferences, but the combination of a federated difference-space UCB rule, parameter aggregation with dynamic regularization, and a communication pattern that shares only models and sufficient statistics while still improving systematically as more agents participate (Lu et al., 29 Sep 2025).