Poly-Encoder: Bridging Bi and Cross-Encoders

• Poly-Encoder is a transformer-based architecture for multi-sentence scoring that leverages multiple global context features to enrich candidate evaluation.
• It encodes candidate sequences independently for caching while applying candidate-conditioned attention to improve contextual interactions.
• Empirical comparisons show Poly-Encoder achieves near cross-encoder accuracy with significantly lower computational overhead in large candidate settings.

Poly-Encoder is a transformer architecture for multi-sentence scoring that was introduced to occupy the space between bi-encoders and cross-encoders. In its canonical form, it preserves the bi-encoder property that candidate sequences can be encoded independently and cached, while adding a candidate-conditioned attention step over multiple context representations so that scoring is richer than a single global-vector dot product. The architecture was introduced by Humeau et al. for dialogue response selection and retrieval, and later reused or reinterpreted in settings such as word sense disambiguation and persona-guided conversational ranking (Humeau et al., 2019).

1. Origins and problem formulation

The original Poly-Encoder was proposed for multi-sentence scoring tasks in which a context must be matched against a set of candidate sequences. The paper explicitly situates this setting in next-utterance selection for multi-turn dialogue and retrieval-style ranking, including ConvAI2, DSTC7, Ubuntu V2, and Wikipedia Article Search (Humeau et al., 2019).

The architectural motivation is the classical trade-off between two transformer paradigms. A cross-encoder concatenates context and candidate into a single sequence and performs full self-attention across the pair. This generally yields the strongest token-level interactions, but inference is slow because the full transformer must be rerun for every candidate. A bi-encoder encodes context and candidate separately and compares their reduced representations with a dot product, which is computationally attractive because candidate embeddings can be precomputed and reused, but the matching function is correspondingly limited (Humeau et al., 2019).

Poly-Encoder was introduced as an intermediate design. The key idea is not to permit unrestricted token-level interaction between context and candidate at every transformer layer, but also not to collapse the context to a single fixed vector before candidate scoring. Instead, the context is represented by multiple global features, and the candidate attends over those features when computing its score. The original paper characterizes this as learning “global rather than token level self-attention features” (Humeau et al., 2019).

2. Canonical architecture

In the canonical formulation, the context and candidate are encoded separately by transformer encoders. The candidate representation is reduced to a single vector,

$$y_{cand} = \text{red}(T_{cand}(cand)) = \text{first}(T_{cand}(cand)).$$

The context encoder outputs token-level states $h_1,\dots,h_N$, and Poly-Encoder derives $m$ context features from them rather than a single pooled vector (Humeau et al., 2019).

The original Section 3.3 architecture uses learned poly codes $c_1,\dots,c_m$. Each code attends over the context token states: $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$

$y^i_{ctxt} = \sum_{j=1}^{N} w^{c_i}_j \, h_j.$

These $m$ vectors are global context features. A candidate embedding then performs a second attention step over them: $$(w_1, \dots, w_m) = \text{softmax}(y_{cand_i} \cdot y^1_{ctxt}, \dots, y_{cand_i} \cdot y^m_{ctxt}),$$

$y_{ctxt} = \sum_{i} w_i y^i_{ctxt},$

followed by the final score

$s(ctxt, cand_i) = y_{ctxt} \cdot y_{cand_i}.$

This top-level attention makes the effective context representation candidate-specific, even though the candidate itself was encoded independently and can remain cacheable (Humeau et al., 2019).

The paper also evaluates an alternative in which the context features are taken directly from the first $h_1,\dots,h_N$0 transformer outputs: $h_1,\dots,h_N$1 This “First-m outputs” variant is simpler because it does not introduce an extra attention layer for code construction, but it preserves the same overall pattern: multiple global context vectors followed by candidate-conditioned selection among them (Humeau et al., 2019).

A common misconception is that Poly-Encoder is merely a bi-encoder with several pooled vectors. The original architecture is more specific than that characterization. The defining operation is the candidate-conditioned attention over context-side global features, which introduces a limited but explicit interaction pathway absent from an ordinary bi-encoder (Humeau et al., 2019).

3. Computational profile and empirical trade-off

The practical importance of Poly-Encoder lies in its computational asymmetry. Like a bi-encoder, it allows candidate representations to be cached. Unlike a cross-encoder, it does not require a full transformer pass over each context-candidate pair. Its candidate-specific work is the top-level attention over $h_1,\dots,h_N$2 context features and a dot product, rather than pairwise joint encoding (Humeau et al., 2019).

On ConvAI2, when scoring 100 examples, the original latency table reports the following timings (Humeau et al., 2019):

Architecture	1k candidates	100k candidates
Bi-encoder	CPU 115 ms / GPU 19 ms	CPU 160 ms / GPU 22 ms
Poly-encoder 16	CPU 122 ms / GPU 18 ms	CPU 678 ms / GPU 38 ms
Poly-encoder 64	CPU 126 ms / GPU 23 ms	CPU 692 ms / GPU 46 ms
Poly-encoder 360	CPU 160 ms / GPU 57 ms	CPU 837 ms / GPU 88 ms
Cross-encoder	CPU 21,700 ms / GPU 2,600 ms	CPU 2,200,000 ms / GPU 266,000 ms

These numbers formalize the intended trade-off. Poly-Encoder is only modestly slower than a bi-encoder, while remaining far cheaper than a cross-encoder for large candidate sets. Training cost follows the same pattern: on ConvAI2, the appendix reports 2.0 hours for a bi-encoder, 2.7 hours for Poly-Encoder with 16 codes, 2.8 hours for Poly-Encoder with 64 codes, and 9.4 hours for the cross-encoder on 8 V100 GPUs (Humeau et al., 2019).

The accuracy side of the trade-off is also explicit. With Reddit pre-training, the original paper reports on ConvAI2 test $h_1,\dots,h_N$3: bi-encoder $h_1,\dots,h_N$4, Poly-Encoder 16 $h_1,\dots,h_N$5, Poly-Encoder 64 $h_1,\dots,h_N$6, Poly-Encoder 360 $h_1,\dots,h_N$7, and cross-encoder $h_1,\dots,h_N$8. On Ubuntu V2 test $h_1,\dots,h_N$9, the same paper reports bi-encoder $m$0, Poly-Encoder 16 $m$1, Poly-Encoder 64 $m$2, Poly-Encoder 360 $m$3, and cross-encoder $m$4. The reported pattern is that Poly-Encoder is more accurate than a bi-encoder and often close to a cross-encoder, while remaining practical at scale (Humeau et al., 2019).

4. Variants, design parameters, and training practice

The principal architectural hyperparameter is the number of poly codes $m$5. The original study emphasizes small ($m$6), medium ($m$7), and large ($m$8) settings. Increasing $m$9 provides a more expressive representation of context at the cost of additional computation in the candidate-conditioned attention stage. The paper reports that Poly-Encoder 16 already improves over a bi-encoder, while 64 and 360 codes often improve further; on DSTC7 with original BERT pre-training, Poly-Encoder 360 reaches $c_1,\dots,c_m$0 $c_1,\dots,c_m$1, exceeding the reported cross-encoder score of $c_1,\dots,c_m$2 (Humeau et al., 2019).

The same work treats representation reduction and pre-training as first-order design choices rather than implementation details. For bi- and poly-encoders, it explores reduction by first output, average of outputs, and average of the first $c_1,\dots,c_m$3 outputs, finding that “first output” works slightly better in the main experiments. It also compares three pre-training setups: original BERT weights, an in-house BERT-like pre-training on Toronto Books and Wikipedia, and Reddit pre-training on 174M $c_1,\dots,c_m$4 pairs. The empirical conclusion stated in the paper is that the best results are obtained by pre-training on large datasets similar to the downstream tasks (Humeau et al., 2019).

Fine-tuning behavior is equally specific. The paper reports that fine-tuning only the top layer or top four layers hurts performance, whereas fine-tuning all layers except word embeddings gives the best results. Bi- and poly-encoders exploit batch negatives, using the other labels in the batch as negatives. This is part of their efficiency profile: the architecture is not only lighter at inference, but also better aligned with large-batch discriminative training than a jointly encoded cross-encoder (Humeau et al., 2019).

This suggests that Poly-Encoder is not merely a static architectural block. In the original formulation, it is a family of scoring models whose behavior depends materially on the number of codes, the method used to derive context features, the pre-training corpus, and the fine-tuning regime.

5. Task-specific reinterpretations and extensions

Later work repurposes the Poly-Encoder idea in task-specific ways while preserving its intermediate position between purely independent encoding and full pairwise joint encoding.

In word sense disambiguation, the PolyBERT model builds a poly-encoder on top of BERT and couples it with batch contrastive learning, recasting WSD as an efficient context-gloss matching problem. PolyBERT uses a context encoder $c_1,\dots,c_m$5 and a gloss encoder $c_1,\dots,c_m$6. The local representation of the target word is $c_1,\dots,c_m$7, the gloss representation is $c_1,\dots,c_m$8, and the poly-encoder module replicates the target-word embedding into $c_1,\dots,c_m$9 queries,

$$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$0

which attend over the full context embedding $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$1 through multi-head attention to produce a fused representation $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$2. The gloss side remains simpler, with the $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$3 representation replicated for compatibility. During training, Batch Contrastive Learning uses the correct senses of other target words in the same batch as negatives, so that only the correct gloss per word needs to be encoded. The paper reports overall F1 81.0 for PolyBERT versus 79.0 for BEM and 77.0 for GlossBERT, and a 37.6% GPU-hours reduction for PolyBERT with BCL relative to PolyBERT-A without BCL (Xia et al., 1 Jun 2025).

In conversational ranking with auxiliary persona information, the “Persona-Coded Poly-Encoder” paper uses Poly-Encoder both as a baseline and as a component in a multi-stream extension. The baseline Poly-Encoder pre-fuses persona, dialogue history, and query into a single textual sequence and processes it through one transformer encoder with learned poly codes. The proposed PCPE architecture instead separates the encoding into a persona-coded stream and a poly-encoded stream. The persona-coded stream aggregates persona entries with self-attention, attends from persona to query to construct persona-conditioned query views, and then lets the candidate attend over those views; the poly stream performs the standard Poly-Encoder operation over query embeddings and trainable codes. Post-fusion combines the two streams with either self-attention fusion, candidate-conditioned fusion, or ColBERT-style late interaction. On the PFG dataset, the paper reports improvements over the baseline Poly-Encoder of 3.32% in BLEU score and 2.94% in HR@1, while on ConvAI2 the method is described as nearly at par with strong baselines such as ColBERT (Liu et al., 2023).

These adaptations are structurally significant because they show that “poly codes” need not always be the learned, task-agnostic code vectors of the original formulation. In PolyBERT, the multiple queries are replicated from the target-word embedding rather than learned as free global parameters. In PCPE, persona entries function as task-specific points of view parallel to the trainable poly stream. A plausible implication is that later usage treats Poly-Encoder both as a specific architecture and as a broader design pattern: multiple context-side views, followed by candidate-conditioned selection among them.

6. Relation to competing paradigms, limitations, and later reinterpretation

The most direct later comparison comes from Uni-Encoder, which revisits response selection in generation-based dialogue systems. That paper characterizes Poly-Encoder as a response-ranking method that reduces interaction between context and candidates through “an additional lightweight attention layer,” thereby improving efficiency relative to a cross-encoder but paying a performance cost. In a controlled Ubuntu V2 comparison, it reports $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$4 for a Poly-Encoder with 360 context codes, $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$5 for a standard cross-encoder, and $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$6 for Uni-Encoder; it also states that Uni-Encoder achieves approximately $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$7 faster inference than Cross-Encoder on Ubuntu V2 when the pool size is appropriate (Song et al., 2021).

That comparison does not invalidate the original rationale for Poly-Encoder. The same Uni-Encoder paper explicitly notes a limitation of Uni-Encoder: it is suitable only for generation-based dialogue systems in which the number of responses is small. In retrieval-based systems with very large candidate pools, Poly-Encoder retains its central advantage because responses can be pre-encoded and reused, making it appropriate as a first-stage ranker or retrieval model before a more expensive second-stage reranker (Song et al., 2021).

The limitations of Poly-Encoder are therefore not uniform but regime-dependent. In the original formulation, its main architectural constraints are the absence of token-level cross-attention throughout the network, the reduction of each candidate to a single vector, and the reliance on a fixed number of codes $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$8 (Humeau et al., 2019). In later specialized variants, additional issues appear. PolyBERT remains dependent on large manually annotated corpora such as SemCor and introduces extra hyperparameters such as $$(w^{c_i}_1, \dots, w^{c_i}_N) = \text{softmax}(c_i \cdot h_1, \dots, c_i \cdot h_N),$$9; its richer interaction is also asymmetric, since the gloss side remains relatively simple (Xia et al., 1 Jun 2025). Persona-guided Poly-Encoder variants raise a different class of concerns: the PCPE paper notes the lack of realistic persona datasets, the lack of personalization-specific evaluation metrics, and legal and ethical concerns because persona may include PHI or PII and may create risks of discriminatory or offensive behavior (Liu et al., 2023).

Two further misconceptions can be clarified. First, Poly-Encoder is not defined by dialogue alone: the original paper evaluates it not only on ConvAI2, DSTC7, and Ubuntu V2, but also on Wikipedia Article Search, where Poly-Encoder 360 reaches 71.8 $y^i_{ctxt} = \sum_{j=1}^{N} w^{c_i}_j \, h_j.$0 under Reddit pre-training (Humeau et al., 2019). Second, Poly-Encoder is not synonymous with a single immutable attention block. Across the literature represented here, it denotes an architectural principle in which multiple context-side features mediate candidate-conditioned scoring, and those features may be learned codes, first-$y^i_{ctxt} = \sum_{j=1}^{N} w^{c_i}_j \, h_j.$1 outputs, replicated target-token queries, or streams derived from persona structure.