Diversity-Based Context Blending

• Diversity-based context blending is a technique where context is built as a set-level object to combine relevance with non-redundant, semantically distinct support.
• It employs methods like windowed MMR, submodular mutual information, and reranking to balance similarity and diversity in various applications.
• Empirical results demonstrate improved pre-LLM recall and generation quality in long-context QA, diffusion models, and contextual MARL.

Diversity-Based Context Blending denotes a family of selection, reranking, interpolation, and guidance procedures in which a context is constructed as a set-level object rather than a pure top-$k$ list. Its central premise is that relevance alone is usually too narrow: highly similar passages, demonstrations, prompts, or curriculum states tend to be mutually redundant, whereas a useful context should retain relevance while also preserving coverage, non-redundancy, and semantically distinct support. In retrieval-augmented generation, long-context prompting, and in-context learning, this family of methods addresses finite context budgets and redundancy under quadratic self-attention cost; in contextual MARL and generative modeling, analogous mechanisms are used to prevent mode collapse, attention collapse, and overfitting to a narrow region of the context space (Wang et al., 13 Feb 2025, Kapuriya et al., 3 May 2025, Satheesh et al., 28 Aug 2025, Ram et al., 2024).

1. Core definition and problem setting

The recurring problem is that a context window is limited, while the candidate pool is large and internally redundant. In long-context LLM systems, this is tied to the quadratic time complexity of self-attention, $O(N^2)$, so the selection problem is not merely which items are individually relevant, but which subset best uses the available token budget (Wang et al., 13 Feb 2025). In standard retrieval for ICL, the neighborhood $\mathcal{N}_k(x;\theta)$ is formed by the $k$ most similar examples under a lexical or semantic similarity function, but the retrieved neighbors are often mutually similar as well as similar to the query, which can produce a topical bias and a “hit or miss” prompt (Kapuriya et al., 3 May 2025).

Across domains, “context” refers to different objects but serves the same functional role. In RAG and summarization it consists of selected sentences or chunks; in ICL it consists of demonstrations or exemplars; in multi-turn intent understanding it consists of retrieved prior turns paired with labels; in contextual MARL it consists of parameterized environment settings; and in text-to-image generation it can refer both to explicit textual prompt conditions and to internal contextualized representations that mediate prompt interpretation (Lin, 20 Oct 2025, Satheesh et al., 28 Aug 2025, Dahary et al., 30 Mar 2026).

A consistent motivation is that redundancy is harmful for different reasons in different systems. In ICL it wastes scarce demonstration slots and can over-emphasize one narrow local pattern; in RAG it lowers pre-LLM recall of answer-bearing evidence; in personalization for diffusion models it causes the learned concept token to overfit the small set of training contexts and weaken prompt fidelity on new prompts (2505.19426, Wang et al., 13 Feb 2025, Kim et al., 2024).

2. Retrieval-time formulations and set construction

Several representative formulations instantiate diversity-based context blending as a greedy trade-off between relevance and dissimilarity, or as a direct set objective.

Domain	Representative method	Core rule
RAG / long-context QA	windowed MMR, FPS (Wang et al., 13 Feb 2025)	maximize query relevance while penalizing similarity to recently selected items, or select the farthest point from the selected set
ICL reranking	DICL, TopK-Div (Kapuriya et al., 3 May 2025, 2505.19426)	retrieve a local candidate pool, then greedily balance query similarity and inter-example diversity
ICL targeted selection	InSQuAD (Nanda et al., 28 Aug 2025)	maximize Submodular Mutual Information under a context budget
Multi-turn intent retrieval	LDRA (Lin, 20 Oct 2025)	maximize a set-level combination of label coverage and text diversity under relevance and label constraints

In long-context QA and summarization, windowed Maximum Marginal Relevance is written as

$$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$

where $R$ is the remaining candidate set, $r_i$ is the relevance of candidate $i$ to the query, $\alpha\in[0,1]$ balances relevance versus diversity, and $W$ is a window of the most recently selected items rather than the full selected set. The same study also considers Farthest Point Sampling,

$O(N^2)$0

with Euclidean distance instead of cosine similarity (Wang et al., 13 Feb 2025).

In DICL, diversity is introduced only after local over-retrieval. A larger neighborhood is first formed,

$O(N^2)$1

with $O(N^2)$2 in all experiments, and Maximum Marginal Relevance is then used to rerank within that local pool:

$O(N^2)$3

This keeps selection local to the query while reducing redundancy among the final $O(N^2)$4 demonstrations (Kapuriya et al., 3 May 2025).

A closely related ICL formulation is TopK-Div, which greedily selects demonstrations according to

$O(N^2)$5

where $O(N^2)$6 interpolates between pure similarity and pure diversity. In the same study, a separate Div method constructs a diverse coreset $O(N^2)$7 of size $O(N^2)$8 and only then applies TopK within that reduced set, making the coreset size $O(N^2)$9 the effective diversity–similarity control (2505.19426).

InSQuAD replaces pairwise heuristics with a submodular set objective. The exemplar set is chosen by

$\mathcal{N}_k(x;\theta)$0

and greedy selection proceeds by marginal gain under the same Submodular Mutual Information function. This yields a targeted selection procedure in which quality and diversity are coupled within a unified SMI objective rather than optimized in separate stages (Nanda et al., 28 Aug 2025).

In multi-turn intent retrieval, LDRA makes the set-level structure explicit. It combines label diversity

$\mathcal{N}_k(x;\theta)$1

with text diversity

$\mathcal{N}_k(x;\theta)$2

into

$\mathcal{N}_k(x;\theta)$3

and selects

$\mathcal{N}_k(x;\theta)$4

subject to a relevance threshold $\mathcal{N}_k(x;\theta)$5 and a per-label cap $\mathcal{N}_k(x;\theta)$6. This is explicitly a constrained set-optimization problem rather than a pointwise reranker (Lin, 20 Oct 2025).

3. Coverage, ordering, and what diversity is intended to improve

A central distinction in this literature is between retrieval quality before generation and downstream generation quality after the LLM or diffusion model acts. In long-context QA, pre-LLM recall asks whether the answer appears in the selected context, whereas post-LLM recall asks whether the model output contains the correct answer. This distinction matters because retrieval can improve substantially while downstream gains saturate if the LLM becomes the bottleneck (Wang et al., 13 Feb 2025).

Empirically, diversity-aware retrieval raises pre-LLM recall in both single-document and multi-document QA. On Natural Questions single-document pre-LLM recall, Sentence-BERT retrieval improves from $\mathcal{N}_k(x;\theta)$7 at $\mathcal{N}_k(x;\theta)$8 to $\mathcal{N}_k(x;\theta)$9 with SB+MMR; on NarrativeQA the gains are from $k$0 to $k$1. In summarization, ROUGE improves from $k$2 to $k$3 on gov_report and from $k$4 to $k$5 on legal, while GPT-4 judge win rates rise to about $k$6 and $k$7, respectively (Wang et al., 13 Feb 2025).

Ordering is part of context blending rather than a separate concern. In single-document QA, “index sort” usually works best, meaning selected sentences are restored to their original order before being fed to the LLM. In multi-document QA, placing important chunks near the beginning and end of the prompt can help, consistent with the “Lost in the Middle” effect. In LDRA, the claim that diversity itself matters is tested under equal prompt token budgets and randomized positions; shuffling exemplar order has minor effect, while prefix replacement harms performance, so the reported gains are not reducible to favorable exemplar order or to simply using more tokens (Wang et al., 13 Feb 2025, Lin, 20 Oct 2025).

Theoretical work on ICL argues that diversity helps both through coverage and beyond coverage. Under a linear-regression view of in-context learning, TopK-Div can have lower expected loss than TopK because the second selected example complements the first and covers more of the query’s nonzero coordinates; a second theorem shows lower loss even when both methods already achieve full coverage, indicating that the gain is not exhausted by a simple coverage explanation (2505.19426). A plausible implication is that diversity-based context blending often changes the geometry of the selected set, not merely the count of covered local features.

4. Textual, checkpoint-level, and internal-state blending in diffusion models

In text-to-image personalization, diversity-based context blending has been implemented as contextual enrichment in text space. “Learning to Customize Text-to-Image Diffusion In Diverse Context” creates a contextually rich set of prompts around the personal concept token, spanning Human interactive prompts, Relative position prompts, Background prompts, Image style prompts, and Attributes-changing prompts. Non-concept tokens are masked with probability $k$8, with 15% masking as default, and the training objective combines denoising and masked language modeling:

$k$9

The method requires no architectural modifications and is combined with Textual Inversion, Extended Textual Inversion, DreamBooth, and CustomDiffusion. Reported CLIP-T gains are $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$0 for TI, $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$1 for XTI, $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$2 for DreamBooth, and $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$3 for CustomDiffusion (Kim et al., 2024).

DreamBlend performs blending at inference time across fine-tuning checkpoints. An earlier, underfit checkpoint preserves prompt fidelity and diversity; a later, overfit checkpoint preserves subject fidelity but suffers from catastrophic attention collapse. DreamBlend stores reference cross-attention maps from the early checkpoint and regularizes the later checkpoint by

$$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$4

with latent update

$$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$5

On the DreamBooth benchmark for Stable Diffusion 1.5, DreamBlend reports DINO 0.675, CLIP-I 0.808, and CLIP-T 0.308, compared with DreamBooth at 0.659, 0.805, and 0.296. In the diversity preference study, DreamBlend is preferred over DreamBooth by 61.82% and over Custom Diffusion by 72.70% (Ram et al., 2024).

A more internal formulation appears in diffusion transformers. “On-the-fly Repulsion in the Contextual Space for Rich Diversity in Diffusion Transformers” intervenes on enriched text tokens $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$6 inside multimodal attention blocks, using

$$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$7

where $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$8 is the negative von Neumann entropy of a cosine-similarity kernel over the batch. The method reports runtime overhead of about 20%–30%, with examples including Flux-dev $$\operatorname*{argmax}_{i \in R} \left[ \alpha \cdot r_i - (1-\alpha)\cdot \max_{j \in W}\text{cos}(i,j) \right],$$9, SD3.5-Turbo $R$0, and SD3.5-Large $R$1, while increasing Vendi Score across all tested models (Dahary et al., 30 Mar 2026).

A separate zero-shot concept-blending study reports a user study with 100 participants and four methods—ALT, SWI, PRO, and UNE—across Same, Different, Compound Words, Concept and Style, and Architecture. The table gives GLOBAL TOTAL means of 2.33 for ALT, 2.32 for SWI, 2.75 for PRO, and 2.60 for UNE. The abstract states that no single approach dominates in all scenarios and that prompt ordering, conceptual distance, and random seed affect the outcome (Olearo et al., 30 Jun 2025).

5. Curriculum generation, latent-space search, and adjacent formulations

In contextual MARL, diversity-based context blending appears as a history-aware curriculum safeguard. cMALC-D stores prior $R$2 pairs in a buffer $R$3, queries an LLM with a sliding window $R$4, computes a similarity score

$R$5

and, if similarity remains too high for too many consecutive steps, blends the proposed next context with a random prior context:

$R$6

with $R$7. The visible hyperparameters include $R$8 and $R$9. Across traffic-control environments, cMALC-D is reported to improve sample efficiency, converge more smoothly, achieve the strongest generalization, and remain more stable than both cMALC without diversity and cMALC-$r_i$0, which applies random blending with probability $r_i$1 (Satheesh et al., 28 Aug 2025).

An adjacent formulation performs blending in a latent search space rather than in prompt or curriculum space. “Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencoder” trains a VAE on level segments and uses MAP-Elites over a 32-dimensional latent space. In the blended domain spanning SMB, KI, and MM, Blend-Elites reports QD-score 8267.31, coverage 49.64%, and optimality 99.66% for Density-Nonlinearity; QD-score 5262, coverage 62.22%, and optimality 99.72% for Symmetry-Similarity; and QD-score 455, coverage 88.87%, and optimality 100% for Game Elements (Sarkar et al., 2021). This suggests a broader interpretation in which blending is the structured illumination of a shared latent manifold under explicit quality-diversity search.

A further adjacent line appears in compositional zero-shot learning. CDS-CZSL models attribute specificity from both object diversity and object context, defining a binary specificity signal by whether two object embeddings with the same attribute fall in the same K-Means cluster, and learning a specificity predictor $r_i$2. The reported AUC improvements over the second-best method are +1.8%, +3.2%, and +0.6% in closed-world CZSL on MIT-States, UT-Zappos, and C-GQA, with open-world gains of +1.6%, +2.0%, and +0.28% (Li et al., 2024). The mechanism is not a retrieval pipeline, but it operationalizes the same theme: context and diversity are combined to refine which compositions should be preferred.

6. Empirical regularities, misconceptions, and limitations

A common misconception is that these methods win simply by adding more context. Several studies explicitly control for this. LDRA evaluates under equal prompt token budgets, introduces a Top-$r_i$3+rand-add fairness baseline, and reports that randomized positions do not erase the gains (Lin, 20 Oct 2025). DICL likewise keeps the mechanism lightweight by reranking only within a local pool with $r_i$4, rather than by retrieving arbitrary distant examples (Kapuriya et al., 3 May 2025).

A second misconception is that diversity is uniformly beneficial. The evidence is more conditional. DICL outperforms or matches the standard similarity-only baseline in 17 out of 24 settings, about 70% of cases, and improvements are most common when $r_i$5 and $r_i$6 (Kapuriya et al., 3 May 2025). In the broader ICL study, TopK is usually best on simple classification tasks such as SST-2, while diversity-aware methods improve more on GeoQuery, math, reading comprehension, hard examples, and OOD settings (2505.19426). This suggests that diversity is most valuable when the task requires complementary structure rather than a single dominant local cue.

A third regularity is that retrieval gains and downstream gains can decouple. In long-context QA, MMR is slightly better than FPS and much faster in Python, but stronger retrieval does not always produce proportionally stronger post-LLM performance. The clearest saturation case is TriviaQA, where downstream accuracy stays around 64% for GPT-3.5 and around 76% for GPT-4 almost regardless of the retrieval variant (Wang et al., 13 Feb 2025). In such regimes, the generator rather than the retriever is the bottleneck.

The methods also expose domain-specific limitations. cMALC-D uses a similarity trigger $r_i$7 whose exact closed form is not specified in the provided excerpt, and its linear interpolation assumes comparable parameterizations of contexts (Satheesh et al., 28 Aug 2025). DreamBlend cannot recover prompt fidelity if the pretrained model itself cannot follow the prompt, and cross-attention guidance weakens when the edit model is too overfit (Ram et al., 2024). In diffusion transformers, intervention placement matters: later intervention improves ImageReward but reduces diversity, while early or full-trajectory intervention increases diversity but can hurt fidelity (Dahary et al., 30 Mar 2026).

Across these lines of work, a plausible implication is that diversity-based context blending shifts the design target from the single best item to the geometry of the selected set or trajectory. Whether implemented as MMR over sentences, SMI over exemplars, set-wise label and text diversity, MLM over prompt contexts, cross-attention guidance across checkpoints, or similarity-triggered curriculum interpolation, the recurring claim is not that relevance is dispensable, but that relevance without diversity is frequently structurally incomplete.