F2RVLM: RL for Fine-Grained Fragment Retrieval

• F2RVLM is a generation-based multimodal LLM that retrieves semantically coherent dialogue fragments by generating discrete ID sequences.
• It integrates reinforcement learning with Group-Relative Policy Optimization to enhance coherence, format correctness, and fragment order consistency.
• The model leverages transformer backbones and difficulty-aware curriculum sampling to effectively extract fine-grained fragments from long multi-topic conversations.

F2RVLM is a generation-based, reinforcement-learning-enhanced multimodal LLM designed for Fine-grained Fragment Retrieval (FFR) in multi-modal long-form dialogues. In this setting, the goal is not to retrieve a single utterance or image, but to locate semantically coherent fragments consisting of multiple utterances and images that match a query. The model serves as the main retriever in the single-dialogue setting and as the fine-grained reasoning module in the corpus-level two-stage system FFRS, within a broader framework for retrieving multimodal dialogue fragments from long, multi-topic conversational histories (Bi et al., 3 Jun 2026, Bi et al., 25 Aug 2025).

1. Task definition and motivation

Fine-grained Fragment Retrieval is defined as the task of locating semantically coherent fragments in long multimodal dialogues, given a user query $q$. A dialogue is represented as

$D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$

where each turn $t$ has a speaker $u_t$ and message $m_t$, which may contain text and/or images. In the single-dialogue setting, the model predicts sets of utterance and image IDs,

$$\hat{Y} = \mathcal{F}(D, q) = (\hat{I}_{\text{utt}}, \hat{I}_{\text{img}}).$$

In the corpus-level setting, retrieval spans a dialogue collection $\mathcal{D} = \{D_1, \dots, D_N\}$, and the output is a set of dialogue and turn identifiers associated with retrieved fragments (Bi et al., 3 Jun 2026).

The motivating claim is that existing paradigms such as utterance-level response selection and single-item retrieval fail to capture users’ actual information needs in long, multi-topic chats. The reported failure mode is not simply low relevance, but incoherence: retrieved outputs may contain isolated messages, off-topic content, or misaligned turns and images. Even strong multimodal LLMs and embedding models are described as lacking explicit supervision on fragment coherence and as exhibiting over-retrieval, under-retrieval, and turn–image misalignment when asked to reconstruct coherent historical segments (Bi et al., 25 Aug 2025).

A common misconception is to treat F2RVLM as a conventional response-selection model or an embedding-only retriever. It is instead a structured generative retriever that directly predicts utterance and image IDs corresponding to a fragment. This design is motivated by the observation that users frequently need to revisit “the part” of a conversation related to a topic, event, or shared image, rather than the next response or a single matching item (Bi et al., 3 Jun 2026).

2. Model architecture and generative retrieval formulation

F2RVLM is built on top of the Qwen2 and Qwen2.5-VL series, including 2B, 3B, and 7B variants. The backbone consists of a vision encoder that maps images to visual tokens and a transformer LLM backbone that processes text tokens, special ID markers, and visual tokens in a shared multimodal context. The model does not modify the backbone architecture; the changes are in dialogue structuring, output format, and training objective (Bi et al., 3 Jun 2026).

To make identity and position explicit, the dialogue is serialized with dedicated markers. Utterances are tagged with <|utt_id_start|> ... <|utt_id_end|>, and images are tagged with <|img_id_start|> ... <|img_id_end|>. The prompt contains the natural-language query and the entire multi-turn dialogue with these ID markers and embedded images. This makes the retrieval target discrete and verifiable: the model is asked to generate ID sequences rather than free-form textual answers (Bi et al., 25 Aug 2025).

The output schema is likewise structured. For utterances, the target form is $m_t$2 and for images, $m_t$3 with no duplicate IDs and proper syntax. Internally, the model still performs standard autoregressive generation with next-token probabilities $p_\theta(y_t \mid y_{<t}, D, q)$, but the output sequence now encodes retrieval decisions rather than natural-language completion (Bi et al., 3 Jun 2026).

This formulation shifts retrieval from dual-encoder similarity scoring to multimodal sequence generation. A plausible implication is that the model can use the full expressive capacity of a long-context vision–language transformer to jointly reason over dialogue structure, cross-modal evidence, and query intent. The papers explicitly frame this as “retrieval as generation,” in contrast to rank-and-select pipelines (Bi et al., 25 Aug 2025).

3. Training procedure, reward design, and curriculum

F2RVLM is trained in two stages: supervised fine-tuning and reinforcement fine-tuning with Group-Relative Policy Optimization (GRPO). In the supervised stage, each query–dialogue pair is paired with ground-truth utterance and image ID sets, and the model is optimized with standard cross-entropy to produce the correct structured ID sequence. This stage provides a “cold start” policy that learns task formatting and approximate fragment alignment (Bi et al., 3 Jun 2026).

The reinforcement stage uses GRPO, described as a PPO-style RL method without a learned critic and designed for verifiable, rule-based rewards. For each input $Q$, the current policy samples a group of $G$ candidate outputs,

$D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$0

and each candidate receives a scalar reward $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$1. GRPO computes a relative advantage,

$D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$2

and updates the policy toward outputs with higher relative reward, while regularizing against divergence from a reference model (Bi et al., 3 Jun 2026).

The total reward combines three objectives. The Format Reward $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$3 is binary and checks whether both utterance and image sections exist and whether duplicate IDs are absent. The Retrieval F1 Reward $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$4 measures precision and recall over utterance and image ID sets and multiplies the result by an exponential penalty based on the difference between predicted and ground-truth fragment lengths,

$D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$5

The Fragment Order Consistency Reward $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$6 addresses coherence by embedding retrieved utterances and images into a shared space, ordering them according to the dialogue, and computing average cosine similarity between adjacent elements,

$D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$7

If $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$8, a fallback reward such as $D = \{(u_1, m_1), (u_2, m_2), \dots, (u_T, m_T)\},$9 is used (Bi et al., 25 Aug 2025).

A further training component is difficulty-aware curriculum sampling. Using the cold-start model, each training example is scored by retrieval F1 $t$0 and prediction entropy $t$1, then categorized as Easy, Medium, Confusing, or Hard according to quantiles: $t$2 Training begins primarily with Easy and Medium samples and gradually introduces Confusing and Hard samples. The reported purpose is to improve reasoning over dispersed, noisy, and long-range fragments (Bi et al., 3 Jun 2026).

4. Datasets, evaluation protocol, and empirical results

The primary dataset is MLDR, described as the longest-turn multimodal dialogue retrieval dataset to date. It contains 37,030 dialogues, 194,543 images, and 942,414 turns, with an average of 25.45 turns per dialogue and an average of 3.00 topics per dialogue. It is built from DialogCC through strict cleaning, dialogue triplet semantic matching, long-form generation with Qwen3-235B or Qwen3-256B, multi-granularity annotation, and query sampling. Each sample includes a natural-language query, a long multimodal dialogue, and ground-truth utterance and image IDs forming the fragment (Bi et al., 25 Aug 2025).

The real-world benchmark is a WeChat-based test set constructed from conversations voluntarily shared by 12 participants. After cleaning and segmentation, it contains 580 dialogue samples and 1,250 query–dialogue pairs, with an average of 75.38 turns per sample. The data is described as noisy, informal, and multi-topic, with more than 50% of conversations labeled “Work & Tech” due to contributor demographics and privacy filtering (Bi et al., 25 Aug 2025).

Evaluation uses fragment-level Precision, Recall, F1, and Matthews Correlation Coefficient (MCC), computed separately for utterances and images and then combined into joint metrics. Additional structural metrics include Fragment Order Consistency and Query–Fragment Similarity. For the corpus-level retriever FEM, Recall@K is reported at $t$3. For fragment boundary prediction, Pairwise Precision/Recall/F1 and B-cubed Precision/Recall/F1 are used (Bi et al., 3 Jun 2026).

On MLDR validation, F2RVLM-Qwen2-VL-7B achieves Joint F1 $t$4 and MCC $t$5, outperforming Qwen2-VL-7B at F1 $t$6, MCC $t$7; InternVL3-8B at F1 $t$8, MCC $t$9; and MiMo-7B-RL at F1 $u_t$0, MCC $u_t$1. On the WeChat test set, F2RVLM-Qwen2-VL-7B reports Joint F1 $u_t$2 and MCC $u_t$3, compared with GPT-4o at F1 $u_t$4, MCC $u_t$5, and Gemini-2.5-Flash at F1 $u_t$6, MCC $u_t$7 (Bi et al., 25 Aug 2025).

The ablation results attribute part of the model’s generalization to the coherence reward and curriculum schedule. Removing $u_t$8 or the curriculum reduces cross-domain performance, and a mixed training ratio such as 25% SFT plus 75% GRPO yields better real-domain generalization than SFT-only or GRPO-only. Table 5 in the paper further reports that adding curriculum sampling on top of the rewards improves MLDR validation F1 from 84.69% to 87.00% and WeChat test F1 from 50.65% to 55.60% for the 3B setting (Bi et al., 25 Aug 2025).

5. Role in FFRS and corpus-level retrieval

F2RVLM also functions as the fine-grained reasoning component of FFRS, the two-stage system for Fine-grained Fragment Retrieval within a dialogue corpus. This extension addresses the scalability problem that would arise if F2RVLM were applied exhaustively to every dialogue in a large corpus (Bi et al., 3 Jun 2026).

In the offline stage, each long dialogue is decomposed into minimal semantic fragments by a Fragment Boundary Prediction Model based on Qwen2.5-VL-3B fine-tuned for segmentation. Each fragment receives a global dialogue ID, a fragment ID, and local turn-level IDs. These fragments are then encoded by a Fragment Embedding Model (FEM), whose backbone is GME-Qwen2-VL-2B and whose training objective combines inter-fragment contrastive loss and intra-fragment contrastive loss: $u_t$9 The resulting fragment embeddings are stored in a vector database along with metadata (Bi et al., 3 Jun 2026).

At inference time, FEM encodes the query and retrieves Top-$m_t$0 candidate fragments by cosine similarity,

$m_t$1

F2RVLM is then applied to each candidate fragment, returning localized turn and image indices within that fragment. The system aggregates these outputs by dialogue ID, sorts them by turn order, and merges overlaps to reconstruct coherent dialogue-level fragments (Bi et al., 3 Jun 2026).

Empirically, FEM reports Recall@1 = 62.8% and Recall@20 = 98.6% on Query–Fragment retrieval, with higher average query–fragment cosine similarity than zero-shot embedding baselines. On a 537-dialogue WeChat corpus, FFRS reports average latency of approximately 18 seconds per query versus approximately 720 seconds when using F2RVLM alone over all dialogues, along with human evaluation means of about 4.2 for relevance, 4.1 for coverage, and 4.16 for overall quality on a 5-point Likert scale (Bi et al., 3 Jun 2026).

6. Relation to prior work, applications, and limitations

F2RVLM is positioned against several prior retrieval paradigms. Utterance-level selection focuses on picking single responses rather than multi-turn fragments. Single-item retrieval tasks retrieve one utterance or image rather than a structured segment. Dual-encoder retrieval methods and multimodal embedding models such as CLIP, ALIGN, UniIR, UniVL-DR, VLM2Vec, GME, and LamRA are described as efficient but as lacking explicit fragment-level supervision and as not modeling intra-fragment discourse structure. F2RVLM differs by treating retrieval as generation of ID sequences, optimizing with RL for precision and coherence, and explicitly rewarding cross-modal fragment order consistency (Bi et al., 3 Jun 2026).

The reported strengths of F2RVLM are that it is explicitly trained for fragment retrieval rather than response generation, that its multi-objective RL optimizes semantic accuracy, length calibration, intra-fragment coherence, and structural correctness, that difficulty-aware curriculum improves reasoning on long and dispersed fragments, and that it integrates naturally with FFRS for scalable corpus-level retrieval (Bi et al., 3 Jun 2026).

The stated limitations are that it is computationally more expensive than pure embedding-based retrieval, that RL training requires group sampling and multiple candidates per input, that performance depends on the quality of MLDR and WeChat annotations, and that backbone capacity and context length still constrain performance on extremely long or highly noisy dialogues. The WeChat benchmark is also described as topic-biased toward work and technology, which may limit conclusions about other domains (Bi et al., 3 Jun 2026).

The papers identify several application domains: customer service log search, enterprise knowledge management, social messaging history search on platforms such as WeChat, Slack, and Teams, retrieval-augmented conversational assistants, and multimodal CRM analytics. This suggests a broader role for F2RVLM as a retrieval layer for systems that need to recover coherent multimodal “semantic slices” from long conversational histories rather than isolated items (Bi et al., 25 Aug 2025).