---
title: 'ConvFill: Real-Time Conversational Infill Model'
url: https://www.emergentmind.com/topics/convfill
type: topic
---

# ConvFill: Real-Time Conversational Infill Model

ConvFill is a **360M-parameter on-device conversational infill model** for real-time voice agents that separates low-latency conversational surface generation from high-capability backend reasoning. In the formulation introduced in "ConvFill: Model Collaboration for Responsive Conversational Voice Agents" [2511.07397], a lightweight local model begins responding almost immediately, while a larger cloud model processes the full dialogue in parallel and streams short knowledge chunks that are incorporated into the unfolding response. The central claim is that this **decouples user-perceived response latency from backend model capability**, yielding a system that remains responsive while still accessing cloud-scale knowledge. The reported headline result is **sub-200 ms response latency** together with **36–42 percentage-point accuracy improvements over standalone small models of the same size**.

## 1. Latency–capability tradeoff and the motivating problem

ConvFill is motivated by a deployment tension in conversational voice systems. **Cloud-based foundation models** offer stronger reasoning, broader knowledge, and greater flexibility for domain specialization, but they also introduce delays that are especially damaging in spoken interaction. In a cascaded **ASR \(\rightarrow\) LLM \(\rightarrow\) TTS** system, the critical quantity is not only total completion time but also **time-to-first-response**: several seconds of silence can make the system appear awkward or broken. By contrast, **on-device small models** provide low time-to-first-token and immediate responsiveness, but they are markedly weaker in reasoning and factual or domain knowledge [2511.07397].

The paper frames responsiveness as a property of **natural spoken interaction**, not merely a latency benchmark. Prior work cited there is summarized as indicating that latency above a few seconds harms experience, that generic fillers can be worse than contextually appropriate responses, and that contingent conversational continuations are preferred to merely fast ones. ConvFill therefore targets a stronger objective than emitting placeholders such as “let me think.” The local model is trained to generate **natural, context-sensitive partial dialogue** while awaiting delayed backend knowledge.

This yields a specific notion of **model collaboration**. The backend is not the direct conversational front-end. Instead, it operates at the **conversation level**, seeing the full dialogue history and returning either concise knowledge chunks or silence. The on-device model operates at the **turn level**, seeing the current user utterance together with the streamed signals for the current turn and producing the user-facing dialogue.

## 2. Conversational infill as a formal task

The paper defines **conversational infill** as a task in which a small on-device model accepts **external textual knowledge chunks** from a large backend model, receives them **delayed in time** and in **streaming form**, and uses them to generate a natural response over the course of the turn. This is explicitly not standard sequence completion, span infilling, or speculative decoding. The local model is not proposing tokens for later verification; it is performing **streaming phrase-level continuation under delayed external conditioning** [2511.07397].

For a single conversational turn, time is indexed by \(t\), and the external stream is

\[
E = (e_1, e_2, \dots),
\]

where each event \(e_i\) is either a backend knowledge chunk \(l_i \in L\) or a silence token \(s_i \in S\). In inference, a silence token is inserted whenever a duration \(d\) has passed since the last knowledge chunk, with

\[
d = 1 \text{ second}.
\]

At step \(i\), ConvFill conditions on the current user utterance, the streamed external sequence so far \((e_1, \dots, e_i)\), and its own prior in-turn phrases \((c_1, \dots, c_{i-1})\), and then generates the next conversational phrase \(c_i\). The turn-level alignment constraint is

\[
\text{len}(S) + \text{len}(L) = \text{len}(C) = n.
\]

A central grounding requirement is the entailment relation

\[
e_i \rightarrow c_i \quad \text{if } e_i \in L,
\]

meaning that when the external input is a backend knowledge chunk, the generated phrase should be entailed by that chunk. For silence steps, the paper indicates that **no entailment relation is required** between \(s_i\) and \(c_i\). It also notes a sentence that appears to contain a typo, but the intended interpretation is clear from context: silence-conditioned phrases are unconstrained by backend entailment.

This formulation makes ConvFill closer to **incremental turn construction** than to classical infilling. Previously emitted text is not revised, and blanks inside a prewritten sequence are not filled. Instead, the model extends the turn in alternating response to backend chunks and silence markers.

## 3. Architecture and inference protocol

ConvFill is built by fine-tuning **SmolLM2-360M-Instruct**, and the resulting system remains a **360M parameter model**. Its conditioning format is deliberately simple. The model uses the existing `user` and `assistant` roles from SmolLM2 and adds a new `knowledge` role. The `knowledge` stream contains either a backend utterance chunk or the special token `<|sil|>`, yielding an interleaved prompt structure of the form:

- `user`
- `knowledge`
- `assistant`
- `knowledge`
- `assistant`
- ...

During training, the model is always trained to predict the **final assistant phrase** in that interleaving [2511.07397].

A notable technical feature is what the system does **not** introduce. The paper does not describe a separate encoder for backend chunks, explicit cross-attention blocks, learned gating between local and backend streams, alignment modules, token-level fusion, or revision of previous outputs. Backend information is incorporated entirely through **prompt-level conditioning in an autoregressive causal LM**. The architectural novelty therefore lies primarily in the task framing and the inference protocol rather than in a new transformer mechanism.

The inference pipeline uses two parallel threads. When a user utterance arrives, it is sent simultaneously to ConvFill and to the backend model. A dialogue manager maintains full conversation history for the backend and streams back concise knowledge phrases. The two threads communicate via a **streamed knowledge queue**. If no backend chunk is available, the system pushes `<|sil|>` every \(d=1\) second. At each event, ConvFill takes the current user utterance, all prior in-turn `knowledge` inputs, and all prior in-turn assistant phrases, and generates the next phrase. This realizes an asymmetric collaboration: the backend provides delayed conversation-level intelligence, while the local model preserves immediate turn-level responsiveness.

## 4. Synthetic data generation and training regime

ConvFill is trained on a synthetic corpus rather than on a pre-existing conversational dataset. The paper argues that corpora such as **CANDOR** are a poor fit because they are more casual and contain filler-use patterns that are not ideal for goal-directed assistant behavior. The synthetic data are generated using **GPT-4o** across six domains: **advice**, **assistant queries**, **education**, **event planning**, **customer service**, and **medical conversation** [2511.07397].

The generation process initially targets **1000 conversations per domain**. For **medical**, **customer service**, and **education**, the pipeline first generates **1000 domain-specific personas** to diversify situations. For **advice**, **assistant**, and **planning**, it instead generates **1000 domain-specific subtopics**. Each conversation is represented as JSON with fields `user`, `responder`, and `responder_thoughts`. The `responder` field is a list of standalone sentences spoken by the model, while `responder_thoughts` contains concise thoughts or silence aligned to each sentence. Operationally, responder sentences become target conversational phrases, and responder thoughts become streamed backend knowledge chunks or silence signals.

The final corpus contains **5997 conversations** spanning the six domains and **46,151 user-infill turns**, with conversations ranging from **8–12 turns**. To reduce semantic mismatch between knowledge chunks and target phrases, the authors use a public **DeBERTaV3-base model fine-tuned on MNLI** to verify that each streamed knowledge chunk **entails** the associated conversational phrase. This semantic verification is directly tied to the formal grounding condition \(e_i \rightarrow c_i\).

Training is performed **per turn**, not across full conversations. ConvFill sees only the current user utterance, previous streamed knowledge chunks for the current turn, and previous conversational phrases for the current turn. The backend is therefore responsible for broader dialogue state, while ConvFill is responsible for local turn construction. Reported optimization settings are: **5 epochs**, **learning rate \(1 \times 10^{-5}\)**, **500 warmup steps**, a **cosine learning rate scheduler**, and training on an **NVIDIA L40 GPU**. The tokenizer is modified to include the special token `<|sil|>`, and embeddings are resized accordingly. Because training uses GPT-4o-generated `responder_thoughts` rather than live outputs from the evaluation backends, a plausible implication is the presence of a train–test style mismatch between synthetic supervision and deployment-time backend phrasing.

## 5. Evaluation design and quantitative results

The evaluation focuses primarily on **Natural Questions (NQ)** using **250 randomly sampled questions**, with accuracy and **TTFT** as the main metrics. A second evaluation axis measures **turn-level entailment** using DeBERTaV3 MNLI-style NLI labels: **entailment**, **neutral**, and **contradiction**. Backend models are **GPT-5**, **Claude Sonnet 4.5**, and **Gemini-2.5-Pro**. Local-model comparisons include **SmolLM2-360M (Base)** together with several other small models, including **SmolLM2-1.7B**, **Qwen3-0.6B**, **Qwen1.7B**, **Llama3.2-1B**, **Gemma3-270M**, and **Gemma3-1B**. Local inference is run in **fp16** with **Ollama** on an **Apple MacBook Pro (M2 Pro, 19-core GPU, 16 GB)** [2511.07397].

| System | NQ accuracy | TTFT |
|---|---:|---:|
| GPT-5 | 0.78 | \(0.74 \pm 3.04\) s |
| ConvFill + GPT-5 | 0.46 | \(0.16 \pm 0.04\) s |
| Claude Sonnet 4.5 | 0.80 | \(2.16 \pm 0.99\) s |
| ConvFill + Claude Sonnet 4.5 | 0.52 | \(0.17 \pm 0.04\) s |
| Gemini-2.5-Pro | 0.69 | \(10.9 \pm 2.74\) s |
| ConvFill + Gemini-2.5-Pro | 0.49 | \(0.17 \pm 0.04\) s |
| SmolLM2-360M (Base) | 0.10 | \(0.16 \pm 0.04\) s |

These results establish the central tradeoff. ConvFill preserves the local model’s response onset—**0.16–0.17 s**—while substantially improving open-domain QA accuracy relative to a same-size standalone local model. The reported **36–42% improvements** are, more precisely, **absolute percentage-point increases** over the **0.10** NQ accuracy of the base **SmolLM2-360M**: \(0.46-0.10=0.36\), \(0.52-0.10=0.42\), and \(0.49-0.10=0.39\). The paper also summarizes the latency effect as roughly **4× to 67× faster response times** than backend-only operation.

Turn-level grounding is weaker than the latency numbers but still structured. For GPT-5, Claude Sonnet 4.5, and Gemini-2.5-Pro backends, entailment rates are **34.1%**, **28.1%**, and **35.5%**; neutral rates are **59.9%**, **64.2%**, and **59.2%**; contradiction rates are **6.0%**, **7.7%**, and **5.3%**. The paper interprets the low contradiction rates as encouraging and attributes much of the large neutral category to conversational embellishment rather than outright semantic drift. The main limitation remains clear: ConvFill improves substantially over small local models but remains well below backend-only QA performance.

## 6. Deployment implications and operational limitations

The intended deployment pattern is a **cascaded speech system** in which **ASR transcribes the user’s speech**, the transcript is sent in parallel to a local ConvFill instance and a cloud backend LLM, ConvFill begins speaking almost immediately via **TTS**, and the backend then streams concise knowledge chunks that are incorporated into later phrases [2511.07397]. This makes the approach particularly applicable to assistants, customer support, education or coaching, and planning or advice agents where silence is undesirable but backend processing remains valuable.

Several strengths follow directly from the reported design. ConvFill offers **very low TTFT**, allows a **modular backend choice**, preserves responsiveness through an **on-device front-end**, and does so **without bespoke architecture** beyond language-model fine-tuning and a queue-based inference system. Because the backend is exposed as a text stream rather than through hidden-state interfaces, the overall system is operationally simple.

The limitations are equally explicit. Final answer quality still lags backend-only systems: backend accuracy ranges from **0.69–0.80**, whereas ConvFill ranges from **0.46–0.52**. Grounding is imperfect, and the model sees only the **current turn** rather than full conversation history. Training is synthetic, evaluation is primarily text-based QA, and the paper reports **no direct human evaluation** of naturalness, interruption handling, or user satisfaction. It also does not provide a detailed mechanism for **barge-in**, incremental ASR updates, turn-end prediction, or spoken overlap management. Reported sub-200 ms latency refers to **model response latency**, not full speech-to-speech latency including ASR and TTS. The paper likewise does not report power consumption, memory footprint, or mobile-device benchmarks beyond the laptop runtime setup.

## 7. Conceptual positioning and distinction from similarly named work

ConvFill occupies a middle position between **cloud-only assistants** and **purely on-device assistants**. Relative to cloud-only systems, it sacrifices some final answer quality in exchange for much lower startup latency. Relative to purely local systems, it preserves local responsiveness while importing a meaningful portion of backend knowledge. The paper also distinguishes ConvFill from **speculative decoding**: speculative decoding accelerates generation of a single target token sequence by having a small model propose tokens that a larger model verifies or corrects, whereas ConvFill uses the backend as a **streaming knowledge provider** for a separate phrase-level conversational generation task [2511.07397].

The work’s novelty is therefore primarily at the level of **task formulation**, **hybrid role separation**, **synthetic supervision**, and **deployment protocol**, not at the level of a novel transformer architecture. The name “ConvFill” refers to filling the **conversational gap** produced by backend latency, not to token infilling in the masked-language-model sense.

A common terminological confusion arises because of the similarly named paper "From Missing Pieces to Masterpieces: Image Completion with Context-Adaptive Diffusion" [2504.14294], which introduces **ConFill**, not ConvFill. That work addresses **image completion / inpainting with diffusion models**, centered on **Context-Adaptive Discrepancy (CAD)** and **Dynamic Sampling**, whereas ConvFill addresses **responsive conversational voice agents** through on-device conversational infill. The similarity in names is accidental; the two systems belong to different problem domains, use different model classes, and pursue different optimization objectives.

Taken together, the evidence presented for ConvFill supports a specific systems interpretation: a small local language model can act as a **responsive conversational surface**, while a slower backend supplies delayed knowledge that is integrated phrase by phrase. This does not eliminate the backend-quality gap, but it does establish conversational infill as a concrete operating point in the broader latency–capability design space for voice agents.

Source: https://www.emergentmind.com/topics/convfill