ConvFill: Real-Time Conversational Infill Model

• The paper introduces a dual-model collaboration where a lightweight local model generates immediate responses while a cloud model streams delayed knowledge chunks.
• ConvFill is defined as a conversational infill task that incrementally integrates streaming external information to build coherent responses in real time.
• Empirical evaluations show sub-200 ms response latency and 36–42 percentage-point accuracy improvements over standalone small models, highlighting its practical benefits.

ConvFill is a 360M-parameter on-device conversational infill model for real-time voice agents that separates low-latency conversational surface generation from high-capability backend reasoning. In the formulation introduced in "ConvFill: Model Collaboration for Responsive Conversational Voice Agents" (Srinivas et al., 10 Nov 2025), a lightweight local model begins responding almost immediately, while a larger cloud model processes the full dialogue in parallel and streams short knowledge chunks that are incorporated into the unfolding response. The central claim is that this decouples user-perceived response latency from backend model capability, yielding a system that remains responsive while still accessing cloud-scale knowledge. The reported headline result is sub-200 ms response latency together with 36–42 percentage-point accuracy improvements over standalone small models of the same size.

1. Latency–capability tradeoff and the motivating problem

ConvFill is motivated by a deployment tension in conversational voice systems. Cloud-based foundation models offer stronger reasoning, broader knowledge, and greater flexibility for domain specialization, but they also introduce delays that are especially damaging in spoken interaction. In a cascaded ASR $\rightarrow$ LLM $\rightarrow$ TTS system, the critical quantity is not only total completion time but also time-to-first-response: several seconds of silence can make the system appear awkward or broken. By contrast, on-device small models provide low time-to-first-token and immediate responsiveness, but they are markedly weaker in reasoning and factual or domain knowledge (Srinivas et al., 10 Nov 2025).

The paper frames responsiveness as a property of natural spoken interaction, not merely a latency benchmark. Prior work cited there is summarized as indicating that latency above a few seconds harms experience, that generic fillers can be worse than contextually appropriate responses, and that contingent conversational continuations are preferred to merely fast ones. ConvFill therefore targets a stronger objective than emitting placeholders such as “let me think.” The local model is trained to generate natural, context-sensitive partial dialogue while awaiting delayed backend knowledge.

This yields a specific notion of model collaboration. The backend is not the direct conversational front-end. Instead, it operates at the conversation level, seeing the full dialogue history and returning either concise knowledge chunks or silence. The on-device model operates at the turn level, seeing the current user utterance together with the streamed signals for the current turn and producing the user-facing dialogue.

2. Conversational infill as a formal task

The paper defines conversational infill as a task in which a small on-device model accepts external textual knowledge chunks from a large backend model, receives them delayed in time and in streaming form, and uses them to generate a natural response over the course of the turn. This is explicitly not standard sequence completion, span infilling, or speculative decoding. The local model is not proposing tokens for later verification; it is performing streaming phrase-level continuation under delayed external conditioning (Srinivas et al., 10 Nov 2025).

For a single conversational turn, time is indexed by $t$, and the external stream is

$E = (e_1, e_2, \dots),$

where each event $e_i$ is either a backend knowledge chunk $l_i \in L$ or a silence token $s_i \in S$. In inference, a silence token is inserted whenever a duration $d$ has passed since the last knowledge chunk, with

$d = 1 \text{ second}.$

At step $i$, ConvFill conditions on the current user utterance, the streamed external sequence so far $\rightarrow$0, and its own prior in-turn phrases $\rightarrow$1, and then generates the next conversational phrase $\rightarrow$2. The turn-level alignment constraint is

$\rightarrow$3

A central grounding requirement is the entailment relation

$\rightarrow$4

meaning that when the external input is a backend knowledge chunk, the generated phrase should be entailed by that chunk. For silence steps, the paper indicates that no entailment relation is required between $\rightarrow$5 and $\rightarrow$6. It also notes a sentence that appears to contain a typo, but the intended interpretation is clear from context: silence-conditioned phrases are unconstrained by backend entailment.

This formulation makes ConvFill closer to incremental turn construction than to classical infilling. Previously emitted text is not revised, and blanks inside a prewritten sequence are not filled. Instead, the model extends the turn in alternating response to backend chunks and silence markers.

3. Architecture and inference protocol

ConvFill is built by fine-tuning SmolLM2-360M-Instruct, and the resulting system remains a 360M parameter model. Its conditioning format is deliberately simple. The model uses the existing user and assistant roles from SmolLM2 and adds a new knowledge role. The knowledge stream contains either a backend utterance chunk or the special token <|sil|>, yielding an interleaved prompt structure of the form:

• user
• knowledge
• assistant
• knowledge
• assistant
• ...

During training, the model is always trained to predict the final assistant phrase in that interleaving (Srinivas et al., 10 Nov 2025).

A notable technical feature is what the system does not introduce. The paper does not describe a separate encoder for backend chunks, explicit cross-attention blocks, learned gating between local and backend streams, alignment modules, token-level fusion, or revision of previous outputs. Backend information is incorporated entirely through prompt-level conditioning in an autoregressive causal LM. The architectural novelty therefore lies primarily in the task framing and the inference protocol rather than in a new transformer mechanism.

The inference pipeline uses two parallel threads. When a user utterance arrives, it is sent simultaneously to ConvFill and to the backend model. A dialogue manager maintains full conversation history for the backend and streams back concise knowledge phrases. The two threads communicate via a streamed knowledge queue. If no backend chunk is available, the system pushes <|sil|> every $\rightarrow$7 second. At each event, ConvFill takes the current user utterance, all prior in-turn knowledge inputs, and all prior in-turn assistant phrases, and generates the next phrase. This realizes an asymmetric collaboration: the backend provides delayed conversation-level intelligence, while the local model preserves immediate turn-level responsiveness.

4. Synthetic data generation and training regime

ConvFill is trained on a synthetic corpus rather than on a pre-existing conversational dataset. The paper argues that corpora such as CANDOR are a poor fit because they are more casual and contain filler-use patterns that are not ideal for goal-directed assistant behavior. The synthetic data are generated using GPT-4o across six domains: advice, assistant queries, education, event planning, customer service, and medical conversation (Srinivas et al., 10 Nov 2025).

The generation process initially targets 1000 conversations per domain. For medical, customer service, and education, the pipeline first generates 1000 domain-specific personas to diversify situations. For advice, assistant, and planning, it instead generates 1000 domain-specific subtopics. Each conversation is represented as JSON with fields user, responder, and responder_thoughts. The responder field is a list of standalone sentences spoken by the model, while responder_thoughts contains concise thoughts or silence aligned to each sentence. Operationally, responder sentences become target conversational phrases, and responder thoughts become streamed backend knowledge chunks or silence signals.

The final corpus contains 5997 conversations spanning the six domains and 46,151 user-infill turns, with conversations ranging from 8–12 turns. To reduce semantic mismatch between knowledge chunks and target phrases, the authors use a public DeBERTaV3-base model fine-tuned on MNLI to verify that each streamed knowledge chunk entails the associated conversational phrase. This semantic verification is directly tied to the formal grounding condition $\rightarrow$8.

Training is performed per turn, not across full conversations. ConvFill sees only the current user utterance, previous streamed knowledge chunks for the current turn, and previous conversational phrases for the current turn. The backend is therefore responsible for broader dialogue state, while ConvFill is responsible for local turn construction. Reported optimization settings are: 5 epochs, learning rate $\rightarrow$9, 500 warmup steps, a cosine learning rate scheduler, and training on an NVIDIA L40 GPU. The tokenizer is modified to include the special token <|sil|>, and embeddings are resized accordingly. Because training uses GPT-4o-generated responder_thoughts rather than live outputs from the evaluation backends, a plausible implication is the presence of a train–test style mismatch between synthetic supervision and deployment-time backend phrasing.

5. Evaluation design and quantitative results

The evaluation focuses primarily on Natural Questions (NQ) using 250 randomly sampled questions, with accuracy and TTFT as the main metrics. A second evaluation axis measures turn-level entailment using DeBERTaV3 MNLI-style NLI labels: entailment, neutral, and contradiction. Backend models are GPT-5, Claude Sonnet 4.5, and Gemini-2.5-Pro. Local-model comparisons include SmolLM2-360M (Base) together with several other small models, including SmolLM2-1.7B, Qwen3-0.6B, Qwen1.7B, Llama3.2-1B, Gemma3-270M, and Gemma3-1B. Local inference is run in fp16 with Ollama on an Apple MacBook Pro (M2 Pro, 19-core GPU, 16 GB) (Srinivas et al., 10 Nov 2025).

System	NQ accuracy	TTFT
GPT-5	0.78	$t$0 s
ConvFill + GPT-5	0.46	$t$1 s
Claude Sonnet 4.5	0.80	$t$2 s
ConvFill + Claude Sonnet 4.5	0.52	$t$3 s
Gemini-2.5-Pro	0.69	$t$4 s
ConvFill + Gemini-2.5-Pro	0.49	$t$5 s
SmolLM2-360M (Base)	0.10	$t$6 s

These results establish the central tradeoff. ConvFill preserves the local model’s response onset—0.16–0.17 s—while substantially improving open-domain QA accuracy relative to a same-size standalone local model. The reported 36–42% improvements are, more precisely, absolute percentage-point increases over the 0.10 NQ accuracy of the base SmolLM2-360M: $t$7, $t$8, and $t$9. The paper also summarizes the latency effect as roughly 4× to 67× faster response times than backend-only operation.

Turn-level grounding is weaker than the latency numbers but still structured. For GPT-5, Claude Sonnet 4.5, and Gemini-2.5-Pro backends, entailment rates are 34.1%, 28.1%, and 35.5%; neutral rates are 59.9%, 64.2%, and 59.2%; contradiction rates are 6.0%, 7.7%, and 5.3%. The paper interprets the low contradiction rates as encouraging and attributes much of the large neutral category to conversational embellishment rather than outright semantic drift. The main limitation remains clear: ConvFill improves substantially over small local models but remains well below backend-only QA performance.

6. Deployment implications and operational limitations

The intended deployment pattern is a cascaded speech system in which ASR transcribes the user’s speech, the transcript is sent in parallel to a local ConvFill instance and a cloud backend LLM, ConvFill begins speaking almost immediately via TTS, and the backend then streams concise knowledge chunks that are incorporated into later phrases (Srinivas et al., 10 Nov 2025). This makes the approach particularly applicable to assistants, customer support, education or coaching, and planning or advice agents where silence is undesirable but backend processing remains valuable.

Several strengths follow directly from the reported design. ConvFill offers very low TTFT, allows a modular backend choice, preserves responsiveness through an on-device front-end, and does so without bespoke architecture beyond language-model fine-tuning and a queue-based inference system. Because the backend is exposed as a text stream rather than through hidden-state interfaces, the overall system is operationally simple.

The limitations are equally explicit. Final answer quality still lags backend-only systems: backend accuracy ranges from 0.69–0.80, whereas ConvFill ranges from 0.46–0.52. Grounding is imperfect, and the model sees only the current turn rather than full conversation history. Training is synthetic, evaluation is primarily text-based QA, and the paper reports no direct human evaluation of naturalness, interruption handling, or user satisfaction. It also does not provide a detailed mechanism for barge-in, incremental ASR updates, turn-end prediction, or spoken overlap management. Reported sub-200 ms latency refers to model response latency, not full speech-to-speech latency including ASR and TTS. The paper likewise does not report power consumption, memory footprint, or mobile-device benchmarks beyond the laptop runtime setup.

7. Conceptual positioning and distinction from similarly named work

ConvFill occupies a middle position between cloud-only assistants and purely on-device assistants. Relative to cloud-only systems, it sacrifices some final answer quality in exchange for much lower startup latency. Relative to purely local systems, it preserves local responsiveness while importing a meaningful portion of backend knowledge. The paper also distinguishes ConvFill from speculative decoding: speculative decoding accelerates generation of a single target token sequence by having a small model propose tokens that a larger model verifies or corrects, whereas ConvFill uses the backend as a streaming knowledge provider for a separate phrase-level conversational generation task (Srinivas et al., 10 Nov 2025).

The work’s novelty is therefore primarily at the level of task formulation, hybrid role separation, synthetic supervision, and deployment protocol, not at the level of a novel transformer architecture. The name “ConvFill” refers to filling the conversational gap produced by backend latency, not to token infilling in the masked-language-model sense.

A common terminological confusion arises because of the similarly named paper "From Missing Pieces to Masterpieces: Image Completion with Context-Adaptive Diffusion" (Shamsolmoali et al., 19 Apr 2025), which introduces ConFill, not ConvFill. That work addresses image completion / inpainting with diffusion models, centered on Context-Adaptive Discrepancy (CAD) and Dynamic Sampling, whereas ConvFill addresses responsive conversational voice agents through on-device conversational infill. The similarity in names is accidental; the two systems belong to different problem domains, use different model classes, and pursue different optimization objectives.

Taken together, the evidence presented for ConvFill supports a specific systems interpretation: a small local LLM can act as a responsive conversational surface, while a slower backend supplies delayed knowledge that is integrated phrase by phrase. This does not eliminate the backend-quality gap, but it does establish conversational infill as a concrete operating point in the broader latency–capability design space for voice agents.