AI Video Chat Systems

Updated 14 July 2026

AI Video Chat is an emerging multimodal technology that integrates real-time video, audio, and AI dialogue systems for interactive communication.
The architecture employs modular pipelines that combine video encoding, dialogue management, and response synthesis under strict latency constraints.
Systems optimize network transport, emotion recognition, and personalized interaction to enable context-aware, user-centric engagements.

AI video chat is a form of real-time communication in which one conversational endpoint is an AI system rather than a human, typically a cloud-hosted multimodal LLM that ingests live camera and microphone streams and returns text, speech, animation, or grounded visual responses. In the recent literature, the term spans several closely related system classes: emotionally sensitive face-to-face dialogue frameworks, avatar-mediated audio-visual chatbots, chat-centric video understanding models, personalized subject-aware video question answering systems, referentially grounded video dialogue agents, and assistive live-video agents for real-world tasks (Wu et al., 14 Jul 2025, Alnuhait et al., 2023, Park et al., 2024). Across these variants, the recurring technical substrate is an end-to-end loop that couples capture, multimodal encoding, dialogue management, response generation, speech synthesis, rendering, and network transport under tight latency constraints.

1. Conceptual scope and research lineage

AI video chat did not emerge from a single line of work. One line focused on communication systems, including low-bitrate video calling based on transmitted landmarks and receiver-side generative reconstruction rather than conventional pixel streaming (Oquab et al., 2020). A second line developed chat-centric video understanding models that connect video encoders to LLMs through neural token interfaces, enabling multi-round dialogue grounded in spatiotemporal content (Li et al., 2023, Jin et al., 2023). A third line emphasized embodied interaction, including browser-based face-to-face dialogue with emotion-aware prompting, 3D avatar chat, and real-time non-verbal response synthesis (Alnuhait et al., 2023, Park et al., 2024, Costea et al., 2024). More recent work reframed the entire real-time communication stack around a cloud MLLM as the receiver, explicitly arguing for a shift from “humans watching video” to “AI understanding video” (Wu et al., 14 Jul 2025, Wu et al., 13 Feb 2026).

A concise way to organize the field is by what the AI is expected to do with the incoming stream.

System family	Core mechanism	Reported emphasis
Low-bandwidth video chat	Landmarks + generative face reconstruction	Few kbit/s mobile-compatible calling (Oquab et al., 2020)
Emotion-aware or avatar chat	ASR, LLM dialogue, TTS, animation	Face-to-face interaction and expressivity (Alnuhait et al., 2023, Park et al., 2024)
Chat-centric video understanding	Video encoder + token interface + LLM	Spatiotemporal reasoning and video-grounded conversation (Li et al., 2023, Jin et al., 2023)
Personalized or grounded video chat	Subject-aware ViLLM or LLM+SAM2	Identity-aware QA and referential grounding (Shi et al., 21 Mar 2025, Sun et al., 24 May 2025)
AI-oriented RTC	Network control optimized for MLLM understanding	Latency and accuracy under unstable uplinks (Wu et al., 14 Jul 2025, Wu et al., 13 Feb 2026)

This literature also changes the operative quality criterion. In traditional RTC, metrics such as SSIM, VMAF, and stalls are central. In AI video chat, recent systems instead foreground MLLM response accuracy and end-to-end latency as the primary quality axes, because the receiver is an inference stack rather than a human observer (Wu et al., 14 Jul 2025, Xu et al., 1 Oct 2025). A plausible implication is that AI video chat is less a single application than a systems umbrella joining multimodal modeling, interaction design, and transport optimization.

2. End-to-end architectures and processing loops

Several concrete architectures illustrate how AI video chat systems are assembled. FaceChat is implemented as a WebRTC-based browser front-end plus a Python server back-end. The browser captures raw video at $400\times 300@20$ FPS and audio downsampled to $16$ kHz, then streams chunks over Socket.IO. The server buffers packets and runs three asynchronous services: an engagement thread, an emotion thread, and a main dialogue thread. Within the main thread, the system waits for the engagement module to signal end of utterance, invokes Whisper ASR on buffered audio, queries GPT-3 with a prompt containing persona, history, user text, and facial emotion, synthesizes the reply with VITS, drives talking-face animation, and streams bot audio plus APNG back to the client (Alnuhait et al., 2023).

AVIN-Chat adopts a different embodiment but a similarly modular decomposition. It is organized into Facial Avatar Generation, Text–Speech Processing, and Speech-Driven Facial Animation, orchestrated by a Unity front-end and a Python back-end. Offline, HRN reconstructs a 3D face mesh and texture from a single RGB image, and LDT generates $N=52$ blendshape meshes. Online, Whisper performs STT, ChatGPT produces the text reply, EmotiVoice synthesizes speech, and EmoTalk predicts per-frame blendshape weights $w^t$ , which Unity applies to the neutral mesh and blendshape deltas in real time (Park et al., 2024). The geometric update is given as

$S^t = B_0 + w^t\cdot(B-B_0),$

where $B_0$ is the neutral mesh and $B=[B_1,\dots,B_n]$ are the blendshape bases (Park et al., 2024).

Maia pushes the architecture further toward non-verbal interaction. Its published pipeline is webcam capture at $30$ FPS, chunking into $T=30$ or $60$-frame segments, facial keypoint extraction with Google Mediapipe, keypoint-to-keypoint transformation by one of three methods, and real-time avatar animation through VSeeFace. The paper reports three complementary reaction generators: PCA-Teacher, NN-Student, and Retrieval (Costea et al., 2024). In this system, engagement is not explicitly scored; the raw keypoint sequence is treated as the current expression and passed directly into the reaction generators (Costea et al., 2024).

Commercial live-video assistants described in the assistive-technology study follow a more opaque but still recognizable loop. ChatGPT Advanced Voice with Video captures camera frames at approximately $16$0–$16$1 fps, performs streaming ASR, fuses conversation history, latest transcript, and visual features from recent frame(s), and produces a response with $16$2–$16$3 s latency. The loop is turn-based: the system does not proactively speak between user turns (Chang et al., 5 Aug 2025). That turn-taking property becomes important in applications requiring continuous environmental guidance.

3. Multimodal representation, fusion, and grounding

The field contains markedly different fusion strategies. FaceChat uses a late-fusion, prompt-injection design rather than a learned attention fusion network. Its emotion thread writes the current label $16$4 to a shared variable, and when ASR yields utterance $16$5, the prompt becomes a concatenation of persona, instruction, user text, and the textualized visual cue “The user looks $16$6.” No additional fine-tuning is performed; adaptation is driven purely by the inserted sentence (Alnuhait et al., 2023). This design is minimal, but it establishes a general pattern in AI video chat: non-text modalities are often converted into language-side control tokens.

VideoChat represents a more explicitly learned alternative. In VideoChat-Embed, a video encoder $16$7 processes sampled frames, a Q-Former-based token interface $16$8 converts video features into a small set of language-compatible video tokens, and a frozen or lightly tuned LLM autoregressively generates responses (Li et al., 2023). Nearly all video encoder and LLM weights remain frozen; only GMHRA, the Q-Former queries, and the projection head are trained (Li et al., 2023). The model was positioned as an early prototype for chat-centric video understanding rather than a low-latency live system.

Chat-UniVi generalizes this line by using dynamic visual tokens to uniformly represent images and videos through a unified visual representation. It starts from CLIP patch embeddings, applies parameter-free DPC-KNN clustering at multiple scales, and for video first groups frame-level representations into temporal events. For images, the reported multi-scale configuration uses $16$9, $N=52$ 0, and $N=52$ 1 clusters, so that $N=52$ 2 visual tokens are passed to the LLM instead of the original per-frame token count (Jin et al., 2023). The resulting token sequence is linearly projected and prepended to the text prompt, allowing the Vicuna backbone to use standard self-attention over both visual and textual tokens (Jin et al., 2023). This suggests a token-efficiency agenda that is directly relevant to interactive video systems.

PVChat specializes the representation problem around identity. It introduces a one-shot personalized ViLLM that learns a new subject from a single reference video and then answers subject-aware questions in a query video (Shi et al., 21 Mar 2025). Architecturally, it starts from a standard ViLLM pipeline with a frozen visual encoder, a Q-Former or lightweight projector, and an autoregressive LLM, then replaces standard multi-head attention with a Mixture-of-Heads extension and a ReLU Routing MoH attention mechanism. The system is trained on an automatically synthesized dataset containing identity-preserving positives, hard negatives, and four QA types: existence, appearance, action, and location (Shi et al., 21 Mar 2025). At inference time, identity-specialized heads activate only for frames containing the target subject, enabling personalized answers after one-shot learning (Shi et al., 21 Mar 2025).

SAMA extends grounding even further by unifying multi-turn referential dialogue and segmentation. Its architecture combines a visual encoder, a Spatio-Temporal Context aggregator, an LLM, and SAM2. Keyframe tokens preserve fine detail, long-range context tokens are summarized by the STC aggregator, and when the LLM emits a special $N=52$ 3 token, the hidden state $N=52$ 4 is fed into the frozen SAM2 decoder to produce a pixel mask (Sun et al., 24 May 2025). The associated dataset, SAMA-239K, contains $N=52$ 5 clips, $N=52$ 6 object-level descriptions, and $N=52$ 7 refer-ground QA pairs (Sun et al., 24 May 2025). In encyclopedia terms, SAMA marks the point at which “video chat” ceases to mean only descriptive QA and becomes a grounded interface over spatiotemporal regions.

4. Emotion, embodiment, and non-verbal interaction

A distinct branch of AI video chat treats emotional and non-verbal behavior as first-class interaction variables. FaceChat’s facial-expression pipeline uses DeepFace with OpenCV+SSD in a single-stage detector to crop the face and a 4-way expression classifier over happy, sad, angry, and neutral. In the deployed system, this runs at $N=52$ 8 FPS in its own thread, and the most recent classification is cached and passed to the prompt generator (Alnuhait et al., 2023). The important design point is not only that emotion is recognized, but that the recognized state directly conditions language generation.

AVIN-Chat also tunes dialogue and voice by emotional state, but it does not perform automatic emotion recognition. Instead, users choose from preset labels such as “happy,” “calm,” and “excited” through the Unity GUI, and the chosen label is concatenated into prompts for both ChatGPT and EmotiVoice (Park et al., 2024). The paper describes this as user-selected emotional state tuning. The system therefore closes the affective loop at the interface level rather than the perception level.

Maia approaches non-verbal interaction through transformation of facial motion rather than linguistic prompting. Mediapipe Face Mesh outputs $N=52$ 9 $w^t$ 0 landmarks per frame, and the system uses one of three transformation methods—PCA-Teacher, NN-Student, or Retrieval—to map the observed sequence into an expressive avatar animation (Costea et al., 2024). The custom 3D character “Maia,” created in VRoidStudio with an oil-paint texture, is driven by the predicted keypoints via VSeeFace. The published system does not include body-keypoint or skeleton modeling, and body movement is limited to head and facial motion (Costea et al., 2024). In this line of work, emotional translation is realized through motion synthesis rather than text-conditioned prosody.

These systems also expose a recurring misconception: realistic voice or animated embodiment does not by itself imply robust situational intelligence. The assistive study found that participants often perceived ChatGPT’s live-video voice as “cheery,” “natural,” and “realistic,” yet still encountered hallucinations, assumptions about visual ability, and risky spatial inaccuracies (Chang et al., 5 Aug 2025). Human-likeness in output style and grounded reliability in the environment are therefore separate properties.

5. Real-time transport, compression, and latency optimization

Recent work treats network transport as a core component of AI video chat rather than as a transparent substrate. In the Artic framework, one endpoint is a cloud MLLM and the QoE objective is shifted to response accuracy and end-to-end latency (Wu et al., 14 Jul 2025). This immediately exposes a mismatch with conventional RTC. The paper reports that current MLLMs process no more than $w^t$ 1 FPS and no more than $w^t$ 2 K pixels per frame, so clients must downsample. It also reports that GPT-4o audio-only tokens cost approximately $w^t$ 3 ms of compute, implying that if total response latency is to remain below $w^t$ 4 ms, network transmission must finish within $w^t$ 5 ms (Wu et al., 14 Jul 2025).

Artic’s first network mechanism is context-aware streaming. In the 2025 formulation, the latest frame is split into patches, CLIP visual features $w^t$ 6 are compared with a text feature $w^t$ 7 derived from the current utterance $w^t$ 8, and importance is computed by cosine similarity

$w^t$ 9

Patch quantization is then assigned according to

$S^t = B_0 + w^t\cdot(B-B_0),$ 0

so that bitrate is focused on chat-important regions (Wu et al., 14 Jul 2025). The same paper adds Loss-Resilient Adaptive Frame Rate, exploiting the fact that the sender can transmit redundancy within the MLLM’s sampling window when loss is high (Wu et al., 14 Jul 2025).

The 2026 Artic paper generalizes this into a broader AI-oriented RTC stack with Response Capability-aware Adaptive Bitrate and Zero-overhead Context-aware Streaming (Wu et al., 13 Feb 2026). RCAB incorporates both congestion-control bandwidth estimate $S^t = B_0 + w^t\cdot(B-B_0),$ 1 and MLLM confidence feedback $S^t = B_0 + w^t\cdot(B-B_0),$ 2. The update rule is

$S^t = B_0 + w^t\cdot(B-B_0),$ 3

where $S^t = B_0 + w^t\cdot(B-B_0),$ 4 is derived from the confidence gap relative to threshold $S^t = B_0 + w^t\cdot(B-B_0),$ 5 (Wu et al., 13 Feb 2026). ZCS then uses MLLM-provided bounding boxes and predicted $S^t = B_0 + w^t\cdot(B-B_0),$ 6 s trajectories to drive per-patch quantization without additional client-side saliency computation (Wu et al., 13 Feb 2026).

This systems focus sits alongside earlier compression work that reduced the transmitted representation itself. “Low Bandwidth Video-Chat Compression using Deep Generative Models” replaces a conventional video stream with facial landmarks plus a single reference image. Landmark displacement vectors are quantized and Huffman-encoded into, on average, $S^t = B_0 + w^t\cdot(B-B_0),$ 7 bits per frame for $S^t = B_0 + w^t\cdot(B-B_0),$ 8 landmarks, corresponding to approximately $S^t = B_0 + w^t\cdot(B-B_0),$ 9 kbit/s at $B_0$ 0 FPS. The receiver then reconstructs the face with a small generator network. Quantized models are about $B_0$ 1 MB and run on an iPhone 8 CPU at $B_0$ 2– $B_0$ 3 FPS end-to-end (Oquab et al., 2020). This is not an MLLM chat system, but it remains directly relevant as a communication-efficient embodiment substrate.

Measurements on mainstream applications show that production AI video chat is still highly heterogeneous. One 2025 campaign reports that ChatGPT, Grok, Doubao, and Yuanbao use RTP/RTCP, while Gemini uses QUIC. Reported uplink bitrates are approximately $B_0$ 4– $B_0$ 5 Mbps for ChatGPT and Grok, approximately $B_0$ 6 Mbps for Doubao and Yuanbao, and approximately $B_0$ 7 Mbps for Gemini; downlink remains below $B_0$ 8 Mbps in all cases. Reported frame rates range from $B_0$ 9– $B=[B_1,\dots,B_n]$ 0 fps for ChatGPT to $B=[B_1,\dots,B_n]$ 1 fps for Yuanbao, $B=[B_1,\dots,B_n]$ 2 fps for Doubao, and approximately $B=[B_1,\dots,B_n]$ 3 fps for Grok and Gemini (Xu et al., 1 Oct 2025). This diversity indicates that the transport layer has not converged on a standard design.

6. Evaluation regimes, applications, and unresolved issues

Evaluation in AI video chat remains fragmented because different systems optimize different objectives. FaceChat evaluates end-to-end latency from “user-finished-speaking” to “system-starts-speaking.” On an RTX 3090 Ti server, the reported mean $B=[B_1,\dots,B_n]$ 4 standard deviation for GPT-3 davinci with Whisper medium is $B=[B_1,\dots,B_n]$ 5 s, and the paper selected this configuration as an “acceptable if < 2 s” trade-off. Emotion classification and VAD latencies are reported as sub- $B=[B_1,\dots,B_n]$ 6 ms and parallelized. No formal user-study scores are reported, although the examples qualitatively show empathetic or de-escalating replies when the detected emotion changes (Alnuhait et al., 2023).

AVIN-Chat emphasizes subjective experience. In a study with $B=[B_1,\dots,B_n]$ 7 participants comparing text-only, speech-only, and full audio-visual interaction, AVIN-Chat received mean preference scores of $B=[B_1,\dots,B_n]$ 8 for intimacy, $B=[B_1,\dots,B_n]$ 9 for immersiveness, $30$0 for empathy, and $30$1 for overall satisfaction on 5-point scales. The paper reports that AVIN-Chat outperforms the baselines by approximately $30$2–$30$3 points, with paired $30$4-tests yielding $30$5 (Park et al., 2024). Maia, by contrast, reports human classification accuracy over three emotion classes; majority accuracies are $30$6 for PCA-Teacher, $30$7 for NN-Student, and $30$8 for Retrieval (Costea et al., 2024).

Subject-aware and grounded systems use more task-specific metrics. PVChat evaluates existence with Accuracy, and appearance, action, and location with BLEU and BERTScore, alongside Entity Specificity and Descriptive Completeness. Across six scenarios, PVChat reports Acc $30$9, BLEU $T=30$ 0, BERTScore $T=30$ 1, ES $T=30$ 2, and DC $T=30$ 3, compared with InternVideo2 at Acc $T=30$ 4, BLEU $T=30$ 5, BERTScore $T=30$ 6, and VideoLLaMA2 at Acc $T=30$ 7, BLEU $T=30$ 8, BERTScore $T=30$ 9 (Shi et al., 21 Mar 2025). SAMA-Bench evaluates multi-turn referential understanding and region-level captioning with mIoU, Recall@0.5, METEOR, CIDEr, CLAIR, and standard segmentation metrics; the excerpted benchmark reports SAMA-8B at mIoU $60$0, Recall $60$1, METEOR $60$2, CIDEr $60$3, and CLAIR $60$4 (Sun et al., 24 May 2025).

At the application level, the literature spans counseling, emotional support, and personalized customer service for FaceChat (Alnuhait et al., 2023); smart healthcare and smart home identity-aware comprehension for PVChat (Shi et al., 21 Mar 2025); and blind or low-vision assistance in unfamiliar indoor and outdoor environments for live-video agents (Chang et al., 5 Aug 2025). Yet the limitations are equally explicit. In the assistive study, current live-video AI was effective for guidance and answers in static visual scenes but failed to provide essential live descriptions in dynamic situations. One participant issued $60$5 separate umbrella-search prompts before success. The study also reports spatial and distance inaccuracies, hallucinations, generic responses, sycophantic reversals, and confusion caused by assumptions about users’ visual abilities (Chang et al., 5 Aug 2025).

System-level benchmarking reinforces these concerns. The 2025 measurement campaign reports real-time visual understanding accuracy between $60$6 and $60$7, while humans under the WhatsApp baseline are reported at approximately $60$8. It also reports that proactive output scored zero across all tested AI chatbots, median response delays were approximately $60$9–$16$00 s depending on app, setup time was at least $16$01 s for ChatGPT and no more than $16$02 s for the others, and peak-hour response delays for ChatGPT and Doubao increased by approximately $16$03 under load (Xu et al., 1 Oct 2025). These observations constrain any strong claim that current AI video chat has already reached humanlike conversational continuity.

The open problems identified across the literature are consistent. Artic highlights proactive context awareness, long-term MLLM memory, and context-aware token pruning (Wu et al., 14 Jul 2025). The BVI study argues for persistent user profiles, explicit spatial memory via Visual SLAM, proactive intervention timing beyond turn-taking, and additional sensing such as IMU or depth sensors (Chang et al., 5 Aug 2025). AVIN-Chat proposes automatic emotion recognition, full-body avatar animation, and lightweight on-device LLMs (Park et al., 2024). Taken together, these directions suggest that the central research challenge is no longer merely making an AI talk over video; it is coordinating multimodal understanding, grounding, embodiment, memory, and network control so that the interaction remains accurate, timely, and safe under real-world conditions.