MaineCoon: Real-Time Social World Model

• MaineCoon is a 22B-parameter real-time social world model that generates synchronized audio-video streams to simulate human-centric social dynamics.
• It uses an autoregressive streaming framework with conditional flow-matching to process visual and acoustic tokens, ensuring sub-second interaction and high-FPS performance.
• Training innovations like self-resampling, cross-modal alignment, and online-policy distillation enable efficient, long-horizon generation on a single GPU.

Searching arXiv for the MaineCoon paper and a small set of explicitly mentioned related works for citation support. MaineCoon is a 22B-parameter real-time audio-visual autoregressive model positioned as the generative core of a “social world model,” a model class intended to simulate human-centric social dynamics rather than only physical scene dynamics. In the reported formulation, MaineCoon models the joint distribution of visual and acoustic behavioral tokens conditioned on multimodal history and human inputs, supports causal streaming generation with sub-second interaction, and reaches up to 47.5 FPS at 480p on a single H100 GPU. The system is explicitly targeted at AI-native social platforms, including streaming avatars, conversational agents, interviews, debates, musical performances, dance, reaction content, and “social memes” (Bai et al., 16 Jun 2026).

1. Social world models and the problem MaineCoon addresses

MaineCoon is introduced under the label social world model, with a formal distinction from conventional world models. Instead of modeling a physical transition law of the form

$P(X_t \mid X_{<t}, U_{<t}),$

the model is framed as learning

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$

where $V_t$ represents visual behavioral tokens such as facial expression, gestures, and framing; $A_t$ represents acoustic behavioral tokens such as phonemes, prosody, and ambient audio; and $H_{<t}$ summarizes the user’s multimodal interaction and state (Bai et al., 16 Jun 2026).

The stated conceptual difference from classic world models is that humans become the primary coordinate system. Traditional world models for robotics or games center objects, geometry, physics, and explicit actions; MaineCoon instead treats speech timing, gaze, emotion, and conversational pacing as the dominant latent structure. This places the system in a different operational regime: not environment simulation in the narrow physical sense, but high-fidelity, long-horizon social interaction.

A recurrent misconception is that MaineCoon is presented as a complete social agent. The paper does not make that claim. Its scope is the real-time generative core, described as the reactive “System 1” side: synchronized audio-video generation, causal streaming, sub-second interaction, and long-horizon stability on a single GPU. Higher-level memory, planning, and reasoning are explicitly assumed to reside above this interface.

2. Autoregressive formulation and backbone design

MaineCoon is a 22B-parameter causal audio-visual diffusion transformer trained natively in streaming mode. The model operates on synchronized chunks

$$\mathbf{x}_{1:T} = \{\mathbf{x}_1, \dots, \mathbf{x}_T\},$$

with each chunk defined as

$\mathbf{x}_t = (\mathbf{x}_t^v, \mathbf{x}_t^a),$

where the visual and audio latents cover the same temporal extent. The training chunk size is 2, meaning two latents per chunk sharing the same time window. The causal factorization is

$$p_\theta(\mathbf{x}_{1:T} \mid \mathbf{c}) = \prod_{t=1}^T p_\theta(\mathbf{x}_t \mid \mathbf{x}_{<t}, \mathbf{c}),$$

with $\mathbf{c}$ aggregating conditions such as text prompt, speech transcript, scene description, and domain tags (Bai et al., 16 Jun 2026).

Each chunk is generated with conditional flow-matching (rectified flow). For a target chunk $\mathbf{x}_t$, the noised latent and velocity target are

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$0

with $P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$1 and $P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$2. The network predicts the velocity field conditioned on the noised target chunk, noise level, committed history, and conditioning context. The native autoregressive training objective is

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$3

The representation layer is initialized from LTX-2.3. Video is encoded by a VAE-style tokenizer into spatio-temporal latents, with temporal downsampling factor 8 and valid frame counts

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$4

such as 9, 25, and 41. Audio is tokenized into latents aligned to the same temporal grid. The backbone itself is a single 22B unified audio-visual DiT that processes a doubled sequence

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$5

combining history latents and current target latents within one transformer stream.

Synchronization is enforced structurally through joint audio-video token processing with modality-specific self-attention and cross-modal audio-to-video and video-to-audio attention paths. Noise conditioning is injected by AdaLN-like mechanisms with history tokens tagged by noise level 0 and target tokens tagged by the sampled $P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$6:

$P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$7

The paper emphasizes that attention patterns, KV-cache behavior, chunk ordering, and masking during training exactly match streaming inference. This suggests that latency and stability are treated as architectural constraints rather than after-the-fact deployment optimizations.

The temporal attention mask is block-causal and further constrained by a sink-plus-window retention scheme. A first $P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$8-chunk sink is preserved, while the most recent $P(V_t, A_t \mid V_{<t}, A_{<t}, H_{<t}),$9 chunks form a sliding window. The resulting mask is implemented as a FlexAttention block mask, which is the mechanism used to preserve scalability over long horizons.

3. Training innovations: self-resampling, alignment, specialization, and consolidation

The first major training mechanism is self-resampling, designed to remove the train-test mismatch induced by pure teacher forcing. Instead of always conditioning on clean history, the training procedure samples, with probability $V_t$0, whether to replace clean history $V_t$1 with self-resampled history $V_t$2. The history chunks are rolled out by a single-step flow-matching sampler under stop-gradient:

$V_t$3

The self-resampled objective is

$V_t$4

and the curriculum combines clean-history and self-resampled training:

$V_t$5

Early training uses small $V_t$6 and short rollouts; later training increases both. The technical significance is that MaineCoon is trained directly under the deployed causal streaming regime rather than through a non-causal teacher-forcing proxy (Bai et al., 16 Jun 2026).

The second mechanism is cross-modal representation alignment, also called streaming REPA in the paper. A frozen V-JEPA 2 encoder produces teacher features on each clip, and MaineCoon aligns mid-to-late-layer visual target-token representations to the teacher through token-relation distillation. If $V_t$7 denotes the cosine relation matrix across flattened spatio-temporal tokens, the hinge-style loss is

$V_t$8

Only visual targets are aligned; audio is not explicitly constrained by this loss. The reported effect is earlier emergence of semantics, motion structure, and cross-frame coherence.

The third component is domain-aware preference optimization. The paper argues that social video is heterogeneous across far-shot content, multi-person dialogue, high-motion scenes, animation, and dance, so a single objective cannot balance all quality desiderata. The solution is a specialize-then-consolidate pipeline. First, the base streaming model is trained with the native and alignment objectives. Second, domain-specific preference pairs $V_t$9 are constructed, where $A_t$0 are high-quality real videos and $A_t$1 are generated dispreferred samples. Third, domain-specific DPO experts are trained as LoRA adapters $A_t$2 over the base model $A_t$3, with expert parameters

$A_t$4

The DPO-style loss is defined in flow-matching space relative to the base model and augmented by a small winner reconstruction term.

The consolidation step is reinforced online-policy distillation (ROPD). Rather than deploying a bank of LoRAs or an explicit router, ROPD distills multiple frozen domain experts into one unified student policy. Candidate chunks are sampled from a slow-moving EMA behavior policy, binary domain-specific verifiers produce outcomes $A_t$5, and an adaptive weight $A_t$6 controls expert influence. The reward-weighted target velocity is

$A_t$7

and the student learns to regress to this velocity target. If all candidates fail, the expert is strongly injected; if all succeed, expert influence vanishes. The stated outcome is a single unified streaming MaineCoon policy, with all LoRAs and verifiers removed at deployment.

A final systems-critical step is step distillation. The paper reports the use of DMD-style distribution-matching distillation and variants to obtain an almost lossless 4-step sampler from the original LTX-2.3 teacher. These distilled weights are described as critical for the few-step, high-FPS streaming regime.

4. Streaming inference, agentic control, and long-horizon memory

MaineCoon is embedded in an agentic streaming inference framework composed of three controller subsystems: an agentic planner and observer, an agentic cache manager, and an agentic look-ahead buffer controller (Bai et al., 16 Jun 2026). The framework is training-free: the controllers are external to the learned generator.

On a single H100 GPU, the reported throughput is approximately 31 FPS at 480p with chunk size 2, and 47.5 FPS at 480p over 20 seconds when the inference chunk size is increased to 6. The paper attributes this to a combination of the native causal architecture with sliding KV cache, the 4-step rectified-flow sampler, sliding-window VAE decoding, block-by-block compilation, and overlap between generation and decoding or encoding in separate processes. Cost is reported as significantly below \$0.001 per second of generated video.

The planner and observer is a locally deployed Gemma 4 26B MoE agent. In planner mode, it maintains a bounded planning history and transcript, emits beat-level prompts containing visual conditions, one line of speech, and ambient scene context, and restates character specifications to preserve identity. In observer mode, it monitors the generation head ahead of what the viewer sees, using inexpensive photometric drift metrics and periodic VLM checks such as identity or wardrobe drift. When defects are detected, the repair ladder is forward-only: refresh the subject anchor, restate canonical appearance in the prompt, regenerate the beat, or narratively steer out of the bad state. The paper is explicit that there is no hard reset of the stream.

The cache manager provides the model’s long-horizon memory. After each chunk is generated, a final zero-noise pass yields a clean latent, and that clean latent is committed to a single persistent KV cache for the session. The keep-set is non-contiguous:

$A_t$8

Here $A_t$9 denotes early scene-sink chunks, $H_{<t}$0 scene anchors, $H_{<t}$1 recent chunks, $H_{<t}$2 subject anchors, and $H_{<t}$3 restored chunks when a scene returns. Positions remain within the training horizon by assignment to epoch slots, with cache rebuilds from retained clean latents when slots are exhausted. The reported result is up to approximately 45 minutes of continuous streaming with negligible degradation.

Two anchor mechanisms are used for drift control. The statistical anchor applies per-channel statistic matching only on the committed copy:

$H_{<t}$4

The subject anchor uses an open-vocabulary segmenter to identify the subject region from text, harvests a fixed number of high-confidence latent tokens from clean latents, and warms them into the cache as a dedicated anchor block that is never decoded to output. The paper states that with anchors, drift becomes reversible.

The look-ahead buffer controller regulates the trade-off between smooth playback and responsiveness. Because generation is faster than playback, already-generated but unwatched content accumulates. The controller maintains a target lead by throttling or sprinting, pauses generation when the viewer pauses, and switches prompts only when the current scripted line has actually been spoken, determined from decoded audio silence or ASR coverage. This ensures prompt changes occur at natural utterance boundaries rather than arbitrary time cuts. A plausible implication is that responsiveness is engineered not only through raw model speed but also through explicit temporal policy over when a prompt is allowed to take effect.

5. Data construction, benchmark design, and empirical performance

The training corpus is deliberately shaped around social video rather than cinematic video. It has two principal sources: synthetic audio-visual data from LTX-2.3 and real short-video social content. The total set is reported as approximately 1M extremely high-quality long clips, chosen to keep 22B streaming training tractable at approximately $H_{<t}$5 H100 GPU hours (Bai et al., 16 Jun 2026).

The synthetic pipeline is designed to create multi-segment, prompt-switching stories with full sampling trajectories retained for step distillation and streaming supervision. A director-style LLM samples scenarios from a taxonomy of 225 scenes across 10 groups, 15 visual styles, and 12 camera shots. Each scenario is decomposed into 3–4 linked clips totaling approximately 20 seconds, with fixed character identity but evolving shot, action, dialogue, and audio. Quality filtering combines analyzers for video, audio, sync, and caption quality, as well as caption-frame consistency scoring from Gemini 3.1 Flash.

The real data pipeline processes tens of millions of raw social videos into person-centric, speech-synchronized clips. Low-level filtering removes abnormal FPS, unsupported durations and resolutions, and persistent on-screen text via EasyOCR. Shot segmentation uses TransNetV2. High-level filtering uses SCRFD to remove faceless content, a spectral pitch heuristic to ensure a single speaker, and SyncNet to keep only windows where the visible speaker matches the audio. Speech transcription uses Demucs for source separation and Faster-Whisper ASR for time-aligned sentence segments. Valid clips are bucketed according to the latent-frame rule $H_{<t}$6 with forms $H_{<t}$7, resized to 480 on the short edge, cropped to either $H_{<t}$8 or $H_{<t}$9, and re-encoded with libx264.

Because the pre-training corpus is dominated by easy close-up, low-motion talking heads, the paper also constructs domain-balanced post-training data. Domain profiling decodes eight frames per clip, estimates shot scale with YOLO11x from largest-person area, measures motion by mean SigLIP embedding change, and clusters content into domains such as wide shot, high motion, multi-person, first-person, and close-up. Rare hard domains are upweighted for domain-aware preference training and ROPD.

Evaluation is carried out on SocialVideo Bench, a 700-sample benchmark covering 7 domains × 100 prompts: dense speech, two-person interaction, music and vocal, emotional performance, dance, creative stress test, and social memes. Each sample contains two consecutive 10-second segments, with the second segment using an updated prompt. The benchmark is thus designed not only for audiovisual fidelity but also for conditioning changes under temporal continuity.

The paper reports nine metrics: Vis, Mot, Aud, IB-TV, IB-TA, IB-AV, AV-Al, AVH, and JAVIS, plus an Average metric defined as the average of max-normalized versions of the nine metrics. The reported headline results are summarized below.

Item	Reported value	Context
Parameters	22B	Unified audio-visual DiT
Throughput	47.5 FPS	480p, 20 seconds, single H100, chunk size 6
Visual quality	4.71	Best on SocialVideo Bench
Audio quality	4.35	Best on SocialVideo Bench
AVH	0.308	Best; previous best 0.291
JAVIS	0.272	Best; previous best 0.247
Average	0.934	Best; next best 0.895 from SoulX-FlashTalk

The comparative framing is notable. MaineCoon is a causal streaming T2AV system, while several strong baselines, including LTX-2.3 and MoVA, are described as bidirectional generation systems operating over full sequences. Nevertheless, MaineCoon is reported to dominate the joint audio-visual metrics and remain at least competitive on the others. In latency terms, the paper lists LiveAvatar at 6.7 FPS, SoulX-FlashTalk at 6.6 FPS, Causal Forcing at 19.1 FPS, Helios-Distilled at 18.2 FPS, LTX-2.3-distilled at 20.7 FPS, and MaineCoon at 47.5 FPS. The authors interpret this as evidence that the efficiency gain comes from architecture and streaming design rather than from parameter reduction.

6. Scope, limitations, safety questions, and projected trajectory

The paper’s own scope statement is narrow in one important respect: MaineCoon is not presented as solving higher-level social reasoning. The generator is strong at synchronized audio-video production, but planning, safety, and personality coherence are delegated to the external agent. This is central to understanding both its contribution and its limits (Bai et al., 16 Jun 2026).

The explicit limitations include long-horizon failure modes, where extreme-duration streams may still show subtle appearance drift, identity softening, or behavioral repetition if planner prompts degenerate. The system also remains vulnerable to audio-visual desynchronization arising from planner prompt discipline, ASR errors, or buffer misconfiguration. These are not framed as theoretical failures of the autoregressive core alone, but as coupled system-level risks across generation, prompting, observation, and timing control.

The safety discussion is limited. The data pipeline is human-centric and speech-centric, and the paper does not describe dedicated bias mitigation, harmful-content filtering, or moderation beyond avoiding on-screen text and watermarks and selecting natural social videos from public platforms. It identifies potential concerns: harmful or offensive social behaviors learned from data, persona and identity biases, and misuse of realistic avatars for impersonation or misinformation. The paper suggests, implicitly rather than formally, that mitigation would occur at the platform layer through the external agent and additional filters.

The projected research trajectory is toward fuller social world models with active observation modules and internal social simulators, described as “System 2,” layered above the MaineCoon-like reactive generator. The paper also points toward full-duplex interaction, in which a low-latency multimodal “reactive cerebellum” is paired with a larger deliberative planning model. This suggests a decomposition in which real-time social embodiment and high-level social cognition are treated as separable but tightly coupled subsystems.

In that sense, MaineCoon occupies a specific position in the research landscape: it is a real-time, streaming, causally trained audio-visual generative policy for social interaction, not a complete conversational intelligence stack. Its importance lies in making synchronized, high-FPS, long-horizon social video generation operational on a single GPU while preserving an interface on top of which memory, planning, verification, and moderation can be built.